Abnormal job processing method, system, device, equipment and medium

Abstract

The invention provides an abnormal job processing method, system, device and equipment and a medium, is applied to a job scheduling system, and relates to the technical field of computers, the method comprises the following steps: according to an execution instruction, distributing a target machine to a target job, the target machine being used for executing the target job, and after the target job starts to be executed, based on the state of the target machine, executing the target job; whether execution of the target operation is abnormal or not is detected, when it is detected that the target operation is abnormal, the target operation is added into an abnormal queue, at least one time of fault detection is conducted on the target machine according to configuration information, and the configuration information at least comprises a fault detection time interval of fault detection and a fault detection frequency threshold value; and according to a detection result, determining a scheduling scheme and re-executing the target job. Therefore, by means of a set fault detection and target job scheduling scheme determination mechanism, the intervention of an administrator is reduced, the stability and the automation level of the job scheduling system are improved, and then the job scheduling efficiency and reliability are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an abnormal operation processing method, system, device, equipment and medium. Background Art

[0002] With the continuous advancement of science and technology and the advent of the big data era, the scale and complexity of simulation tasks are increasing, and the requirements for job scheduling and resource management are becoming higher and higher. Therefore, it is particularly important for job scheduling systems to solve abnormal job problems.

[0003] In modern technology applications, simulation tasks often need to run for a long time and require a lot of computing resources. If an abnormal operation occupies resources for a long time, it will affect the execution of other normal operations, resulting in a decline in the performance of the entire system, and may even cause the system to crash. Therefore, timely detection and processing of abnormal operations is a key link in ensuring system stability and reliability. However, the current processing of abnormal operations cannot meet the needs of efficient, stable and automated processing of abnormal operations. Summary of the invention

[0004] In view of this, embodiments of the present application provide an abnormal operation processing method, system, device, equipment and medium to overcome the above-mentioned problems or at least partially solve the above-mentioned problems.

[0005] A first aspect of an embodiment of the present application provides a method for handling abnormal jobs, which is applied to a job scheduling system, and the method includes:

[0006] receiving an execution instruction carrying the target job from a user terminal, and assigning a target machine to the target job according to the execution instruction, wherein the target machine is used to execute the target job;

[0007] After the target job starts to be executed, based on the state of the target machine, detecting whether the execution of the target job is abnormal;

[0008] When an exception is detected in the execution of the target job, the target job is added to an exception queue, configuration information of the target job is obtained from a database, and at least one fault detection is performed on the target machine according to the configuration information, wherein the configuration information is pre-set by the user end for the target job and stored in the database, and the configuration information at least includes a fault detection time interval and a fault detection number threshold of the fault detection;

[0009] A scheduling scheme for the target job is determined according to a detection result of the at least one fault detection, and the target job is re-executed based on the scheduling scheme.

[0010] Optionally, the job scheduling system is configured with multiple queues, a single queue corresponds to multiple machines, and the allocating a target machine to the target job according to the execution instruction includes:

[0011] In a case where the execution instruction includes a first machine specified by the user end for executing a target job, determining the first machine as the target machine;

[0012] In a case where the execution instruction includes the queue specified by the user end, determining a target machine for executing the target job from a plurality of second machines corresponding to the queue specified by the user end;

[0013] In a case where the execution instruction does not include the queue and machine specified by the user, a target machine for executing the target job is determined from idle machines.

[0014] Optionally, performing at least one fault detection on the target machine according to the configuration information includes: performing at least one fault detection on the target machine according to the fault detection time interval and within a range not exceeding the fault detection number threshold, and updating the number of fault detections of the target machine after each fault detection is completed;

[0015] The step of determining a scheduling scheme for the target job according to the detection result of the at least one fault detection comprises:

[0016] When the number of fault detections does not exceed the fault detection number threshold and the target machine is detected to be normal, determining that the detection result of the target machine is normal, and determining a first scheduling scheme as the scheduling scheme for the target job, wherein the first scheduling scheme is: cleaning up data generated during the execution of the target job, and reallocating the target job to the target machine;

[0017] In the case where the number of fault detections exceeds the fault detection number threshold, determining that the detection result of the target machine is abnormal, and determining a second scheduling scheme as the scheduling scheme for the target job, the second scheduling scheme is: sending the detection result for the target machine to an administrator, performing fault detection on a third machine other than the target machine in the queue where the target machine is located, and allocating the target job to the third machine whose detection result is normal;

[0018] In the process of performing fault detection on the third machines other than the target machine in the queue where the target machine is located, each third machine is subjected to at most one fault detection.

[0019] Optionally, the configuration information further includes a machine specified by the user end for executing the target job, and the performing fault detection on a third machine other than the target machine in the queue where the target machine is located includes:

[0020] determining the number of the third machines, and determining a total number of fault detections based on the number of the third machines;

[0021] Perform fault detection on each of the third machines in turn, and update the remaining number of fault detections after each fault detection is completed, wherein the initial value of the remaining number of fault detections is the value of the total number of fault detections.

[0022] Optionally, the method further comprises:

[0023] In the process of sequentially performing fault detection on each of the third machines, when the detection result of the third machine currently undergoing fault detection is determined to be normal, it indicates that a third machine with a normal detection result is determined, and the fault detection of the remaining third machines is stopped;

[0024] Cleaning up the data generated during the execution of the target job, reassigning the target job to a third machine with a normal detection result, and updating the machine specified by the user end for executing the target job in the configuration information of the target job in the database according to the third machine with a normal detection result;

[0025] When the remaining fault detection times is 0, and the detection results of the third machine are all abnormal, the detection results for the third machine are sent to the administrator.

[0026] Optionally, the method further comprises:

[0027] When the target job starts to be executed, assigning a job identification number to the target job;

[0028] Associating the job identification number with the configuration information of the target job in the database;

[0029] The acquiring the configuration information of the target job from the database includes: acquiring the configuration information of the target job based on the job identification number of the target job in which the exception occurs.

[0030] A second aspect of an embodiment of the present application provides an abnormal job processing system, the system comprising a user terminal and a job scheduling system:

[0031] The user terminal is used to send a target job to the job scheduling system, and to set configuration information according to the target job, and store the configuration information in a database;

[0032] The job scheduling system is used to receive an execution instruction carrying the target job from a user terminal, and allocate a target machine to the target job according to the execution instruction, wherein the target machine is used to execute the target job;

[0033] The job scheduling system is used to detect whether an abnormality occurs in the execution of the target job based on the state of the target machine after the target job starts to be executed;

[0034] The job scheduling system is used to add the target job to an exception queue when detecting that an exception occurs in the execution of the target job, obtain configuration information of the target job from the database, and perform at least one fault detection on the target machine based on the configuration information, wherein the configuration information is pre-set by the user end for the target job and stored in the database, and the configuration information at least includes a fault detection time interval and a fault detection number threshold of the fault detection;

[0035] The job scheduling system is used to determine a scheduling scheme for the target job according to the detection result of the at least one fault detection, and re-execute the target job based on the scheduling scheme.

[0036] A third aspect of the embodiments of the present application provides an abnormal job processing device, which is applied to a job scheduling system, and the device includes:

[0037] A receiving module, configured to receive an execution instruction carrying the target job from a user terminal, and to allocate a target machine to the target job according to the execution instruction, wherein the target machine is used to execute the target job;

[0038] A machine detection module, configured to detect whether an abnormality occurs in the execution of the target job based on the state of the target machine after the target job starts to be executed;

[0039] a fault detection module, configured to, when detecting that an exception occurs in the execution of the target job, add the target job to an exception queue, obtain configuration information of the target job from a database, and perform at least one fault detection on the target machine according to the configuration information, wherein the configuration information is pre-set by the user end for the target job and stored in the database, and the configuration information at least includes a fault detection time interval and a fault detection number threshold of the fault detection;

[0040] The scheduling scheme determination module is used to determine the scheduling scheme of the target job according to the detection result of the at least one fault detection, and re-execute the target job based on the scheduling scheme.

[0041] A fourth aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the abnormal operation processing method as described in the first aspect.

[0042] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the abnormal job processing method as described in the first aspect is implemented.

[0043] Beneficial effects of this application:

[0044] The embodiments of the present application provide an abnormal job processing method, system, device, equipment and medium, which are applied to a job scheduling system, the method comprising: receiving an execution instruction carrying the target job issued by a user end, and allocating a target machine to the target job according to the execution instruction, the target machine being used to execute the target job; after the target job starts to execute, detecting whether the execution of the target job is abnormal based on the state of the target machine; when it is detected that the execution of the target job is abnormal, adding the target job to an abnormal queue, obtaining the configuration information of the target job from a database, and performing at least one fault detection on the target machine based on the configuration information, the configuration information is pre-set by the user end for the target job and saved in the database, and the configuration information at least includes a fault detection time interval and a fault detection number threshold of the fault detection; determining a scheduling scheme for the target job according to the detection result of the at least one fault detection, and re-executing the target job based on the scheduling scheme.

[0045] Through the technical solution of the present application, after receiving the execution instruction, the target machine can be assigned to the target job for automatic execution of the job, and during the execution of the target job, the target machine executing the target job can be detected in real time. When an abnormality is detected in the target machine, the target job is determined to be abnormal, and the target job is added to the abnormal queue. According to the fault detection time interval and the fault detection number threshold in the configuration information of the target job, at least one fault detection is performed on the target machine to obtain the detection result, and according to the detection result, a scheduling plan for further processing the target job in the abnormal queue is determined, and the target job is re-executed according to the scheduling plan. Therefore, through the set fault detection and the determination mechanism of the scheduling plan of the target job, the intervention of the administrator is reduced, the stability and automation level of the system are improved, and the efficiency and reliability of job scheduling are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.

[0047] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 It is a flow chart of an abnormal operation processing method provided by an embodiment of the present application;

[0049] Figure 2 is a flow chart of another abnormal operation processing method provided by an embodiment of the present application;

[0050] Figure 3 It is a structural diagram of an abnormal operation processing system provided by an embodiment of the present application;

[0051] Figure 4 It is a structural schematic diagram of an abnormal operation processing device provided by an embodiment of the present application;

[0052] Figure 5 It is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0053] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] bsub: A command used to submit jobs in the OpenLava system.

[0056] bjobs: A command used to query the job status in the OpenLava system.

[0057] bkill: A command used to terminate jobs in the OpenLava system.

[0058] bhosts: A command used to query machine information in the OpenLava system.

[0059] Zombie Job: refers to jobs that cannot be terminated normally because the execution node loses response.

[0060] JOB ID: Job identification number, used to uniquely identify a job in the system.

[0061] Fault detection: The process by which the system detects whether an execution node is available.

[0062] Resource Management: refers to the effective management and allocation of resources in a computer cluster to ensure that jobs can be executed smoothly and resources are fully utilized.

[0063] Fault Tolerance: refers to the ability of a system to maintain normal operation in the face of hardware or software failures, usually including automatic detection, repair and recovery functions.

[0064] Database Integration: refers to storing key data such as job configuration and status information in the database, and managing this data through the database to achieve automation and informatization of job management.

[0065] Abnormal jobs usually refer to processes (or tasks) that consume relatively large amounts of resources, which can also be referred to as zombie jobs (a term used in operating systems). Abnormal jobs may run for a long time without terminating, or may still occupy some system resources after running. The handling of abnormal jobs is usually part of system maintenance and optimization. System administrators can solve or prevent abnormal job problems by monitoring system resource usage, regularly cleaning up processes that have not been completed for a long time, and using system tools to detect and handle abnormal jobs. In some cases, when designing and debugging programs, developers will also try to avoid creating processes that run for a long time and do not release resources, in order to improve the overall performance and stability of the system.

[0066] There are currently some solutions to the problem of abnormal operations, such as regularly checking the queues where abnormal operations are located and manually clearing them. In this process, abnormal operations occupy the overall resources of the storage system. The need to manually handle abnormal operations not only consumes human resources and time, but is also prone to omissions or errors. When a machine fails, administrator intervention is also required. Therefore, the handling of abnormal operations in related technologies cannot meet the needs of efficient, stable and automated handling of abnormal operations.

[0067] Based on the above technical defects, an embodiment of the present application provides an abnormal job processing method, which is applied to a job scheduling system.

[0068] Figure 1is a flowchart of an abnormal operation processing method provided by an embodiment of the present application. Figure 2 It is a flowchart of another abnormal operation processing method provided by an embodiment of the present application.

[0069] like Figure 1 As shown, the method comprises the following steps:

[0070] Step S101, receiving an execution instruction carrying the target job from a user terminal, and assigning a target machine to the target job according to the execution instruction, wherein the target machine is used to execute the target job;

[0071] Step S102, after the target job starts to be executed, based on the state of the target machine, detecting whether the execution of the target job is abnormal;

[0072] Step S103, when it is detected that the execution of the target job is abnormal, the target job is added to the abnormal queue, the configuration information of the target job is obtained from the database, and according to the configuration information, at least one fault detection is performed on the target machine, wherein the configuration information is pre-set by the user end for the target job and stored in the database, and the configuration information at least includes a fault detection time interval and a fault detection number threshold of the fault detection;

[0073] Step S104: determining a scheduling scheme for the target job according to the detection result of the at least one fault detection, and re-executing the target job based on the scheduling scheme.

[0074] Firstly, the technical solution of the present application provides an abnormal job processing system, including a user terminal and a job scheduling system.

[0075] In step S101, when the host of the user needs to complete the target job, it sends an execution instruction carrying the target job to the job scheduling system. The execution instruction contains information related to the target job that the user wants to execute. After receiving the execution instruction of the target job, the job scheduling system will parse the execution instruction to understand the specific task of the target job that the user wants to execute, as well as the requirements of the user when executing the target job, such as: the execution machine, queue, and the fault detection number threshold and fault detection time interval required for fault detection when the job is abnormal. After parsing the execution instruction, the job scheduling system can further determine the target machine for executing the target job and assign the target machine to the target job.

[0076] The job scheduling system is used to schedule the execution of jobs in multiple queues, and each job corresponds to multiple machines for executing the job.

[0077] Further, in step S102, if Figure 2 As shown, after the target machine starts to execute the target job, the abnormal job processing system will continue to monitor the execution process of the target job and detect the target machine to ensure that the execution process is carried out within the normal range. And by obtaining the status of the target machine in real time, it is determined whether the execution of the target job is abnormal. When the execution time is too long, the resource usage is abnormal, the process crashes, the error output, etc. occur, it is considered that the execution of the target job is abnormal. In the process of detecting the target machine, the abnormal job processing system collects the status information of the execution process through the built-in monitoring mechanism or interacts with the target machine, and compares it with the preset normal execution parameters to determine whether the target job is abnormal.

[0078] Further, in step S103, if Figure 2 As shown, when an exception is detected in the execution of a target job, the target job is added to an exception queue, wherein the exception queue is used to temporarily store jobs with abnormal execution. Since the jobs in the exception queue cannot be executed temporarily, the exception queue has no corresponding machine.

[0079] Further, if Figure 2 As shown, configuration information related to the target job is obtained from the database, and based on the configuration information of the target job, a target machine for executing the target job is determined, and at least one fault detection is performed on the target machine according to the fault detection time interval and fault detection number threshold included in the configuration information. The fault detection time interval is the interval time between each fault detection, and the fault detection number threshold is the maximum number of fault detections.

[0080] The fault detection process involves checking the hardware status of the target machine, system resource usage, process monitoring, etc. to determine the root cause of the abnormality during the execution of the target job.

[0081] Among them, the configuration information is pre-set by the user side for the target job and saved in the database. In addition, when the pre-set configuration information is insufficient, the complete configuration information can also be obtained during the execution of the target job. For example, when the user side executes an instruction for a specified machine, the information of the target machine that actually executes the target job can be added to the configuration information during the execution of the target job.

[0082] Finally, in step S104, Figure 2As shown, after performing at least one fault detection on the target machine in step S103, at least one fault detection result is obtained. According to the at least one fault detection result, the current state information of the target machine can be determined, and then according to the current state information, the scheduling scheme of the target job is determined. The scheduling scheme can be used to decide how to adjust the subsequent execution strategy of the target job, including immediately re-executing the target job (possibly after repairing or adjusting certain configurations), re-executing the target job within a specific time, or abandoning the execution of the target job and adopting a backup plan, etc. Finally, the target job is re-executed according to the determined scheduling scheme.

[0083] Through the above-mentioned embodiments, after receiving the execution instruction, the target machine can be assigned to the target job for automatic execution of the job, and during the execution of the target job, the target machine executing the target job can be detected in real time. When an abnormality is detected in the target machine, the target job is determined to be abnormal, and the target job is added to the abnormal queue. According to the fault detection time interval and the fault detection number threshold in the configuration information of the target job, at least one fault detection is performed on the target machine to obtain the detection result, and according to the detection result, a scheduling plan for further processing the target job in the abnormal queue is determined, and the target job is re-executed according to the scheduling plan. Therefore, through the set fault detection and the determination mechanism of the scheduling plan of the target job, the intervention of the administrator is reduced, the stability and automation level of the system are improved, and the efficiency and reliability of job scheduling are improved.

[0084] Optionally, the job scheduling system is configured with multiple queues, a single queue corresponds to multiple machines, and the allocating a target machine to the target job according to the execution instruction includes:

[0085] In a case where the execution instruction includes a first machine specified by the user end for executing a target job, determining the first machine as the target machine;

[0086] In a case where the execution instruction includes the queue specified by the user end, determining a target machine for executing the target job from a plurality of second machines corresponding to the queue specified by the user end;

[0087] In a case where the execution instruction does not include the queue and machine specified by the user, a target machine for executing the target job is determined from idle machines.

[0088] Specifically, Figure 2As shown, in one embodiment, multiple queues are configured in the job scheduling system, and each queue corresponds to a group of machines. This design allows the job scheduling system to manage the execution process of different jobs in a more fine-grained manner. For example, it can select a suitable queue or machine based on factors such as job type, machine load, resource requirements, etc. Each queue contains a group of machines that can execute different jobs simultaneously or in rotation. The effective use of resources and efficient execution of jobs are guaranteed through queue management.

[0089] In step S101: according to the execution instruction, the process of allocating the target job to the target machine specifically includes the following steps:

[0090] When the user needs to execute a target job, the user can choose whether to specify the machine used to execute the target job or the queue of the target job. Therefore, the job scheduling system needs to allocate target machines for target jobs according to different execution instructions.

[0091] When the execution instruction includes the first machine specified by the user for executing the target job, it is implicit that the user has specified the queue corresponding to the first machine. Therefore, the first machine can be directly used as the target machine. This situation is suitable for when the user has specific preferences or requirements and expects the job to be executed on a specific machine to meet resource requirements, performance requirements or depend on the specific environment settings of the machine.

[0092] When the execution instruction includes a queue specified by the user, but does not specifically specify a machine in the queue, the job scheduling system selects the most suitable machine from the queue as the target machine.

[0093] The queue specified by the user usually provides certain priorities or attributes (such as resource priority, load conditions, etc.). The job scheduling system will select the most suitable machine to execute the target job based on the status of each machine in the queue specified by the user and the configuration information of the machine, which is conducive to flexible scheduling according to the business resource requirements of the target job.

[0094] When the execution instruction does not include the queue and machine specified by the user, the job scheduling system determines the target machine for executing the target job from multiple idle machines. In other words, when the execution instruction does not explicitly specify a queue or a specific machine, the job scheduling system will select a machine from the currently idle (not occupied by other jobs) machines as the target machine to execute the target job. This ensures that the job can be reasonably scheduled for execution even if the user does not specify it explicitly, which is suitable for automation or automatic scheduling scenarios.

[0095] Optionally, when there are multiple standby machines available for selection, the optimal target machine for executing the target job may be selected according to the following strategy:

[0096] Resource matching: First, evaluate the matching degree between the resources (such as CPU, memory, storage, bandwidth, etc.) of each backup machine and the current target job requirements. Selecting the machine whose resources best match the job requirements of the target job can ensure the efficiency of job execution. For example, if the target job mainly consumes computing resources, a machine with higher CPU and memory configuration should be selected.

[0097] Historical performance data: Use historical operation data to evaluate the stability and performance of different machines. Selecting machines with stable performance, low failure rate, and high operating efficiency in the past can reduce the possibility of job interruption and ensure high availability of the job.

[0098] Task adaptability: Analyze the hardware and software configurations of different machines and select the machine that best suits the current target job type. For example, if the target job involves a large amount of data reading and writing operations, it may be more appropriate to select a machine with good IO performance.

[0099] Load balancing: Considering the overall load of the current system, selecting machines with lower current loads can avoid creating new resource bottlenecks while ensuring full utilization of all available resources.

[0100] Cost-effectiveness: Choose the most cost-effective machine given the resources needed to meet the job requirements. This may include considering electricity costs, maintenance costs, and long-term resource efficiency.

[0101] Automated decision-making mechanism: Using algorithms and automation tools, such as machine learning models, to automatically select the optimal machine based on real-time system status, historical data, and predictive models. This can improve the efficiency and accuracy of decision-making.

[0102] Dynamic adjustment strategy: Dynamically adjust the machine selection strategy according to changes in system status. For example, during high load periods, you can select the machine with the most available resources.

[0103] Through the comprehensive application of these strategies, the decision-making process can be effectively optimized to ensure the efficient selection of the most suitable machine to perform the target job, thereby improving the stability and availability of the entire system.

[0104] Through the above embodiments, the execution instructions issued by the user are efficiently responded to through flexible configuration and response mechanisms, and the optimal target machine is allocated to the target job. It is possible to provide personalized job execution solutions according to different business needs and scenarios. This improves the efficiency of resource utilization and enhances the flexibility and responsiveness of the job scheduling system.

[0105] Optionally, performing at least one fault detection on the target machine according to the configuration information includes: performing at least one fault detection on the target machine according to the fault detection time interval and within a range not exceeding the fault detection number threshold, and updating the number of fault detections of the target machine after each fault detection is completed;

[0106] The step of determining a scheduling scheme for the target job according to the detection result of the at least one fault detection comprises:

[0107] When the number of fault detections does not exceed the fault detection number threshold and the target machine is detected to be normal, determining that the detection result of the target machine is normal, and determining a first scheduling scheme as the scheduling scheme for the target job, wherein the first scheduling scheme is: cleaning up data generated during the execution of the target job, and reallocating the target job to the target machine;

[0108] In the case where the number of fault detections exceeds the fault detection number threshold, determining that the detection result of the target machine is abnormal, and determining a second scheduling scheme as the scheduling scheme for the target job, the second scheduling scheme is: sending the detection result for the target machine to an administrator, performing fault detection on a third machine other than the target machine in the queue where the target machine is located, and allocating the target job to the third machine whose detection result is normal;

[0109] In the process of performing fault detection on the third machines other than the target machine in the queue where the target machine is located, each third machine is subjected to at most one fault detection.

[0110] Specifically, Figure 2 As shown, in one embodiment, according to the configuration information, at least one fault detection is performed on the target machine, specifically including: according to the fault detection time interval contained in the configuration information, within the range not exceeding the fault detection number threshold, a fault detection is performed on the target machine once every fault detection time interval (for example: every 10 minutes, 1 hour, etc.) (fault detection includes checking the health status of the target machine, resource usage, status of running tasks, etc.), and after each fault detection, the number of fault detections of the target machine is updated until the number of fault detections of the target machine reaches the fault detection number threshold (for example: a maximum of 5 fault detections), or when the target machine is detected to be normal, the updating of the fault detection number is stopped.

[0111] In step S104, determining a scheduling scheme for the target job according to the detection result of the at least one fault detection includes the following steps:

[0112] When the number of fault detections does not exceed the fault detection number threshold, and the target machine is detected to be normal, then the detection result of the target machine is determined to be normal. At this time, the first scheduling plan is determined as the scheduling plan for the target job, wherein the first scheduling plan is: cleaning the data generated during the execution of the target job, and reallocating the target job to the target machine;

[0113] When the number of fault detections exceeds a threshold value for the number of fault detections, the detection result of the target machine is determined to be abnormal, and a second scheduling plan is determined as the scheduling plan for the target job, wherein the second scheduling plan is: sending the detection result for the target machine to the administrator, and performing fault detection on a third machine other than the target machine in the queue where the target machine is located, and allocating the target job to the third machine whose detection result is normal; wherein, in the process of performing fault detection on multiple third machines other than the target machine in the queue where the target machine is located, each third machine performs at most one fault detection, to ensure that the detection process is efficient and does not consume excessive resources.

[0114] Through the above embodiments, it is ensured that when the target machine may fail or its performance degrades, the system can promptly detect the abnormality and initiate fault detection, adjust the scheduling plan according to the detection results, avoid the interruption of the target job due to the failure of the target machine, and reallocate the target job to a machine in normal status to maintain service continuity and effective use of resources.

[0115] Optionally, the configuration information further includes a machine specified by the user end for executing the target job, and the performing fault detection on a third machine other than the target machine in the queue where the target machine is located includes:

[0116] determining the number of the third machines, and determining a total number of fault detections based on the number of the third machines;

[0117] Perform fault detection on each of the third machines in turn, and update the remaining number of fault detections after each fault detection is completed, wherein the initial value of the remaining number of fault detections is the value of the total number of fault detections.

[0118] Specifically, Figure 2As shown, in one embodiment, the configuration information also includes the machine specified by the user end for executing the target job. In the process of performing fault detection on multiple third machines other than the target machine in the queue where the target machine is located, first, the number of third machines needs to be determined, and the total number of fault detections is determined according to the number of third machines, and each third machine is sequentially tested for faults. Each third machine is tested at most once in this process, and may not be detected because the previous third machine is detected as normal. After each fault detection is completed, the remaining number of fault detections in the fault detection process is updated. The remaining number of fault detections indicates how many times the current number of fault detections is still required, and the initial value of the remaining number of fault detections is the value of the total number of fault detections.

[0119] As an example, suppose we have a queue containing 4 machines, one of which is the target machine on which we are executing the target task, and the other three are third machines. Now, the target machine fails, and suppose our configuration information contains a "fault detection interval" (for example, detection every 30 minutes) and a "fault detection number threshold" (for example, the maximum number of fault detections we allow is 2 times).

[0120] Configuration information: Fault detection interval: 30 minutes, Fault detection threshold: 2 times

[0121] Fault detection process: Determine the number of third machines: Based on the configuration information, we know that in addition to the target machine, there are 3 "third machines" in the queue.

[0122] Determine the total number of fault detections: Based on the number of the third machines (3), determine the total number of fault detections to be 3 times.

[0123] Sequential fault detection: Fault detection is performed on the three third machines in a certain order (such as a for loop in computer programming, starting from the first third machine and going to the third third machine in sequence).

[0124] After the first test: If the first third machine is tested, the remaining number of fault tests is 2;

[0125] After the second test: After testing the second machine, the remaining number of fault tests is 1;

[0126] After the third test: After the third machine is tested, the remaining fault detection times are 0.

[0127] Update the remaining fault detection times: Each time a fault detection is completed, the system automatically updates the remaining fault detection times.

[0128] Suppose we start testing from the first machine:

[0129] After the first detection: Assuming that the first third machine is in normal condition, the remaining fault detection times are subtracted by one, and the remaining fault detection times are 2. Since the first third machine has been determined to be a normal machine, in order to ensure that the detection process is efficient and does not consume excessive resources, there is no need to detect the remaining two third machines. At this time, the fault detection process of the remaining two third machines is directly ended, and the remaining fault detection times are no longer updated.

[0130] When the target machine is still faulty after the number of fault detections reaches the fault detection threshold, other machines in the queue are automatically checked for faults in turn. Each machine is checked at most once until one machine is determined to be normal or all machines are faulty. Then the fault detection ends. This allows the machine fault detection process to be automatically completed after the target job becomes abnormal, reducing administrator intervention and improving system stability and automation.

[0131] Optionally, the method further comprises:

[0132] In the process of sequentially performing fault detection on each of the third machines, when the detection result of the third machine currently undergoing fault detection is determined to be normal, it indicates that a third machine with a normal detection result is determined, and the fault detection of the remaining third machines is stopped;

[0133] Cleaning up the data generated during the execution of the target job, reassigning the target job to a third machine with a normal detection result, and updating the machine specified by the user end for executing the target job in the configuration information of the target job in the database according to the third machine with a normal detection result;

[0134] When the remaining fault detection times is 0, and the detection results of the third machine are all abnormal, the detection results for the third machine are sent to the administrator.

[0135] Specifically, Figure 2 As shown, in one embodiment, when the detection result of the third machine is normal, once it is detected that the state of the "third machine" currently being detected is normal, the job scheduling system will immediately stop fault detection of all remaining "third machines". In other words, if a machine in normal state has been found, there is no need to continue to detect other machines. Further, the job scheduling system will clean up the data generated during the execution of the target job, and then reallocate the target job to the "third machine" in normal state. The target job will continue to run on the third machine in good state, thereby ensuring the continuity of the target job and the consistency of the data.

[0136] At the same time, the job scheduling system will update the configuration information of the target job in the database, and replace the original abnormal target machine with the normal third machine that is currently executing the target job.

[0137] When the remaining fault detection times are 0 and the detection results of all third machines are abnormal, a detection result notification needs to be issued: informing the administrator that the fault detection times of all "third machines" have been exhausted and the current status of all third machines is abnormal.

[0138] This prompts administrators to intervene to maintain or repair existing machines. Administrators check the status of these machines and decide whether additional actions are needed, such as replacing machines, performing hardware maintenance, or performing system updates.

[0139] Through the above embodiments, the stable operation of the job is ensured. In the automated detection process, the administrator intervention is reduced while the continuity of the abnormal target job is ensured. When a problem occurs, the relevant personnel can be effectively responded to and notified to intervene in the management. This improves the reliability and efficiency of the abnormal job processing system and also ensures that the resource allocation requirements specified by the user are met.

[0140] Optionally, the method further comprises:

[0141] When the target job starts to be executed, assigning a job identification number to the target job;

[0142] Associating the job identification number with the configuration information of the target job in the database;

[0143] The acquiring the configuration information of the target job from the database includes: acquiring the configuration information of the target job based on the job identification number of the target job in which the exception occurs.

[0144] When the target job starts executing, the system assigns it a unique job ID. This job ID is usually a number or a string that uniquely identifies this specific job execution instance. This process helps the system track and manage multiple jobs running in parallel.

[0145] Specifically, Figure 2As shown, in one embodiment, the job scheduling system can assign a job identification number (i.e., JOB ID) to each target job after each target job starts to execute. After the job identification number is assigned, the job scheduling system will associate the job identification number with the configuration information of the target job in the database. In addition to the fault detection number threshold, fault detection time interval, machine, and queue, the configuration information also includes detailed parameters and information required to execute the job, such as the name of the job, execution time, required resources, execution command, and any specific environment or condition settings. Through this association, the job scheduling system can quickly find and access this configuration information when needed.

[0146] When the system needs to obtain the configuration information of the target job, it obtains the job identification number of the target job that has an exception. The job scheduling system uses this identification number as a query parameter to search for related configuration information. Once a matching job identification number is found, the system can obtain the corresponding configuration information, which is a crucial step for analyzing job exceptions, debugging problems, or restoring normal execution of jobs.

[0147] The above embodiments help the job scheduling system to efficiently manage and monitor the execution of target jobs, especially when dealing with abnormal situations, and can quickly locate the problem, thereby improving the overall system stability and response efficiency.

[0148] Below, the job scheduling system takes OpenLava as an example. Figure 2 The various embodiments of the present application are fully described.

[0149] First, before submitting the target job, the user needs to configure the relevant information about the target job, including but not limited to the submission queue (queue), submission machine (machine), submission command (shell), fault detection number threshold (number), fault detection time interval (minutes), etc., and store this information in the database as configuration parameters to facilitate subsequent retrieval of relevant information of the target job.

[0150] Further, use the bsub command to submit the target job to OpenLava.

[0151] Furthermore, if the target machine for execution is specified when submitting the target job, the job is executed on the target machine; if no specific machine is specified when submitting the job, wait for OpenLava to allocate a machine from multiple queues.

[0152] Further, after waiting for OpenLava to allocate a target machine to execute the target job, the target job starts to execute. At this time, OpenLava allocates a job identification number (JOB ID) for the job and stores this JOB ID in the corresponding configuration parameters of the database for association.

[0153] Furthermore, when the target job is executed, it is necessary to determine whether the target job can be executed normally based on the status of the target machine;

[0154] Furthermore, when the target machine is in a normal state during the execution of the job, and no node failure or downtime occurs, the job is executed normally until completion, and no abnormal state occurs.

[0155] Furthermore, when the target machine does not experience node failure or downtime during job execution, the job will become a zombie and enter the exception queue.

[0156] Furthermore, the JOB IDs of all zombie jobs in OpenLava are obtained regularly through the bjobs command to provide a basis for database query for the next step of processing these abnormal jobs.

[0157] Furthermore, through the above steps, the JOB ID of the zombie job is obtained, and the configuration parameters of the target job saved in the database are queried through the JOB ID, including but not limited to the JOB ID, submission queue, submission machine, submission command, number of retries, retry interval, etc.

[0158] Furthermore, through the above steps, the fault detection number threshold ${number} and the fault detection time interval ${minutes} (in minutes) of the job can be obtained to enter the fault detection.

[0159] Furthermore, if the fault detection is performed once every ${minutes} minutes, and the total number of detections is ${number}, if it does not continue to reach the maximum number of detections, it is determined that the target machine is normal and no fault occurs.

[0160] Furthermore, when the target machine does not fail during fault detection, the bkill command is used to clean up the zombie job, obtain the submission queue, submission machine, submission command and other information of the job, and resubmit the target job based on this information until the target job is executed normally to the end.

[0161] Furthermore, when the number of fault detections reaches the fault detection threshold and the target machine is still faulty, the administrator of OpenLava is notified via SMS or email to manually intervene in the target machine in a timely manner.

[0162] Further, after notifying the OpenLava administrator, the program automatically enters failover. Through the configuration parameters of the target job, the queue where the target job is located can be obtained, and then the information of other target machines (i.e., the third machine) in this queue can be obtained through the bhosts command combined with the name of the queue, and the information of other machines in the queue (the machine name of the third machine and the total number of machines n) can be recorded.

[0163] Furthermore, a cyclic fault detection is performed on other machines in the queue, and each machine is detected only once, because detecting the same machine multiple times will also waste time and may delay the running of the OpenLava job.

[0164] Furthermore, if other machines in the queue are performing fault detection, as long as one machine is normal, the fault detection logic of the remaining machines will be stopped and the job will be directly switched to this machine. In addition, this command needs to specify the machine that has passed the current fault detection to execute bsub submission. At the same time, the data generated during the execution of the original target job is cleaned up, and the JOB ID and submission machine information in the database are updated.

[0165] Further, the target job is executed on the machine detected as normal by the fault detection until it is completed.

[0166] Furthermore, when other machines in the queue perform cyclic fault detection, each machine is detected only once. As long as one machine fails, the total number of machines n is reduced by 1.

[0167] Further, it is determined whether the remaining number of fault detections, that is, n, is greater than 0.

[0168] Furthermore, when n>0, it indicates that there are still machines that have not been fault-checked, and the fault-check of the third machine is continued.

[0169] Furthermore, when n=0, that is, all machines in the queue have failed the fault detection, this means that all machines in the queue have failed, and it is necessary to send emails or text messages to notify the OpenLava administrator and related operators in time and manually intervene to handle the problem.

[0170] At this point, the entire technical solution of this application is completed.

[0171] Optionally, machine learning and artificial intelligence technologies can also be applied to the abnormal operation processing system to significantly improve the intelligence and adaptability of the system. The expansion and design can be mainly carried out from the following aspects:

[0172] Anomaly detection based on historical data: Use time series analysis, statistical methods, or deep learning models (such as RNN, LSTM) to identify anomalies in the execution of target jobs. By analyzing patterns and trends in historical data, anomalies that deviate from normal behavior can be identified, thereby promptly detecting faults or potential problems.

[0173] Real-time monitoring and abnormal warning: Integrate real-time monitoring systems, such as using tools such as Prometheus and Grafana, to continuously monitor system performance indicators (CPU usage, memory usage, I / O load, etc.), and identify possible abnormal growth trends based on machine learning models to provide early warnings.

[0174] Learning-based resource scheduling: Using methods such as reinforcement learning (RL) or genetic algorithms, dynamically adjust resource allocation strategies based on historical data and current system status to optimize the execution efficiency and resource utilization of target jobs. For example, by learning the relationship between different job types and machine resources, the system can automatically match the most appropriate resource combination to achieve optimal resource utilization.

[0175] Automatic fault recovery and resource reallocation: Based on machine learning models, predict and prevent resource bottlenecks, and automatically adjust job load distribution to avoid system crashes caused by single point failures. When a failure occurs, quickly identify the affected jobs and resources, and automatically reallocate resources through optimization algorithms.

[0176] Intelligent priority decision-making: Through deep learning models, the system analyzes multi-dimensional factors such as the urgency, priority, and resource requirements of jobs, and automatically adjusts the execution order of jobs to ensure that key tasks are executed first.

[0177] Dynamic job scheduling: Use intelligent algorithms (such as genetic algorithms, ant colony algorithms, etc.) to dynamically allocate jobs among multiple machines, taking into account multiple factors such as machine load conditions and historical performance, to achieve efficient execution of jobs and balanced utilization of resources.

[0178] Fault prediction and prevention: Analyze historical fault data through machine learning models, predict possible fault types and causes, take preventive measures in advance, and reduce the possibility of faults.

[0179] Intelligent recovery strategy: When a failure occurs, fault recovery strategies such as data retry, task sharding, and intelligent rescheduling are automatically executed to restore services at the lowest cost.

[0180] Integration with monitoring systems: Integrate with existing monitoring and alarm systems to achieve real-time status updates and rapid response to exceptions. Share data through interfaces such as APIs or message queues to improve the overall monitoring capabilities of the system.

[0181] Integration with resource management systems: Integrate with resource management systems (such as Kubernetes, Docker Swarm, etc.) to dynamically manage the relationship between jobs and resources, and implement functions such as automatic job deployment, scaling, and resource optimization.

[0182] Security policy learning: Use artificial intelligence technology to analyze system behavior and network traffic, learn normal security behavior patterns, and identify and prevent potential security threats.

[0183] Data privacy protection: When processing sensitive data, machine learning models are used for data desensitization and privacy protection technologies (such as differential privacy and homomorphic encryption) to ensure data privacy and security.

[0184] Based on the same inventive concept, another embodiment of the present application provides an abnormal job processing system, the system comprising a user terminal and a job scheduling system:

[0185] The user terminal is used to send a target job to the job scheduling system, and to set configuration information according to the target job, and store the configuration information in a database;

[0186] The job scheduling system is used to receive an execution instruction carrying the target job issued by a user terminal, and allocate a target machine to the target job according to the execution instruction, wherein the target machine is used to execute the target job;

[0187] The job scheduling system is used to detect whether an abnormality occurs in the execution of the target job based on the state of the target machine after the target job starts to be executed;

[0188] The job scheduling system is used to add the target job to an exception queue when detecting that an exception occurs in the execution of the target job, obtain configuration information of the target job from the database, and perform at least one fault detection on the target machine based on the configuration information, wherein the configuration information is pre-set by the user end for the target job and stored in the database, and the configuration information at least includes a fault detection time interval and a fault detection number threshold of the fault detection;

[0189] The job scheduling system is used to determine a scheduling scheme for the target job according to the detection result of the at least one fault detection, and re-execute the target job based on the scheduling scheme.

[0190] Figure 3 is a structural diagram of an abnormal operation processing system provided by an embodiment of the present application, such as Figure 3 shown.

[0191] The abnormal job processing system includes a user end and a job scheduling system. The job scheduling system is configured with multiple queues (such as Figure 3 Queue 1, queue 2...queue n), each queue corresponds to multiple machines.

[0192] Figure 4 is a schematic diagram of the structure of an abnormal operation processing device provided by an embodiment of the present application, such as Figure 4 shown.

[0193] Based on the same inventive concept, another embodiment of the present application further provides an abnormal operation processing device, which is applied to a job scheduling system, and the device includes:

[0194] The receiving module 11 is used to receive an execution instruction carrying the target job sent by the user end, and allocate a target machine to the target job according to the execution instruction, and the target machine is used to execute the target job;

[0195] A machine detection module 12, configured to detect whether an abnormality occurs in the execution of the target job based on the state of the target machine after the target job starts to be executed;

[0196] A fault detection module 13 is used to add the target job to an abnormality queue when an abnormality is detected in the execution of the target job, obtain configuration information of the target job from a database, and perform at least one fault detection on the target machine according to the configuration information, wherein the configuration information is pre-set by the user end for the target job and stored in the database, and the configuration information at least includes a fault detection time interval and a fault detection number threshold of the fault detection;

[0197] The scheduling scheme determining module 14 is used to determine the scheduling scheme of the target job according to the detection result of the at least one fault detection, and re-execute the target job based on the scheduling scheme.

[0198] Optionally, the job scheduling system is configured with multiple queues, a single queue corresponds to multiple machines, and the receiving module 11 includes:

[0199] a first target machine determining unit, configured to determine the first machine specified by the user end for executing the target job as the target machine if the execution instruction includes the first machine specified by the user end for executing the target job;

[0200] A second target machine determining unit, configured to determine a target machine for executing a target job from a plurality of second machines corresponding to the queue specified by the user end, when the execution instruction includes the queue specified by the user end;

[0201] The third target machine determination unit is configured to determine a target machine for executing a target job from idle machines when the execution instruction does not include the queue and machine specified by the user.

[0202] Optionally, the fault detection module 13 includes:

[0203] a fault detection unit, configured to perform at least one fault detection on the target machine according to the fault detection time interval and within a range not exceeding the fault detection number threshold, and to update the number of fault detections of the target machine after each fault detection is completed;

[0204] The scheduling scheme determination module 14 includes:

[0205] a first scheduling scheme determining unit, configured to determine, when the number of fault detections does not exceed the fault detection number threshold and the target machine is detected to be normal, that the detection result of the target machine is normal, and determine a first scheduling scheme as the scheduling scheme for the target job, wherein the first scheduling scheme is: cleaning up data generated during the execution of the target job and reallocating the target job to the target machine;

[0206] a second scheduling scheme determining unit, configured to determine, when the number of fault detections exceeds the fault detection number threshold, that the detection result of the target machine is abnormal, and determine a second scheduling scheme as the scheduling scheme for the target job, wherein the second scheduling scheme is: sending the detection result for the target machine to an administrator, performing fault detection on a third machine other than the target machine in the queue where the target machine is located, and allocating the target job to the third machine whose detection result is normal;

[0207] In the process of performing fault detection on the third machines other than the target machine in the queue where the target machine is located, each third machine is subjected to at most one fault detection.

[0208] Optionally, the configuration information further includes a machine specified by the user end for executing the target job, and the second scheduling scheme determining unit includes:

[0209] a fault detection total number determining unit, configured to determine the number of the third machines, and determine the fault detection total number based on the number of the third machines;

[0210] An updating unit is used to perform fault detection on each of the third machines in turn, and update the remaining number of fault detections after each fault detection is completed, wherein the initial value of the remaining number of fault detections is the value of the total number of fault detections.

[0211] Optionally, the device further comprises:

[0212] a normal third machine determining unit, configured to, in the process of sequentially performing fault detection on each of the third machines, indicate that a third machine with a normal detection result is determined when the detection result of the third machine currently performing fault detection is normal, and stop fault detection on the remaining third machines;

[0213] a reallocation unit, configured to clean up data generated during the execution of the target job, reallocate the target job to a third machine whose detection result is normal, and update the machine specified by the user end for executing the target job in the configuration information of the target job in the database according to the third machine whose detection result is normal;

[0214] The reporting unit is used to send the detection result of the third machine to the administrator when the remaining fault detection times is 0 and the detection results of the third machine are all abnormal.

[0215] Optionally, the device further comprises:

[0216] A job identification number allocation unit, used for allocating a job identification number to the target job when the target job starts to be executed;

[0217] an associating unit, configured to associate the job identification number with the configuration information of the target job in the database;

[0218] The fault detection module 13 further includes: a configuration information acquisition unit, which is used to acquire configuration information of the target job based on the job identification number of the target job with the abnormality.

[0219] Based on the same inventive concept, another embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the abnormal operation processing method as described in any of the above embodiments.

[0220] Figure 5 is a schematic diagram of an electronic device provided by an embodiment of the present application. Figure 5 As shown, the electronic device 500 includes: a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus communication. A computer program is stored in the memory 510. The computer program can be run on the processor 520 to implement the steps in the abnormal operation processing method disclosed in the embodiment of the present application.

[0221] Based on the same inventive concept, another embodiment of the present application further provides a computer program product, including a computer program, which is executed by a processor as the abnormal operation processing method described in any of the above embodiments.

[0222] Based on the same inventive concept, another embodiment of the present application further provides a computer-readable medium on which a computer program is stored, wherein when the program is executed by a processor, the abnormal operation processing method as described in any of the above embodiments is implemented.

[0223] As for the device, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0224] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0225] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0226] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0227] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0228] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0229] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0230] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0231] The above is a detailed introduction to an abnormal operation processing method, system, device, equipment and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for handling abnormal operations, characterized in that: Applied to a job scheduling system, the method comprises: receiving an execution instruction carrying the target job from a user terminal, and assigning a target machine to the target job according to the execution instruction, wherein the target machine is used to execute the target job; After the target job starts to be executed, based on the state of the target machine, detecting whether the execution of the target job is abnormal; When an exception is detected in the execution of the target job, the target job is added to an exception queue, configuration information of the target job is obtained from a database, and at least one fault detection is performed on the target machine according to the configuration information, wherein the configuration information is pre-set by the user end for the target job and stored in the database, and the configuration information at least includes a fault detection time interval and a fault detection number threshold of the fault detection; A scheduling scheme for the target job is determined according to a detection result of the at least one fault detection, and the target job is re-executed based on the scheduling scheme.

2. The abnormal operation processing method according to claim 1, characterized in that: The job scheduling system is configured with multiple queues, and a single queue corresponds to multiple machines. The target machine is assigned to the target job according to the execution instruction, including: In a case where the execution instruction includes a first machine specified by the user end for executing a target job, determining the first machine as the target machine; In a case where the execution instruction includes the queue specified by the user end, determining a target machine for executing the target job from a plurality of second machines corresponding to the queue specified by the user end; In a case where the execution instruction does not include the queue and machine specified by the user, a target machine for executing the target job is determined from idle machines.

3. The abnormal operation processing method according to claim 2, characterized in that: The performing at least one fault detection on the target machine according to the configuration information includes: performing at least one fault detection on the target machine according to the fault detection time interval and within a range not exceeding the fault detection number threshold, and updating the number of fault detections of the target machine after each fault detection is completed; The step of determining a scheduling scheme for the target job according to the detection result of the at least one fault detection comprises: When the number of fault detections does not exceed the fault detection number threshold and the target machine is detected to be normal, determining that the detection result of the target machine is normal, and determining a first scheduling scheme as the scheduling scheme for the target job, wherein the first scheduling scheme is: cleaning up data generated during the execution of the target job, and reallocating the target job to the target machine; In the case where the number of fault detections exceeds the fault detection number threshold, determining that the detection result of the target machine is abnormal, and determining a second scheduling scheme as the scheduling scheme for the target job, the second scheduling scheme is: sending the detection result for the target machine to an administrator, performing fault detection on a third machine other than the target machine in the queue where the target machine is located, and allocating the target job to the third machine whose detection result is normal; In the process of performing fault detection on the third machines other than the target machine in the queue where the target machine is located, each third machine is subjected to at most one fault detection.

4. The abnormal operation processing method according to claim 3, characterized in that: The configuration information also includes a machine specified by the user end for executing the target job, and the performing fault detection on a third machine other than the target machine in the queue where the target machine is located includes: determining the number of the third machines, and determining a total number of fault detections based on the number of the third machines; Perform fault detection on each of the third machines in turn, and update the remaining number of fault detections after each fault detection is completed, wherein the initial value of the remaining number of fault detections is the value of the total number of fault detections.

5. The abnormal operation processing method according to claim 4, characterized in that: The method further comprises: In the process of sequentially performing fault detection on each of the third machines, when the detection result of the third machine currently undergoing fault detection is determined to be normal, it indicates that a third machine with a normal detection result is determined, and the fault detection of the remaining third machines is stopped; Cleaning up the data generated during the execution of the target job, reassigning the target job to a third machine with a normal detection result, and updating the machine specified by the user end for executing the target job in the configuration information of the target job in the database according to the third machine with a normal detection result; When the remaining fault detection times is 0, and the detection results of the third machine are all abnormal, the detection results for the third machine are sent to the administrator.

6. The abnormal operation processing method according to any one of claims 1 to 5, characterized in that: The method further comprises: When the target job starts to be executed, assigning a job identification number to the target job; Associating the job identification number with the configuration information of the target job in the database; The acquiring the configuration information of the target job from the database includes: acquiring the configuration information of the target job based on the job identification number of the target job in which the exception occurs.

7. An abnormal operation processing system, characterized in that: The system includes a user terminal and a job scheduling system: The user terminal is used to send a target job to the job scheduling system, and to set configuration information according to the target job and store the configuration information in a database; The job scheduling system is used to receive an execution instruction carrying the target job issued by a user terminal, and allocate a target machine to the target job according to the execution instruction, wherein the target machine is used to execute the target job; The job scheduling system is used to detect whether an abnormality occurs in the execution of the target job based on the state of the target machine after the target job starts to be executed; The job scheduling system is used to add the target job to an exception queue when detecting that an exception occurs in the execution of the target job, obtain configuration information of the target job from the database, and perform at least one fault detection on the target machine based on the configuration information, wherein the configuration information is pre-set by the user end for the target job and stored in the database, and the configuration information at least includes a fault detection time interval and a fault detection number threshold of the fault detection; The job scheduling system is used to determine a scheduling scheme for the target job according to the detection result of the at least one fault detection, and re-execute the target job based on the scheduling scheme.

8. An abnormal operation processing device, characterized in that: Applied to a job scheduling system, the device comprises: A receiving module, configured to receive an execution instruction carrying the target job from a user terminal, and to allocate a target machine to the target job according to the execution instruction, wherein the target machine is used to execute the target job; A machine detection module, configured to detect whether an abnormality occurs in the execution of the target job based on the state of the target machine after the target job starts to be executed; a fault detection module, configured to, when detecting that an exception occurs in the execution of the target job, add the target job to an exception queue, obtain configuration information of the target job from a database, and perform at least one fault detection on the target machine according to the configuration information, wherein the configuration information is pre-set by the user end for the target job and stored in the database, and the configuration information at least includes a fault detection time interval and a fault detection number threshold of the fault detection; The scheduling scheme determination module is used to determine the scheduling scheme of the target job according to the detection result of the at least one fault detection, and re-execute the target job based on the scheduling scheme.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the abnormal operation processing method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the abnormal operation processing method as described in any one of claims 1-6 is implemented.

Images