Task exception processing method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a task exception processing method and system, electronic equipment and a storage medium, and the method comprises the steps: sending an exception notification to a process corresponding to any equipment under the condition that an interrupt signal sent by any equipment is received, so that the process closes file descriptors of all started equipment one by one; taking the device which closes the file descriptor at present as the current device, entering a resource release process of the current device, and detecting the state of the current device; and when it is detected that the current device is in the busy state, generating a confirmation response based on a synchronization task completion condition of the current device to trigger the current device to release the synchronization waiting state. According to the invention, the host issues the confirmation response to relieve the synchronous waiting state of the equipment, an exception handling mechanism based on a software level is realized, and a reset operation on a hardware level does not need to be carried out, so that the risk caused by the hardware operation can be reduced, and the exception handling time is shortened at the same time.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a task exception processing method, system, electronic device and storage medium. Background Art

[0002] During the model training process, the application will continuously send task requests to the computing device and wait for the computing device to complete these tasks and return the results. However, various abnormal situations may occur during the model training process, causing the application to crash, such as insufficient memory, network interruption, disk damage, etc. When these abnormal situations occur, the application will be forced to terminate. Since the application is exited abnormally, some synchronization tasks sent by the application may not be completed, which will cause the computing device to be in a synchronization waiting state. This waiting state will not only cause the current task to fail to complete smoothly, but also occupy the valuable resources of the device, thereby affecting the normal execution of other tasks and the efficiency of the entire model training process.

[0003] In order to deal with the above task exceptions, a reset method is currently usually used to restore the normal operation of the device. However, the reset method involves a hardware-level reset operation, which is not only time-consuming, but also affected by hardware stability, increasing uncertainty and risk in the recovery process. Summary of the invention

[0004] The present invention provides a task exception processing method, system, electronic device and storage medium, which are used to solve the defects of task exception processing in related technologies, such as long time consumption and poor stability.

[0005] The present invention provides a task exception processing method, which is applied to a host, and the host is used to record synchronization tasks executed by each device and the completion condition of each synchronization task. The method includes: When receiving an interrupt signal sent by any device, sending an exception notification to the process corresponding to the any device, so that the process closes the file descriptors of all started devices one by one, wherein the interrupt signal is sent by any device when detecting an abnormality in the synchronization task; The device whose file descriptor is currently closed is used as the current device, a resource release process of the current device is entered, and the status of the current device is detected; When it is detected that the current device is in a busy state, a corresponding confirmation response is generated based on the synchronization task completion condition of the current device, and the confirmation response is sent to the current device, and the confirmation response is used to trigger the current device to release the synchronization waiting state.

[0006] According to a task exception processing method provided by the present invention, when the current device is detected to be in a busy state, a corresponding confirmation response is generated based on the synchronization task completion condition of the current device, including: When the current device is detected to be busy, it enters abnormal recovery mode; In the abnormal recovery mode, the current device is notified to skip the unfinished synchronization task, and a corresponding confirmation response is generated based on the synchronization task completion condition of the current device.

[0007] According to a task exception handling method provided by the present invention, one or more processes are running on the host, each process is used to start one or more devices, and when any process receives the exception notification, it executes error handling and then exits, and closes the file descriptors of all devices started by any process one by one when exiting.

[0008] According to a method for handling task exceptions provided by the present invention, the detecting of the state of the current device further comprises: When it is detected that the current device is in an idle state, a resource release process is executed.

[0009] According to a task exception processing method provided by the present invention, the sending of the confirmation response to the current device further includes: Detecting the status of the current device; When it is detected that the current device is in a busy state and the duration exceeds a preset time, an access request of the current device is blocked, and based on system parameters or user input, the current device is triggered to reset.

[0010] The present invention also provides a method for handling task exceptions, which is applied to a device and includes: When an exception is detected in a synchronization task, an interrupt signal is sent to the host, so that the host sends an exception notification to the process corresponding to the device, the exception notification is used to instruct the process to close the file descriptors of all started devices one by one, and the host is used to record the synchronization tasks executed by each device and the completion condition of each synchronization task; When the file descriptor of the device is closed, a resource release process of the device is entered, so that the host detects the status of the device; A confirmation response is received, and the synchronization waiting state is released based on the confirmation response. The confirmation response is generated and sent by the host based on the synchronization task completion condition of the device when the host detects that the device is in a busy state.

[0011] The present invention also provides a task exception processing device, which is applied to a host, and the host is used to record the synchronization tasks executed by each device and the completion condition of each synchronization task, and the device includes: An exception notification unit is used to send an exception notification to a process corresponding to any device when receiving an interrupt signal sent by any device, so that the process closes the file descriptors of all started devices one by one, and the interrupt signal is sent by any device when detecting an abnormality in the synchronization task; A state detection unit, used for taking the device of the currently closed file descriptor as the current device, entering the resource release process of the current device, and detecting the state of the current device; A response generation unit is used to generate a corresponding confirmation response based on the synchronization task completion condition of the current device when detecting that the current device is in a busy state, and send the confirmation response to the current device, wherein the confirmation response is used to trigger the current device to release the synchronization waiting state.

[0012] The present invention also provides a task exception processing device, which is applied to a device and includes: An interrupt sending unit, configured to send an interrupt signal to a host when an abnormality of a synchronization task is detected, so that the host sends an abnormality notification to a process corresponding to the device, the abnormality notification being used to instruct the process to close the file descriptors of all started devices one by one, and the host being used to record the synchronization tasks executed by each device and the completion condition of each synchronization task; A process entry unit, used for entering a resource release process of the device when the file descriptor of the device is closed, so that the host can detect the status of the device; The state release unit is used to receive a confirmation response and release the synchronization waiting state based on the confirmation response. The confirmation response is generated and sent by the host based on the synchronization task completion condition of the device when the host detects that the device is in a busy state.

[0013] The present invention also provides a task exception processing system, comprising a host and at least one device, wherein the host is used to record synchronization tasks executed by each device and completion conditions of each synchronization task; Any device is used to send an interrupt signal to the host when detecting an abnormality in the synchronization task; The host is used to send an exception notification to the process corresponding to any one of the devices when receiving the interrupt signal, so that the process closes the file descriptors of all the started devices one by one; The host is also used to take the device that currently closes the file descriptor as the current device, enter the resource release process of the current device, and detect the status of the current device. When it is detected that the current device is in a busy state, based on the synchronization task completion condition of the current device, a corresponding confirmation response is generated and sent to the current device. The confirmation response is used to trigger the current device to release the synchronization waiting state.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, any of the above-mentioned task exception handling methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the task exception handling method described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned task exception handling methods.

[0017] The task exception processing method, system, electronic device and storage medium provided by the present invention can send an interrupt signal to the host when any device detects a synchronous task exception, so that the host can respond quickly and send an exception notification to the process corresponding to the abnormal device, thereby enabling the process to close the file descriptors of all devices started by it one by one. This immediate response mechanism ensures that the abnormal situation can be handled in time, thereby reducing the negative impact that may be caused by the task exception. When closing the file descriptor of each device, the resource release process of the device will be entered, so that the status of the device can be detected by the host. When the host detects that the device is in a busy state, it can generate and send a corresponding confirmation response based on the synchronous task completion condition of the device. After receiving the response, the device can release the synchronous waiting state, realizing an exception handling mechanism based on the software level, that is, releasing the synchronous waiting state of the device by sending a confirmation response by the host. This mechanism does not involve a reset operation at the hardware level, which can reduce the risks that may be caused by hardware operations, while shortening the time of exception handling and improving processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a schematic diagram of the structure of the cluster provided by the present invention; Figure 2 This is one of the flowcharts of the task exception handling method provided by the present invention; Figure 3 This is the second flow chart of the task exception handling method provided by the present invention; Figure 4 It is a flowchart of a task exception handling method for a cluster scenario provided by the present invention; Figure 5 It is one of the structural schematic diagrams of the task exception handling device provided by the present invention; Figure 6 This is the second structural diagram of the task exception handling device provided by the present invention; Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] In order to facilitate understanding of the technical solutions and embodiments provided by the present invention, the concepts of cluster, computing node, Pod, model training task, etc. are described in detail below: Cluster refers to connecting multiple computing resources (such as computing nodes) through a network to form a unified computing system. Cluster technology allows these computing resources to work together to handle larger and more complex computing tasks.

[0022] A computing node refers to a single computing resource in a cluster, which is responsible for executing computing tasks in the cluster and communicating with other nodes to share data. A computing node can include a host and multiple computing devices. Here, the host refers to the core device of the computing node, which provides basic computing power and storage space. For example, the host can be a CPU (Central Processing Unit). A computing device refers to a hardware device that can efficiently perform computing tasks, such as a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), a TPU (Tensor Processing Unit), etc.

[0023] Pod is the smallest deployable unit in the cluster. The cluster is responsible for managing and scheduling the deployment and operation of Pods to ensure that they can evenly distribute resources on multiple computing nodes. Pod is a relatively independent and isolated environment constructed on a node using virtualization technology. Each Pod is deployed on a specific node and uses the resources of the node to train the model.

[0024] Model training refers to the process of using specific algorithms and data sets to train machine learning or deep learning models. In this process, the model gradually optimizes its parameters through iterative learning to better fit the training data and have generalization capabilities.

[0025] Figure 1 It is a schematic diagram of the structure of the cluster provided by the present invention, such as Figure 1As shown in the figure, the model training task can be executed on the cluster to utilize the computing power of the cluster to accelerate the training process. The cluster includes multiple Pods, which can process model training tasks in parallel, thereby significantly shortening the training time of the model. A complex model training task is usually split into multiple training subtasks, each of which can be executed on one or more Pods. Each Pod is deployed on a specific computing node and uses the resources on the node (such as CPU, GPU, etc.) to complete the training subtask assigned to it. Pod provides an isolated operating environment for the training subtask and can communicate with other Pods in the cluster to complete the entire training task together. Within each Pod, the training subtask (or part of the training subtask) will be further split into multiple synchronization tasks and distributed to multiple computing devices (such as GPUs) for execution. These GPUs can also communicate with each other. Here, synchronization tasks refer to tasks that are split and distributed to multiple computing devices for execution during the model training process. These tasks need to maintain a certain degree of synchronization during execution, such as data synchronization, parameter synchronization, calculation synchronization, gradient synchronization, etc.

[0026] During the model training process, the application running on the computing node will send synchronization tasks to the computing device and wait for the device to complete these tasks and return the results. If the application exits abnormally before the device completes the task, the device may continue to wait for further instructions or confirmation information from the application, thus falling into a synchronization waiting deadlock. Here, the application refers to the software or program responsible for executing model training tasks and managing the issuance of synchronization tasks. It can be an instance of a machine learning framework (such as TensorFlow, PyTorch, etc.) or a custom training script.

[0027] During the model training process, various abnormal situations may occur, causing the application to crash, such as insufficient memory, network interruption, disk damage, etc. Once these abnormal situations occur, they often cause the application to exit abnormally, which in turn triggers a series of chain reactions. In particular, when the application fails to complete all the issued synchronization tasks before the abnormal exit, the computing device may fall into a synchronization waiting state, which will not only hinder the smooth completion of the current task, but also have an adverse impact on the normal execution of subsequent tasks.

[0028] In response to this problem, in the relevant technology, in scenarios with a small number of computing devices, the exception handling strategy adopted is usually a reset method, that is, resetting the device to restore the normal operation of the hardware. Although this method can solve the hardware-level exception problem to a certain extent, it is inadequate in cluster scenarios. In cluster scenarios, there are often complex synchronization relationships between Pods. Once a Pod has an exception, other Pods will also be affected due to synchronization dependencies.

[0029] However, even in scenarios with a small number of computing devices, the reset method has certain drawbacks. It involves hardware-level reset operations, which are not only time-consuming but also affected by hardware stability, thus increasing uncertainty and risk in the recovery process. When performing a reset operation at the cluster level, these problems become more prominent due to the complex synchronization relationship between Pods, resulting in adverse effects on the performance and reliability of the entire cluster.

[0030] In this regard, the present invention proposes a task exception handling method, which detects the status of each device through the host, and when any device is detected to be in a busy state (i.e., the synchronization task is not completed), instead of the application being forced to terminate, sends a confirmation response to the device to release the synchronization waiting state of the device, thereby completing the execution of the synchronization task on the device and avoiding affecting the normal execution of other tasks. This is an exception handling mechanism based on the software level, which does not involve the reset operation at the hardware level, has better stability, and takes less time, thereby overcoming the above-mentioned defects.

[0031] Figure 2 is one of the flowcharts of the task exception handling method provided by the present invention, such as Figure 2 As shown, the method is applied to a host, and the host is used to record the synchronization tasks executed by each device and the completion condition of each synchronization task. The method includes: Step 210, when receiving an interrupt signal sent by any device, sending an exception notification to the process corresponding to the any device, so that the process closes the file descriptors of all started devices one by one. The interrupt signal is sent by any device when detecting an abnormality in the synchronization task.

[0032] It should be noted that the execution subject of the method provided in the embodiment of the present invention may be a host on a computing node, specifically a KMD (Kernel Mode Driver) running on the host, which can record the synchronization tasks submitted by the application on the node to each device for execution and the completion conditions of each synchronization task. Here, the completion condition of the synchronization task refers to the specific standard or condition that needs to be met in order to determine whether a synchronization task has been successfully completed. The completion condition of each synchronization task is related to the specific hardware implementation of the corresponding device. For example, for a certain device (such as a GPU), its synchronization task completion condition may be to wait for the actual value in a certain memory address to reach a preset expected value. In the process of executing the synchronization task, the device will continuously check whether the actual value in the memory address is equal to the preset expected value. If it is detected that the actual value is not equal to the expected value, the device will continue to maintain the synchronization waiting state and continue to monitor the value in the memory address; if it is detected that they are equal, it can be determined that the synchronization task of the device has been completed.

[0033] Specifically, each device will detect the status of the synchronization task during the execution of the synchronization task. When any device detects that an abnormality has occurred in the synchronization task, it will immediately send an interrupt signal to the host. Here, the synchronization task abnormality refers to an unexpected situation that occurs during the execution of the synchronization task, such as a task execution timeout. An interrupt signal is a notification mechanism used to indicate that a specific event or condition has occurred. In an embodiment of the present invention, an interrupt signal is sent by any one of the devices when an abnormality in the synchronization task is detected to notify the host of the abnormal situation.

[0034] It is understandable that each device can detect whether an abnormality occurs by monitoring the execution process of the synchronization task. For example, each device can set a timer to update the execution time of the synchronization task. When the synchronization task starts, the timer starts counting, and when the synchronization task is completed, the timer stops counting. If the synchronization task is not completed within the set time (i.e., timeout), the synchronization task is considered abnormal, and the device can immediately send an interrupt signal to the host.

[0035] The host can receive interrupt signals from the device through its communication interface, which is generally a PCIE (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) bus. When any device detects an abnormality in a synchronization task and sends an interrupt signal, the host receives the signal through its communication interface and triggers subsequent processing procedures, for example, sending an exception notification to all processes corresponding to the device that sent the interrupt signal to trigger each process to close the file descriptors of all devices it has started.

[0036] Here, all processes corresponding to a device refer to all processes running on the host that start the device. One or more processes can run on the host, each process can start one or more devices, and each device can also be started by multiple processes. File descriptors refer to a mechanism for identifying and accessing files or resources of a specific device. Processes can interact with devices through these file descriptors. File descriptors establish a communication channel between processes and devices, allowing processes to send commands, data, etc. to devices and receive responses or data from devices.

[0037] Step 220, taking the device that currently closes the file descriptor as the current device, entering the resource release process of the current device, and detecting the status of the current device.

[0038] Specifically, when each process corresponding to the device that detects the exception of the synchronization task receives the exception notification sent by the host, these processes will close the file descriptors of all the devices they started one by one. The closing action of each file descriptor will enter the resource release process of a device. When the file descriptor of the current device is closed, the resource release process of the current device will be entered. In this process, the resource release process of other devices will not be executed. The resource release process of other devices will not be entered and executed until the corresponding file descriptor is closed. It should be understood that closing the file descriptor is an important operation, which is used to trigger the corresponding device to enter the resource release process and ensure that the resources are subsequently released correctly. Here, the resource release process refers to a series of cleanup and resource release operations performed by each device after the process exits and closes the file descriptor.

[0039] For each device, when the file descriptor of the device is closed, the resource release process of the device will be entered, and the host will detect whether the status of the device is busy. Specifically, in a cluster environment, the host and each device on the node will establish a connection through the bus for data transmission and status monitoring; nodes and nodes are generally connected through the network for task synchronization. When the host detects whether a device is busy, it can obtain the status of the device by sending a query request or a status request message to the device. Here, the status of the device refers to the current running status of the device when executing a synchronization task, such as busy status, idle status, etc. Among them, the busy status means that the device is executing a synchronization task and has not yet been completed, or some abnormal errors have occurred in the device; the idle status means that the device has completed the synchronization task and no abnormal errors have occurred.

[0040] Exemplarily, for each device, the status of the device can be obtained by detecting the completion condition of its corresponding synchronization task. For example, if the completion condition of the synchronization task of the device is to wait for the actual value in a certain memory address to reach a preset expected value, the actual value in the memory address can be detected and compared with the preset expected value. If the actual value is equal to the expected value, it indicates that the synchronization task of the device has been completed, and it can be determined that the device is currently in an idle state; if the actual value is not equal to the expected value, it is considered that the device is currently in a busy state. In addition, the status of the device can also be obtained by detecting the fault condition of the device, resource usage, etc., and the embodiments of the present invention do not specifically limit this.

[0041] Step 230, when it is detected that the current device is in a busy state, a corresponding confirmation response is generated based on the synchronization task completion condition of the current device, and the confirmation response is sent to the current device, and the confirmation response is used to trigger the current device to release the synchronization waiting state.

[0042] Specifically, when the host detects that the current device is in a busy state, it indicates that the synchronization task of the device has not been completed. In this case, the host can generate a corresponding confirmation response and send it to the device according to the synchronization task completion condition of the device.

[0043] It is understandable that the host will record the synchronization task completion conditions of each device. When the host detects that a certain device is in a busy state, it can generate and send a corresponding confirmation response based on the synchronization task completion conditions of the device. This targeted processing method helps to improve the accuracy and efficiency of processing. For example, if the synchronization task completion condition of a certain device is to wait for the actual value in a certain memory address to reach a preset expected value, when the host detects that the device is in a busy state, it can fill the expected value into the specified memory address according to the synchronization task completion condition, so that the actual value stored in the memory address is equal to the expected value. At this time, the device will no longer maintain the synchronization waiting state, that is, the synchronization waiting state is released. Here, the expected value filled in the memory address is the confirmation response sent by the host to the device. The confirmation response refers to a message sent by the host to the device, which is used to trigger the device to release the synchronization waiting state and instruct the device to complete the current synchronization task.

[0044] The method provided by the embodiment of the present invention is that when any device detects an abnormality in a synchronization task, it sends an interrupt signal to the host, so that the host can respond quickly and send an abnormal notification to the process corresponding to the abnormal device, thereby enabling the process to close the file descriptors of all devices started by it one by one. This immediate response mechanism ensures that the abnormal situation can be handled in time, thereby reducing the negative impact that may be caused by the task abnormality. When closing the file descriptor of each device, the resource release process of the device will be entered, so that the status of the device can be detected by the host. When the host detects that the device is in a busy state, it can generate and send a corresponding confirmation response based on the synchronization task completion condition of the device. After the device receives the response, it can release the synchronization waiting state, realizing an exception handling mechanism based on the software level, that is, the synchronization waiting state of the device is released by sending a confirmation response by the host. This mechanism does not involve a reset operation at the hardware level, which can reduce the risks that may be caused by hardware operations, while shortening the time of exception handling and improving processing efficiency.

[0045] In addition, since the reset operation at the hardware level is avoided, the embodiment of the present invention will not cause excessive impact on the cluster during the exception handling process, thereby improving the stability of the entire cluster. The robustness of the cluster is also further enhanced through precise control and personalized processing at the software level.

[0046] Based on the above embodiment, the host can record the synchronization tasks submitted by the application to each device and the completion conditions of each synchronization task through KMD. Here, KMD refers to a driver running at the kernel level of the operating system, which directly interacts with the hardware device, manages the hardware device resources, and processes requests from UMD (User Mode Driver) or other kernel components. UMD refers to a driver running at the user level of the operating system, which provides an interface for the application, processes requests from the application, and interacts with KMD to execute these requests.

[0047] The device side can detect whether the synchronization task has timed out through the firmware. When the firmware of a device detects that the synchronization task has timed out, it can report MSI (Message Signaled Interrupts) to the KMD on the host side. At this time, the device is an abnormal device. Here, firmware refers to software embedded in the hardware device, which is used to control and manage the relevant operations of the device. MSI is a hardware interrupt mechanism used to transmit interrupt signals between devices (such as GPUs) and hosts (such as CPUs).

[0048] After receiving the interrupt signal, KMD will report the exception to UMD, so that UMD can send an exception notification to the process corresponding to the abnormal device. After receiving the exception notification, these processes can execute the corresponding error handling logic and exit, or they can exit directly. Here, the error handling logic may include recording error logs, displaying error information to the user, performing necessary cleanup operations, etc.

[0049] Based on any of the above embodiments, one or more processes are running on the host, each process is used to start one or more devices, and when any process receives the exception notification, it exits after performing error handling and closes the file descriptors of all devices started by any process one by one when exiting.

[0050] Specifically, one or more processes can be run on the host, and each process can start one or more devices. When the process starts each device, it will open the file descriptor of the device. Each device can correspond to multiple processes, that is, multiple processes can start the same device. When the firmware of a device detects an abnormality in the synchronization task, it will report the MSI interrupt to the KMD. In this case, the device is an abnormal device. After receiving the MSI interrupt, the KMD can send an exception notification to all processes corresponding to the abnormal device through the UMD. When each process corresponding to the abnormal device receives the exception notification, it will perform corresponding error handling and exit. Each process will close the file descriptors of all devices it has started when it exits. When closing the file descriptor of each device, the release function of the KMD will be called to trigger the resource release process of the current device (that is, the device that currently closes the file descriptor).

[0051] It is understandable that when KMD sends an exception notification through UMD, it notifies on a single device basis, that is, notifies all processes corresponding to the abnormal device. After receiving the exception notification, these processes will handle the error according to their own procedures, and then close the file descriptors of all devices started by themselves. The closing action of each file descriptor will enter the resource release process of a device, thereby starting an independent device recovery process. For example, when closing the file descriptor of the current device, the resource release process of the current device will be entered. In this process, the resource release process of other devices will not be executed. The resource release process of other devices will not be entered and executed until the corresponding file descriptor is closed. For each device, when the file descriptor of the device is closed, the resource release process of the device will be entered, and KMD will be used to detect whether the status of the device is busy.

[0052] For example, assume that process 1 and process 2 are running on the host, where process 1 starts devices GPU0 and GPU1, and process 2 starts devices GPU1 and GPU2. If the firmware on GPU1 detects that the synchronization task has timed out, it will report an interrupt to the KMD on the host side. After receiving the interrupt, KMD sends an exception notification to all processes (including process 1 and process 2) that start device GPU1 through UMD. After receiving the exception notification, process 1 will handle the error according to its own process and exit. When exiting, it will close the file descriptors of all devices it has started (including GPU0 and GPU1). When closing the file descriptor of GPU0, it will enter the resource release process of GPU0 to enable KMD to perform status detection on GPU0. When closing the file descriptor of GPU1, it will enter the resource release process of GPU1, so that KMD can perform status detection on GPU1. Similarly, after receiving the exception notification, process 2 will also handle the error according to its own process and exit. When exiting, it will close the resource release process of all devices it has started (including GPU1 and GPU2).

[0053] Based on any of the above embodiments, in step 220, when the current device is detected to be in a busy state, generating a corresponding confirmation response based on the synchronization task completion condition of the current device includes: When the current device is detected to be busy, it enters abnormal recovery mode; In the abnormal recovery mode, the current device is notified to skip the unfinished synchronization task, and a corresponding confirmation response is generated based on the synchronization task completion condition of the current device.

[0054] Specifically, after receiving the interrupt signal, the KMD on the host side will notify all processes corresponding to the abnormal device through the UMD, so that each process will exit after executing the corresponding error handling. When each process exits, it will close the file descriptors of all devices it started to enter the resource release process of each device. For each device, when it enters the resource release process, the KMD will detect the status of the device. If it detects that the device is busy, the KMD will enter the abnormal recovery mode and notify the firmware of the device to skip the remaining unfinished synchronization tasks. Here, the abnormal recovery mode is a special operating mode that allows the system or hardware components to self-repair, recover or reinitialize after an error or abnormality occurs. When the KMD enters the abnormal recovery mode, it means that the KMD has detected that there is an abnormality in the device and needs to perform corresponding operations to repair the abnormal problem of the device, such as releasing the device synchronization waiting state, repairing data errors or reloading the necessary device driver code, etc. These operations are intended to restore the device from an abnormal state.

[0055] It is understandable that unfinished synchronization tasks refer to all synchronization tasks that have been started but not yet completed before the device detects an abnormality in the synchronization task. These tasks may involve various aspects such as data processing, graphics rendering, and computing operations. Since the application that issued these tasks has exited abnormally, it is meaningless for the device to execute subsequent tasks. Therefore, KMD will notify the firmware to skip the execution of the remaining tasks. When the device firmware receives the skip instruction sent by KMD, it will immediately stop the currently executing synchronization task and release the resources related to the task. The specific implementation method of the device skipping the remaining unfinished synchronization tasks depends on the hardware design of the device, and the embodiments of the present invention do not specifically limit this.

[0056] Based on any of the above embodiments, after step 210, the method further includes: When it is detected that the current device is in an idle state, a resource release process is executed.

[0057] Specifically, for each device, when it enters the resource release process, the KMD will detect the status of the device. If it is detected that the device is in an idle state, it means that the current synchronization task of the device has been completed, that is, there is no synchronization task exception or timeout, and the normal resource release process can be executed at this time. It should be understood that the idle state is a type of device state, indicating that the device is currently not executing any tasks and is not occupied by any tasks. In the embodiment of the present invention, detecting that the device is in an idle state is an important basis for judgment, because it allows the host to safely execute the resource release process without interfering with the normal operation of the device or causing data loss.

[0058] It is understood that the execution resource release process usually involves a series of steps, which are designed to ensure that the device can properly release the occupied resources after executing the synchronization task so that other tasks or devices can use these resources. Specifically, the memory and cache resources occupied by the device during the execution of the task can be released, for example, reclaiming the allocated memory block, clearing the data in the cache, or performing other memory management operations; restoring the device to the initial state or default configuration to ensure that it is ready to accept new tasks or operations, for example, resetting the device's internal counters, clearing the status register, or performing other device initialization operations.

[0059] Based on any of the above embodiments, after step 220, the method further includes: Detecting the status of the current device; When it is detected that the current device is in a busy state and the duration exceeds a preset time, an access request of the current device is blocked, and based on system parameters or user input, the current device is triggered to reset.

[0060] It should be noted that for each device, there are many reasons why the device is in a busy state, which may be caused by a synchronization task timeout, or by other software errors or hardware errors. Therefore, after the host sends a confirmation response to the device and releases the device's synchronization waiting state, the device's state can be detected again to confirm whether the measures taken previously (i.e., generating and sending a confirmation response to trigger the device to release the synchronization waiting state) are effective. Sending a confirmation response is a means of trying to solve the synchronization task anomaly, but it cannot guarantee that all problems can be solved. Therefore, it is necessary to detect the device's state again to confirm whether it has returned to normal.

[0061] Specifically, for each device, if the host detects that the device is still in a busy state after releasing the synchronization waiting state, and this state lasts for more than a preset time, it indicates that the device may have a problem that cannot be solved by software means alone, such as hardware failure, unrepairable error, etc. In this case, further measures need to be taken to repair the device. For example, the device can be reset to finally recover the device.

[0062] Specifically, before resetting the device, the device's access request to the host can be blocked. Here, the device's access request refers to the device's request to attempt to communicate with the host. Under normal circumstances, these requests will be processed and responded to normally by the host, but when the device is abnormal or fails, these requests may cause system instability or data loss. Blocking device access requests can be achieved in a variety of ways, depending on the design and implementation of the system. For example, the communication interface or channel between the device and the host can be disabled, such as disabling the SMMU (System Memory Management Unit) to prevent the GPU from accessing the host memory. This ensures that before resetting the device, it will not continue to send access requests to the host or cause system instability.

[0063] Subsequently, the software resources on the device can be released, and a decision can be made based on system parameters or user input whether to trigger the device to reset. Here, system parameters refer to some configurations or settings pre-set by the user in the system. For example, system parameters can include whether to enable the automatic reset function (such as the Host reset enable parameter). User input refers to some options or parameters that the user manually selects or configures through the interactive interface. For example, an abnormal device event can be reported to the user through the interactive interface, and the user can be asked whether to reset the device. The user can choose whether to allow the system to automatically reset the device through the interactive interface, or can manually perform the device reset operation through the corresponding tools.

[0064] In the embodiment of the present invention, when the device is detected to be in a busy state again and the busy state lasts for too long, some irreparable errors of the device can be resolved by resetting the device, thereby restoring the device to a normal working state.

[0065] Based on any of the above embodiments, the method further includes: When the training framework detects a synchronization task exception on any device, it terminates all processes of the training task so that each process closes the file descriptors of all devices started by it.

[0066] It should be noted that since each device will enter the resource release process only when the process closes its file descriptor, when a synchronization task exception occurs in a device, in order to enable other devices without exceptions to quickly enter the resource release process, the embodiment of the present invention has made corresponding collaborative design in the training framework layer of the model.

[0067] Specifically, when the training framework layer of the model detects that a synchronization task exception occurs on a certain device, it will terminate all processes of the training task. When each process is terminated, the file descriptors of all devices it has started will be closed. Accordingly, when the file descriptor of each device is closed, the resource release process of the device will be entered, so that the status of the device can be detected through KMD to enter the recovery process of the device. After entering the recovery process, if it is detected that the device is in an idle state, the resource release process can be executed; if it is detected that the device is in a busy state, it is necessary to first release the synchronization waiting state of the device. Here, the specific implementation process of releasing the synchronization waiting state can refer to the above embodiment, which will not be repeated here.

[0068] Exemplarily, process 0, process 1, and process 2 are running on the host. It is assumed that each device corresponds to only one process, that is, process 0 starts device GPU0, process 1 starts device GPU1, and process 2 starts device GPU2. If the firmware on GPU1 detects that the synchronization task has timed out, the firmware on GPU1 will report an interrupt to KMD, and KMD will send an exception notification to process 1 corresponding to GPU1 through UMD. Process 1 will close the file descriptor of GPU1 to enter the resource release process of GPU1. Subsequently, KMD can perform status detection on GPU1 to restore device GPU1.

[0069] Since KMD's device recovery is a passive process, the process needs to close the file descriptor to enter the recovery process. In order for other GPUs to quickly enter the recovery process after an exception occurs in a GPU, the embodiment of the present invention cooperates with KMD in the training framework layer. As long as the framework layer detects that a GPU (such as GPU1) has an exception, it will terminate all processes of the training task, so process 0 and process 2 will be terminated at this time. When process 0 terminates, it will close the file descriptor of GPU0 that it started, thereby entering the resource release process of GPU0. Similarly, when process 2 is terminated, it will close the file descriptor of GPU2 that it started to enter the resource release process of GPU2.

[0070] Based on any of the above embodiments, Figure 3 FIG. 2 is a flow chart of the task exception handling method provided by the present invention. Figure 3 As shown, the method is applied to a device, and the method includes: Step 310, when a synchronization task exception is detected, an interrupt signal is sent to the host, so that the host sends an exception notification to the process corresponding to the device, the exception notification is used to instruct the process to close the file descriptors of all started devices one by one, and the host is used to record the synchronization tasks executed by each device and the completion condition of each synchronization task; Step 320, when the file descriptor of the device is closed, entering the resource release process of the device so that the host can detect the status of the device; Step 330, receiving a confirmation response, and releasing the synchronization waiting state based on the confirmation response, wherein the confirmation response is generated and sent by the host based on the synchronization task completion condition of the device when the host detects that the device is in a busy state.

[0071] Specifically, the execution subject of the method provided by the embodiment of the present invention can be any device on the computing node. When the device is executing the synchronization task, the firmware on the device will monitor the execution status of the synchronization task. When the firmware detects that the synchronization task on the device has timed out, it will send an interrupt signal to the KMD on the host side. At this time, the device is an abnormal device. After receiving the interrupt signal, the host will send an abnormal notification to all processes corresponding to the abnormal device (that is, all processes that start the abnormal device) to trigger each process to close the file descriptors of all devices it has started. When closing the file descriptor of each device, the resource release process of the device will be entered, and then the status of the device can be detected by the KMD on the host side.

[0072] When the host detects that the device is in a busy state, it indicates that the synchronization task on the device has not been completed or other abnormal errors have occurred in the device. In this case, the host can generate a corresponding confirmation reply based on the recorded synchronization task completion conditions corresponding to the device and send it to the device. After receiving the confirmation reply, the device can release the synchronization waiting state.

[0073] The method provided by the embodiment of the present invention is that when the device detects that the synchronization task is abnormal, the device sends an interrupt signal to the host, so that the host can respond quickly and send an abnormal notification to the process corresponding to the abnormal device, thereby enabling the process to close the file descriptors of all devices started by it one by one. This immediate response mechanism ensures that the abnormal situation can be handled in time, thereby reducing the negative impact that may be caused by the task abnormality. When closing the file descriptor of each device, the resource release process of the device will be entered, so that the status of the device can be detected by the host. When the host detects that the device is in a busy state, it can generate and send a corresponding confirmation response based on the synchronization task completion condition of the device. After the device receives the response, it can release the synchronization waiting state, realizing an exception handling mechanism based on the software level, that is, the synchronization waiting state of the device is released by sending a confirmation response by the host. This mechanism does not involve a reset operation at the hardware level, which can reduce the risks that may be caused by hardware operations, while shortening the time of exception handling and improving processing efficiency.

[0074] Based on any of the above embodiments, an embodiment of the present invention provides a task exception handling system, the system comprising a host and at least one device, the host is used to record the synchronization tasks executed by each device and the completion condition of each synchronization task; Any device is used to send an interrupt signal to the host when detecting an abnormality in the synchronization task; The host is used to send an exception notification to the process corresponding to any one of the devices when receiving the interrupt signal, so that the process closes the file descriptors of all the started devices one by one; The host is also used to take the device that currently closes the file descriptor as the current device, enter the resource release process of the current device, and detect the status of the current device. When it is detected that the current device is in a busy state, based on the synchronization task completion condition of the current device, a corresponding confirmation response is generated and sent to the current device. The confirmation response is used to trigger the current device to release the synchronization waiting state.

[0075] It should be noted that the task exception handling system provided by the embodiment of the present invention can be deployed on each computing node in the cluster. In the process of the cluster being responsible for the model training task, the synchronization tasks submitted by the application to each device and the completion conditions of each synchronization task are recorded through the host on the node. In the process of each device executing the synchronization task, the synchronization tasks on each device are detected by the host. If the application exits abnormally, the host can replace the application that is forced to terminate and send a confirmation response to the device where the exception occurs, so that the device can release the corresponding synchronization waiting state, thereby avoiding affecting the execution of other tasks.

[0076] Figure 4 FIG. 1 is a flow chart of a task exception handling method for a cluster scenario provided by the present invention. Figure 4 As shown, the method can be applied to the above-mentioned task exception handling system. The following takes the device as a GPU, and the system includes 4 GPUs (namely GPU0 to GPU3 shown in the figure) as an example to introduce the steps of the method. The method includes the following steps: Step S0, the KMD on the host side records the synchronization tasks submitted by the application to each GPU, and the completion conditions of each synchronization task; Step S1, the GPU firmware detects that the synchronization task has timed out and reports an MSI interrupt to the KMD, for example, Figure 4 The GPU1 shown in the figure detects that the task has timed out and reports an interrupt to the KMD; Step S2, after receiving the interrupt, KMD reports the exception to UMD, so that UMD sends an exception notification to the application, that is, sends an exception notification to all processes corresponding to the abnormal device (such as GPU1); Step S3, after receiving the abnormal notification, the process corresponding to the abnormal device executes the error handling logic and exits; Step S4, when the process exits, the file descriptors of all GPUs started by the process are closed. When the file descriptor of each GPU is closed, the resource release process of the GPU is entered; Step S5, for the GPU that enters the resource release process, the KMD detects whether the GPU is in a busy state. If the GPU is in a busy state, it enters a recovery mode. If the GPU is in an idle state, the resource release process is executed normally. Step S6, after entering the recovery mode, the KMD notifies the GPU firmware to skip the remaining unfinished synchronization tasks, and sends a corresponding ACK reply to the GPU according to the completion conditions of the synchronization tasks; Step S7, after receiving the ACK reply, the GPU can release the corresponding synchronization waiting state; Step S8, detecting the state of the GPU again, if the GPU is restored to an idle state, executing a normal resource release process, if the GPU is still in a busy state, waiting for a preset time and detecting again; Step S9, after waiting for a preset time, if it is detected that the GPU is in an idle state, it indicates that the GPU anomaly has been resolved and the resource release process can be executed, otherwise the GPU reset process is entered; Step S10, after entering the reset process, first block the GPU from accessing the host, release software resources, and check whether system parameters are set. If there are corresponding system parameters, the GPU is automatically triggered to reset, otherwise wait for the user to manually select or manually reset.

[0077] The embodiment of the present invention provides a task exception handling method based on the software level. By utilizing the kernel driver on the host side to send an ACK reply, the synchronization task on the GPU can be executed and completed. The processing mechanism does not involve a reset operation at the hardware level, has better stability, and takes less time.

[0078] Based on any of the above embodiments, Figure 5 is one of the structural diagrams of the task exception handling device provided by the present invention, such as Figure 5 As shown, the device is applied to a host, and the host is used to record the synchronization tasks performed by each device and the completion conditions of each synchronization task. The device includes: The abnormality notification unit 510 is used to send an abnormality notification to the process corresponding to any device when receiving an interrupt signal sent by any device, so that the process closes the file descriptors of all started devices one by one, and the interrupt signal is sent by any device when detecting an abnormality in the synchronization task; The state detection unit 520 is used to take the device of the currently closed file descriptor as the current device, enter the resource release process of the current device, and detect the state of the current device; The response generation unit 530 is used to generate a corresponding confirmation response based on the synchronization task completion condition of the current device when detecting that the current device is in a busy state, and send the confirmation response to the current device. The confirmation response is used to trigger the current device to release the synchronization waiting state.

[0079] The device provided by the embodiment of the present invention sends an interrupt signal to the host when any device detects an abnormality in the synchronization task, so that the host can respond quickly and send an abnormal notification to the process corresponding to the abnormal device, thereby enabling the process to close the file descriptors of all devices started by it one by one. This immediate response mechanism ensures that the abnormal situation can be handled in time, thereby reducing the negative impact that may be caused by the task abnormality. When closing the file descriptor of each device, the resource release process of the device will be entered, so that the status of the device can be detected by the host. When the host detects that the device is in a busy state, it can generate and send a corresponding confirmation response based on the synchronization task completion condition of the device. After receiving the response, the device can release the synchronization waiting state, realizing an exception handling mechanism based on the software level, that is, the synchronization waiting state of the device is released by sending a confirmation response by the host. This mechanism does not involve a reset operation at the hardware level, which can reduce the risks that may be caused by hardware operations, while shortening the time of exception handling and improving processing efficiency.

[0080] Based on any of the above embodiments, the response generation unit 530 is specifically configured to: When the current device is detected to be busy, it enters abnormal recovery mode; In the abnormal recovery mode, the current device is notified to skip the unfinished synchronization task, and a corresponding confirmation response is generated based on the synchronization task completion condition of the current device.

[0081] Based on any of the above embodiments, one or more processes are running on the host, each process is used to start one or more devices, and when any process receives the exception notification, it exits after performing error handling and closes the file descriptors of all devices started by any process one by one when exiting.

[0082] Based on any of the above embodiments, the device further includes a resource releasing unit, and the resource releasing unit is used to: When it is detected that the current device is in an idle state, a resource release process is executed.

[0083] Based on any of the above embodiments, the device further includes a reset unit, and the reset unit is used to: Detecting the status of the current device; When it is detected that the current device is in a busy state and the duration exceeds a preset time, an access request of the current device is blocked, and based on system parameters or user input, the current device is triggered to reset.

[0084] Based on any of the above embodiments, Figure 6 This is the second structural diagram of the task exception handling device provided by the present invention, such as Figure 6As shown, the device is applied to a device, and the device includes: The interrupt sending unit 610 is used to send an interrupt signal to the host when an abnormality of the synchronization task is detected, so that the host sends an abnormality notification to the process corresponding to the device, and the abnormality notification is used to instruct the process to close the file descriptors of all the started devices one by one, and the host is used to record the synchronization tasks executed by each device and the completion condition of each synchronization task; A process entry unit 620, configured to enter a resource release process of the device when the file descriptor of the device is closed, so that the host detects the status of the device; The state release unit 630 is used to receive a confirmation response and release the synchronization waiting state based on the confirmation response. The confirmation response is generated and sent by the host based on the synchronization task completion condition of the device when the host detects that the device is in a busy state.

[0085] The device provided by the embodiment of the present invention sends an interrupt signal to the host when the device detects an abnormality in the synchronization task, so that the host can respond quickly and send an abnormal notification to the process corresponding to the abnormal device, thereby enabling the process to close the file descriptors of all devices started by it one by one. This immediate response mechanism ensures that the abnormal situation can be handled in time, thereby reducing the negative impact that may be caused by the task abnormality. When closing the file descriptor of each device, the resource release process of the device will be entered, so that the status of the device can be detected by the host. When the host detects that the device is in a busy state, it can generate and send a corresponding confirmation response based on the synchronization task completion condition of the device. After receiving the response, the device can release the synchronization waiting state, realizing an exception handling mechanism based on the software level, that is, the synchronization waiting state of the device is released by sending a confirmation response by the host. This mechanism does not involve a reset operation at the hardware level, which can reduce the risks that may be caused by hardware operations, while shortening the time of exception handling and improving processing efficiency.

[0086] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a task exception handling method, which is applied to a host, and the host is used to record the synchronization tasks executed by each device and the completion conditions of each synchronization task. The method includes: in the case of receiving an interrupt signal sent by any device, sending an exception notification to the process corresponding to any device, so that the process closes the file descriptors of all started devices one by one, and the interrupt signal is sent by any device when the synchronization task is detected to be abnormal; taking the device that currently closes the file descriptor as the current device, entering the resource release process of the current device, and detecting the status of the current device; in the case of detecting that the current device is in a busy state, generating a corresponding confirmation response based on the synchronization task completion condition of the current device, and sending the confirmation response to the current device, and the confirmation response is used to trigger the current device to release the synchronization waiting state.

[0087] The processor 710 can also call the logic instructions in the memory 730 to execute the task exception handling method, which is applied to the device, and the method includes: when a synchronization task exception is detected, sending an interrupt signal to the host so that the host sends an exception notification to the process corresponding to the device, and the exception notification is used to instruct the process to close the file descriptors of all started devices one by one, and the host is used to record the synchronization tasks performed by each device and the completion conditions of each synchronization task; when the file descriptor of the device is closed, entering the resource release process of the device, so that the host can detect the status of the device; receiving a confirmation response, and releasing the synchronization waiting state based on the confirmation response, and the confirmation response is generated and sent by the host based on the synchronization task completion condition of the device when the host detects that the device is in a busy state.

[0088] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the task exception handling method provided by the above methods. The method is applied to a host, and the host is used to record the synchronization tasks performed by each device and the completion conditions of each synchronization task. The method includes: when an interrupt signal sent by any device is received, an exception notification is sent to a process corresponding to any device, so that the process closes the file descriptors of all started devices one by one, and the interrupt signal is sent by any device when an exception is detected in the synchronization task; the device that currently closes the file descriptor is taken as the current device, the resource release process of the current device is entered, and the status of the current device is detected; when it is detected that the current device is in a busy state, a corresponding confirmation response is generated based on the synchronization task completion condition of the current device, and the confirmation response is sent to the current device, and the confirmation response is used to trigger the current device to release the synchronization waiting state.

[0090] When the computer program is executed by the processor, the computer can also execute the task exception handling method provided by the above methods, which is applied to a device, and includes: when a synchronization task exception is detected, sending an interrupt signal to the host so that the host sends an exception notification to the process corresponding to the device, the exception notification is used to instruct the process to close the file descriptors of all started devices one by one, and the host is used to record the synchronization tasks performed by each device and the completion condition of each synchronization task; when the file descriptor of the device is closed, entering the resource release process of the device so that the host detects the status of the device; receiving a confirmation response, and releasing the synchronization waiting state based on the confirmation response, the confirmation response is generated and sent by the host based on the synchronization task completion condition of the device when the host detects that the device is in a busy state.

[0091] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the task exception handling method provided by the above-mentioned methods, the method being applied to a host, the host being used to record the synchronization tasks executed by each device and the completion conditions of each synchronization task, the method comprising: upon receiving an interrupt signal sent by any device, sending an exception notification to a process corresponding to any device, so that the process closes the file descriptors of all started devices one by one, the interrupt signal being sent by any device upon detecting an exception in the synchronization task; taking the device that currently closes the file descriptor as the current device, entering the resource release process of the current device, and detecting the status of the current device; upon detecting that the current device is in a busy state, generating a corresponding confirmation response based on the completion conditions of the synchronization task of the current device, and sending the confirmation response to the current device, the confirmation response being used to trigger the current device to release the synchronization waiting state.

[0092] When the computer program is executed by the processor, it is implemented to execute the task exception handling method provided by the above-mentioned methods. The method is applied to a device, and the method includes: when a synchronization task exception is detected, an interrupt signal is sent to the host, so that the host sends an exception notification to the process corresponding to the device, and the exception notification is used to instruct the process to close the file descriptors of all started devices one by one, and the host is used to record the synchronization tasks executed by each device and the completion condition of each synchronization task; when the file descriptor of the device is closed, the resource release process of the device is entered, so that the host detects the status of the device; a confirmation response is received, and the synchronization waiting state is released based on the confirmation response. The confirmation response is generated and sent by the host based on the synchronization task completion condition of the device when the host detects that the device is in a busy state.

[0093] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0094] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A task exception handling method, characterized in that: The method is applied to a host, and the host is used to record synchronization tasks executed by each device and the completion condition of each synchronization task. The method includes: When receiving an interrupt signal sent by any device, sending an exception notification to the process corresponding to the any device, so that the process closes the file descriptors of all started devices one by one, wherein the interrupt signal is sent by any device when detecting an abnormality in the synchronization task; The device whose file descriptor is currently closed is used as the current device, a resource release process of the current device is entered, and the status of the current device is detected; When it is detected that the current device is in a busy state, a corresponding confirmation response is generated based on the synchronization task completion condition of the current device, and the confirmation response is sent to the current device, and the confirmation response is used to trigger the current device to release the synchronization waiting state.

2. The task exception handling method according to claim 1, characterized in that: The step of generating a corresponding confirmation response based on a synchronization task completion condition of the current device when the current device is detected to be in a busy state includes: When the current device is detected to be busy, it enters abnormal recovery mode; In the abnormal recovery mode, the current device is notified to skip the unfinished synchronization task, and a corresponding confirmation response is generated based on the synchronization task completion condition of the current device.

3. The task exception handling method according to claim 1, characterized in that: One or more processes are running on the host, each process is used to start one or more devices, and when any process receives the exception notification, it exits after performing error handling, and closes the file descriptors of all devices started by any process one by one when exiting.

4. The task exception handling method according to any one of claims 1 to 3, characterized in that: The detecting the state of the current device further comprises: When it is detected that the current device is in an idle state, a resource release process is executed.

5. The task exception handling method according to any one of claims 1 to 3, characterized in that: The step of sending the confirmation response to the current device further comprises: Detecting the status of the current device; When it is detected that the current device is in a busy state and the duration exceeds a preset time, an access request of the current device is blocked, and based on system parameters or user input, the current device is triggered to reset.

6. A task exception handling method, characterized in that: The method is applied to a device, and the method comprises: When an exception is detected in a synchronization task, an interrupt signal is sent to the host, so that the host sends an exception notification to the process corresponding to the device, the exception notification is used to instruct the process to close the file descriptors of all started devices one by one, and the host is used to record the synchronization tasks executed by each device and the completion condition of each synchronization task; When the file descriptor of the device is closed, a resource release process of the device is entered, so that the host detects the status of the device; A confirmation response is received, and the synchronization waiting state is released based on the confirmation response, wherein the confirmation response is generated and sent by the host based on a synchronization task completion condition of the device when the host detects that the device is in a busy state.

7. A task exception handling system, characterized in that: It includes a host and at least one device, wherein the host is used to record synchronization tasks executed by each device and completion conditions of each synchronization task; Any device is used to send an interrupt signal to the host when detecting an abnormality in the synchronization task; The host is used to send an exception notification to the process corresponding to any one of the devices when receiving the interrupt signal, so that the process closes the file descriptors of all the started devices one by one; The host is also used to take the device that currently closes the file descriptor as the current device, enter the resource release process of the current device, and detect the status of the current device. When it is detected that the current device is in a busy state, based on the synchronization task completion condition of the current device, a corresponding confirmation response is generated and sent to the current device. The confirmation response is used to trigger the current device to release the synchronization waiting state.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the task exception handling method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the task exception handling method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the task exception handling method according to any one of claims 1 to 6 is implemented.

Images