Model reasoning exception handling method and device

Abstract

The present application provides a method for handling model inference exceptions, which includes receiving and executing each model inference task of the target model inference session issued by the application processor by the accelerator. The above target model inference session contains multiple target task flows, and one target The task flow contains multiple model inference tasks; when any model inference task in any target task flow in the above-mentioned target model inference session is executed abnormally, the above-mentioned accelerator executes the remaining models contained in the above-mentioned target model inference session according to the exception handling mode Inference tasks, wherein the remaining model inference tasks are all model inference tasks executed after any one of the model inference tasks; the accelerator feeds back the abnormality of the target model inference session to the application processor. By adopting the application, the waste of computing resources caused by model reasoning exceptions can be reduced, the model reasoning abnormalities can be fed back in time, the efficiency of model reasoning can be improved, and the applicability is high.

Description

technical field [0001] The present application relates to the technical field of computer networks, and in particular, to a method and device for abnormal processing of model inference. Background technique [0002] With the development of computer network technology, there are more and more application scenarios that need to build neural network models. For example, in the autonomous driving system, there are a large number of scenarios that need to use artificial intelligence (AI) model inference, and the AI ​​model is essentially a deep neural network model. The neural network model has the characteristics of intensive matrix and vector calculation. The computing power of the system is very high. Ordinary central processing units (Central Processing Units, CPUs) generally cannot meet the computing power requirements of neural network models, so special accelerators are needed to perform inference acceleration, such as graphics processing units (Graphics Processing Units,...

AI Technical Summary

Problems solved by technology

[0003] The inventor of the present application found in the course of research and practice that in the prior art, when an error occurs in the execution of a certain task flow by the accelerator, the error is only recorded in the task flow, and subsequent tasks will still be executed until the task flow It will greatly waste the computing resources of the accelerator, and may also cause other unpredictable errors due to the execution of wrong tasks by the accelerator.

In addition, since the application side only synchronizes with one task flow in the accelerator, when an error occurs in another task flow, the accelerator cannot return the error information to the application side, and the application side cannot know the task flow where the error occurred, which has poor applicability

Images