Cross-platform model reasoning method and system, storage medium and equipment

Abstract

The invention provides a cross-platform model reasoning method and system, a storage medium and equipment. The method comprises the steps od adding a deployment chip in a target back-end module of a deep learning compiling framework as a marking back end; placing a runtime library and a machine learning library of the deployed chip into the deep learning compiling framework so as to realize a performance function of the deployed chip on the deep learning compiling framework and establish an operator warehouse corresponding to the marking back end; analyzing the model file by the deep learning compiling framework to generate calculation graph representation in the deep learning compiling framework; in response to the fact that a target back end is set as the marking back end, searching the implementation of each operator in the computational graph from the operator warehouse, wherein the deep learning compiling framework generates a dynamic link library through the corresponding implementation of each operator and the corresponding performance function; and loading the dynamic link library on a deployment chip to execute model reasoning. According to the invention, the workload of switching the target hardware platform is reduced, and the expansibility of the back end is improved.

Description

technical field [0001] The present invention relates to the technical field of servers, in particular to a method, system, storage medium and equipment for cross-platform model reasoning. Background technique [0002] In the model inference task, it is a very challenging task to deploy the trained model on different target hardware platforms such as CPU, GPU, FPGA and other new artificial intelligence chips such as Cambrian MLU and ensure the efficiency of reasoning. Different types of chips may have large differences in memory hierarchy, supported instructions and data types, etc. Most of the existing model reasoning and optimization methods focus on a single type of chip device. When switching the target hardware platform, the model reasoning method often needs to be replaced accordingly, and the resulting cost will increase with the rapid iteration of artificial intelligence chips. [0003] Currently, TensorRT, the reasoning acceleration framework launched by NVIDIA, par...

AI Technical Summary

Problems solved by technology

[0006] In view of this, the object of the present invention is to propose a general model reasoning method, system, storage medium and equipment that can be used on different hardware platforms, so as to solve the problem of the traditional reasoning framework switching target hardware platform in the prior art. High, insufficient back-end scalability and other issues

Images