Neural network reasoning and speculation acceleration method, system and device and medium

By determining the optimal number of operator layers of each hardware device and optimizing the computing power utilization of hardware devices, the problem of the calculation speed of neural network model acceleration of low-computing hardware devices is solved, and efficient neural network model inference is realized, suitable for multi-platform and multi-architecture hardware devices.

CN118940836APending Publication Date: 2024-11-12XINGFAN XINGQI (CHENGDU) TECH CO LTD
0 Cites 2 Cited by

Patent Information

Application Number
CN202410914836.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

It is difficult for the prior art to use low-computing hardware devices to complete the speculation and acceleration of neural network models, and the hardware performance requirements are high.

Method used

By obtaining the total number of operator layers of the neural network model to be inferred, we determine the optimal number of operator layers of each hardware device, and control each hardware device to infer it, accelerate the inference process of the neural network model, and use edge computing devices, The combination of cloud servers and terminal devices optimizes the computing power utilization of hardware devices and uses SIMD instruction sets and CUDA instruction sets for acceleration.

Benefits of technology

It realizes the speculation acceleration of the neural network model of low-computing hardware devices, avoids compiler and platform restrictions, reduces the cost of development and model inference, improves the speed and efficiency of inference, and is suitable for multi-platform and multi-architecture hardware devices.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a neural network inference and speculation acceleration method, system and device, and a medium. The method comprises the following steps: obtaining the total layer number of operators of a neural network model to be speculated and accelerated; the number of the hardware devices is multiple, the hardware devices are used for speculating the acceleration neural network model to be speculated, the sum of the number of the optimal operator layers corresponding to the hardware devices is equal to the total number of operator layers, and the number of the optimal operator layers corresponding to the hardware devices is equal to the total number of operator layers; the types of the hardware equipment comprise edge computing equipment, a cloud server and terminal equipment; and controlling each hardware device to speculate the corresponding optimal operator layer number. The method belongs to the field of neural network speculation, and by determining the optimal operator layer number corresponding to each hardware device, the low-computing-power device can also complete speculation acceleration of the neural network model to be speculated and accelerated.
Need to check novelty before this filing date? Find Prior Art