Neural network reasoning and speculation acceleration method, system and device and medium
By determining the optimal number of operator layers of each hardware device and optimizing the computing power utilization of hardware devices, the problem of the calculation speed of neural network model acceleration of low-computing hardware devices is solved, and efficient neural network model inference is realized, suitable for multi-platform and multi-architecture hardware devices.
Patent Information
- Application Number
- CN202410914836.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-12
AI Technical Summary
It is difficult for the prior art to use low-computing hardware devices to complete the speculation and acceleration of neural network models, and the hardware performance requirements are high.
By obtaining the total number of operator layers of the neural network model to be inferred, we determine the optimal number of operator layers of each hardware device, and control each hardware device to infer it, accelerate the inference process of the neural network model, and use edge computing devices, The combination of cloud servers and terminal devices optimizes the computing power utilization of hardware devices and uses SIMD instruction sets and CUDA instruction sets for acceleration.
It realizes the speculation acceleration of the neural network model of low-computing hardware devices, avoids compiler and platform restrictions, reduces the cost of development and model inference, improves the speed and efficiency of inference, and is suitable for multi-platform and multi-architecture hardware devices.
