Large language model inference system and large language model inference method

Abstract

A large language model inference system and a large language model inference method are provided. The large language model inference system includes a first computing device. The first computing device is used for setting a model file to be inferred, and selecting an inference framework for inferring a large language model according to the model file to be inferred. The first computing device detects hardware performance information of a plurality of dynamic random access memories and a plurality of video random access memories, and obtains a plurality of model loading parameters. The first computing device distributes model data of the large language model to at least one of the plurality of dynamic random access memories and the plurality of video random access memories according to the hardware performance information and the plurality of model loading parameters. The first computing device operates at least one of the plurality of dynamic random access memories and the plurality of video random access memories to perform an inference operation of the large language model according to at least one inference output screening parameter.

Images