Method and system for inference process in neural network

Abstract

This application relates to a method and system for inference processes in neural networks. The method for inference processes in neural networks includes: in a pre-filling stage, generating a sparse key tensor by sparsifying each key vector in a key tensor, and storing the sparse key tensor in a key-value cache memory device; and in a decoding stage, generating newly generated sparse key vectors in the sparse key tensor by sparsifying newly generated key vectors in the key tensor, storing the newly generated sparse key vectors in the sparse key tensor in a key-value cache memory device, retrieving the sparse key tensor from the key-value cache memory device, and performing attention operations based on the sparse key tensor.

Images