Rotary positional embedding fused general matrix multiplication kernel

Abstract

Systems and techniques are described herein for data processing. For example, a computing device can process, using a computing core of the at least one processor, a plurality of tensors using a general matrix multiplication (GEMM) kernel to generate a first kernel output. The computing device can store the first kernel output in local memory of the computing core. The computing device can process, using a rotary positional embedding (RoPE) function at the computing core, the first kernel output to generate a RoPE of the first kernel output, wherein the RoPE function is included as an epilogue of the GEMM kernel.

Images