Systems and methods of enabling multi-region of interest processing by a model at inference-time

Abstract

A device includes a memory configured to store model data associated with a multimodal model that includes an image encoder and a large language model (LLM). The device also includes one or more processors configured to obtain image data representing an image. The one or more processors are configured to obtain data representing multiple regions of interest (ROIs) within the image, the multiple ROIs including one or more ROIs that include text. The one or more processors are configured to arrange image representations of the ROIs into a single canvas image, and to input the canvas image into the image encoder to obtain image tokens associated with cells of the canvas image. The one or more processors are configured to provide a first input to the LLM, the first input based on the image tokens, to generate a response output that corresponds to a response to a query.

Images