A computing system and method for performing a visual language model

Abstract

A computing system for performing a visual language model includes an edge device and a server is provided. The edge device includes an image sensor and a processor. The image sensor is configured to capture an original image. The processor is configured to downscale the image, divide it into patches, embed the patches into vectors to generate image tokens, and encode the tokens. The server is configured to receive and decode the tokens, extract image features using a vision transformer, provide a user interface for receiving a prompt, process the prompt using a language model to generate text features, and analyze both image and text features to produce a multi-modal output.

Images