Visual program augmented pathological visual language large model

Abstract

The application discloses a visual procedure enhanced pathological visual language large model, relates to the technical field of visual language large models, and comprises a visual procedure pool, a visual procedure selection module and a procedure guided diagnosis module.The visual procedure pool is used for storing callable visual procedures, the visual procedures include a basic segmentation and classification procedure for performing basic segmentation and classification on a pathological image, and a microenvironment feature calculation procedure for calculating microenvironment features based on a basic segmentation and classification result;the visual procedure selection module is used for analyzing an input pathological diagnosis question according to a language large model, selecting relevant visual procedures from the visual procedure pool, and outputting a procedure selection decision package; and the procedure guided diagnosis module is used for processing an input pathological image by using the selected visual procedures in the procedure selection decision package, extracting structured features, fusing the structured features and the diagnosis question into a procedure guided question, performing diagnosis reasoning based on the procedure guided question and the pathological image, and generating a diagnosis result.

Images