Mass spectrometry imaging intelligent analysis system and method based on multi-modal large model

Abstract

The application discloses a mass spectrum imaging intelligent analysis system and method based on a multi-modal large model. The system comprises: an MSI visual encoder for extracting spectral-spatial joint features of mass spectrum imaging data; a visual adapter for mapping visual features to a language embedding space; a large language model for understanding user natural language queries and generating a text answer; and a segmentation decoder for outputting a pixel-level segmentation mask in response to SEG guide tokens generated by the large language model. The application enables the model to perceive the chemical meaning of m / z values through physical perception spectral position coding and tubular mask self-supervised pre-training. Through the SEG guide token mechanism, the synchronous output of the text answer and the segmentation mask is realized. The application supports users to analyze mass spectrum imaging data in a natural language interaction mode, reduces the professional threshold, realizes the integrated output of semantic understanding and accurate segmentation, and can be applied to the fields of disease research, drug development and clinical pathology analysis.

Images