A Mass Spectrometry Imaging Intelligent Analysis System and Method Based on a Multimodal Large Model

Abstract

This invention discloses an intelligent analysis system and method for mass spectrometry imaging based on a multimodal large model. The system includes: an MSI visual encoder for extracting spectral-spatial joint features from mass spectrometry imaging data; a visual adapter for mapping visual features to a language embedding space; a large language model for understanding user natural language queries and generating text responses; and a segmentation decoder for responding to SEG guiding tokens generated by the large language model and outputting pixel-level segmentation masks. This invention enables the model to perceive the chemical meaning of m / z values ​​through physically-aware spectral position encoding and self-supervised pre-training with tubular masks; and achieves synchronous output of text responses and segmentation masks through the SEG guiding token mechanism. This invention supports users in analyzing mass spectrometry imaging data through natural language interaction, lowering the professional threshold and achieving integrated output of semantic understanding and accurate segmentation, applicable to disease research, drug development, and clinical pathology analysis.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and medical image analysis, specifically relating to the application of multimodal large language models in intelligent analysis of high-dimensional spectral imaging data. More specifically, this invention provides an intelligent analysis system and method for mass spectrometry imaging based on a multimodal large model. Background Technology

[0002] Mass Spectrometry Imaging (MSI) technology can acquire spatial distribution information of hundreds to thousands of metabolites, lipids, or proteins in situ on tissue sections, making it an important tool for disease research, drug development, and clinicopathological analysis. However, intelligent analysis of MSI data faces the following technical bottlenecks:

[0003] First, the analysis is highly specialized: interpreting MSI data requires a dual background in metabolomics and pathology. Existing MSI analysis software (such as SCiLS Lab, MATLAB MSI Toolbox, etc.) mainly relies on manually delineating regions of interest, which depends on experts to annotate each region one by one. This process is cumbersome, highly subjective, and difficult to scale.

[0004] Second, segmentation and semantic interpretation are disconnected: existing methods usually output molecular spatial distribution maps and statistical analysis results independently, and cannot generate pixel-level spatial segmentation results and natural language semantic interpretation in a unified manner. Users need to correlate and interpret multiple analysis results on their own, and cannot obtain direct and comprehensive semantic-level conclusions.

[0005] Third, it lacks natural language interaction capabilities: traditional image segmentation or clustering algorithms are "static" analyses that cannot understand the flexible and ever-changing analytical intentions of users (such as "finding areas with abnormally active metabolism at the edge of tumors"), and lack interactive natural language question-and-answer analysis capabilities.

[0006] In recent years, multimodal large language models have made significant progress in general image understanding and dialogue. However, their application to MSI data analysis faces the following inherent limitations: First, existing visual encoders are designed specifically for RGB three-channel images and cannot effectively process MSI data cubes with hundreds to thousands of spectral channels. Second, existing models are not aware of the physicochemical meaning of m / z values ​​and cannot associate spectral channels with specific molecular identities. Third, existing visual language models lack pixel-level segmentation output capabilities, only generating text descriptions and failing to simultaneously output accurate segmentation masks for corresponding target regions. These limitations prevent current technologies from achieving the integrated analysis goal of performing semantic understanding and pixel-level segmentation on professional scientific images driven by natural language commands.

[0007] In view of this, the present invention proposes a mass spectrometry imaging intelligent question-answering system based on a multimodal large model and its construction method to solve the above-mentioned technical problems. Summary of the Invention

[0008] This invention aims to overcome the shortcomings of existing technologies and provide a multimodal large-scale model-based intelligent analysis system and method for mass spectrometry imaging, addressing the following technical problems: First, existing MSI analysis heavily relies on manual expert operation and lacks automated intelligent analysis capabilities; second, existing methods cannot unify pixel-level spatial segmentation and natural language semantic interpretation in their output; third, existing multimodal large-scale models cannot effectively process high-dimensional, multi-channel MSI data and lack pixel-level segmentation output capabilities. This invention achieves the above objectives through the following technical solutions: In terms of method, this invention provides a method for constructing and using a multimodal large-scale model-based intelligent analysis system for mass spectrometry imaging; in terms of system, this invention provides a multimodal large-scale model-based intelligent analysis system for mass spectrometry imaging; in terms of storage medium, this invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above methods. This system can respond to user natural language query commands, perform semantic understanding, pixel-level segmentation of specified regions, and structured text description on high-dimensional mass spectrometry imaging data, realizing interactive intelligent analysis of mass spectrometry imaging data.

[0009] To achieve the above objectives, the present invention provides:

[0010] A mass spectrometry imaging intelligent analysis system based on a multimodal large model includes: an MSI visual encoder, a visual adapter, a large language model, and a segmentation decoder;

[0011] The MSI visual encoder is used to receive the mass spectrometry imaging MSI data cube and its associated m / z value vector, and output a fixed-length visual token sequence and a multi-scale dense feature map that retains spatial structure information.

[0012] The visual adapter is connected to the output of the MSI visual encoder and is used to project the visual token sequence into the embedding space of the large language model to generate a visual-side embedding sequence.

[0013] The large language model is used to receive the visual-side embedding sequence and the text embedding sequence of the user's natural language query instruction, perform autoregressive decoding, generate a structured text answer containing natural language description, and generate an SEG guiding token during the autoregressive decoding process when the current answer needs to output the segmentation result.

[0014] The segmentation decoder has its input connected to the output of the large language model and the output of the MSI visual encoder. In response to the SEG guiding token, it extracts the hidden state vector of the large language model corresponding to the token, maps the vector to a segmentation cue embedding vector, and fuses it with the multi-scale dense feature map to finally output a pixel-level segmentation mask with the same spatial resolution as the original MSI data.

[0015] The system can simultaneously output the structured text answer and the pixel-level segmentation mask.

[0016] Furthermore, the MSI visual encoder includes: a spatial downsampling convolution unit, a physical perception spectral position encoding unit, a spatial position encoding injection unit, an alternating spectral-spatial attention stacking unit, and a dual-branch output unit;

[0017] The spatial downsampling convolutional unit is used to compress the spatial height and width of the input MSI data to 1 / s of the original value, and to expand the number of channels to the encoder feature dimension D.

[0018] The physical sensing spectral position encoding unit is used to normalize the m / z value of each visible channel; apply logarithmically uniformly distributed multi-frequency Fourier encoding to the normalized m / z value to generate a Fourier feature vector; map the Fourier feature vector to an m / z position encoding vector through linear projection; map the intensity value of the visible channel to an intensity encoding vector through linear projection; concatenate the m / z position encoding vector and the intensity encoding vector and then perform fusion projection to generate a spectral token for each visible channel, forming a spectral token sequence; simultaneously, calculate the normalized m / z difference for any two visible channels, map it to a scalar bias value through a multilayer perceptron, and form an m / z correlation bias matrix.

[0019] The spatial location coding injection unit is used to apply a learnable location code to the downsampled spatial grid coordinates and to superimpose the location code onto each spatial location of the spectral token sequence in a broadcast manner.

[0020] The final spatial feature map is obtained by processing the m / z correlation bias matrix and spectral token sequence through the alternating spectral-spatial attention stacking unit.

[0021] The dual-branch output unit includes a cross-attention pooling unit and a multi-scale feature pyramid unit. The cross-attention pooling unit is used to extract a fixed-length visual token sequence from the final spatial feature map through the cross-attention pooling mechanism. The multi-scale feature pyramid unit is used to generate a multi-scale dense feature map from the spatial feature map output by the alternating spectral-spatial attention stacking unit through a feature pyramid network.

[0022] Furthermore, the alternating spectral-spatial attention stacking unit includes N cascaded alternating attention blocks, each block containing sequentially a spectral self-attention sub-block, a spatial self-attention block, and a feature broadcast fusion unit;

[0023] The spectral self-attention sub-block is used to independently perform multi-head self-attention operation on the visible spectral channel sequence at each spatial location, and superimpose the m / z correlation bias matrix into the attention weight matrix, and perform mean pooling along the spectral dimension to aggregate the spectral token sequence at each spatial location into a single pixel feature vector to obtain the spatial feature sequence.

[0024] Spatial self-attention sub-block is used to perform multi-head self-attention operation on spatial feature sequences across all spatial locations, capturing long-range spatial dependencies between different anatomical regions of tissue slices, and obtaining spatial feature sequences enhanced with spatial relationships.

[0025] The feature broadcast fusion unit is used to broadcast the spatial feature sequence after spatial relation enhancement back to the spectral dimension and perform residual superposition with the spectral token sequence to obtain an updated feature tensor for use by the next alternation block.

[0026] After processing through N stacked attention blocks, the final spatial feature map is output.

[0027] Furthermore, the segmentation decoder includes: a SEG token extraction and projection module, a semantically guided feature fusion module, and a mask generation module;

[0028] The SEG token extraction and projection module is used to extract the hidden state vector at the corresponding position of the output of the last Transformer layer of the large language model after detecting the output of the SEG guiding token of the large language model; and to project the hidden state vector into a segmentation cue embedding vector with the same dimension as the feature dimension D of the MSI visual encoder through two layers of multilayer perceptron.

[0029] The semantic guidance feature fusion module is used to align the segmentation cue embedding vector with the multi-scale dense feature map in the channel dimension; broadcast and expand the vector along the spatial dimension so that its shape is the same as the finest-grained dense feature map; add the broadcast vector to the finest-grained dense feature map element by element, and refine the features through a lightweight convolutional layer to obtain a spatial feature map that incorporates semantic guidance information; inject other scale dense feature maps from the MSI visual encoder step by step in the subsequent upsampling process through skip connections;

[0030] The mask generation module is used to upsample the spatial feature map that incorporates semantic guidance information through multiple levels of deconvolution to gradually restore the spatial resolution of the original MSI data; after each level of upsampling, it is followed by BatchNorm normalization and ReLU activation function; finally, it outputs a pixel-level segmentation mask through a 1×1 convolutional layer and Sigmoid activation function, where the value of each pixel represents the confidence that the pixel belongs to the target region specified by the user query.

[0031] A mass spectrometry imaging intelligent analysis method based on a multimodal large model, applied to any of the systems described above, is characterized by comprising the following steps:

[0032] S1. Perform preprocessing on the original MSI data; generate a tubular mask, and perform physical sensing spectral-spatial joint encoding on the masked MSI data; perform self-supervised training by reconstructing the mask spectrum, and optimize the training parameters of the MSI visual encoder with a combined loss function;

[0033] S2. Construct training data containing pure dialogue subsets, visual question answering subsets, and segmentation reasoning subsets; perform multimodal joint training on the pre-trained MSI visual encoder, visual adapter, large language model, and segmentation decoder;

[0034] S3. Deploy the trained system on a computing device to receive MSI data and natural language query commands uploaded by the user; perform preprocessing and visual encoding on the MSI data to obtain a visual token sequence and a multi-scale dense feature map; project the visual token sequence onto the language embedding space through a visual adapter to obtain a visual-side embedding sequence; convert the user's natural language query command into a query text embedding sequence through a large language model; concatenate the visual-side embedding sequence and the query text embedding sequence and input them into the large language model for autoregressive decoding; when the SEG guiding token is detected, trigger the segmentation decoder to generate a pixel-level segmentation mask; and simultaneously output the structured text answer and the segmentation mask.

[0035] Furthermore, in S1, the physical sensing spectral-spatial joint encoding includes: normalizing the m / z value of each visible channel; applying logarithmically uniformly distributed multi-frequency Fourier encoding to the normalized m / z value to generate a Fourier feature vector; mapping the Fourier feature vector to an m / z position encoding vector via linear projection; mapping the intensity value of the visible channel to an intensity encoding vector via linear projection; concatenating the m / z position encoding vector and the intensity encoding vector and then performing fusion projection to generate a spectral token for each visible channel, forming a spectral token sequence; simultaneously, calculating the normalized m / z difference for any two visible channels, mapping it to a scalar bias value via a multilayer perceptron, and forming an m / z correlation bias matrix.

[0036] The combined loss is composed of a weighted sum of the mean square error loss and the spectral angle mapping loss.

[0037] Furthermore, in S2, the pure dialogue subset consists of natural language questions and corresponding answers, and does not contain MSI image data or segmentation mask annotations;

[0038] The visual question-answering subset consists of an MSI data cube, a matching m / z value vector, natural language questions and text answers, but does not include a segmentation mask.

[0039] The segmentation inference subset includes an MSI data cube, a matching m / z value vector, a natural language question, a text response containing a SEG guiding token, and a corresponding pixel-level segmentation mask.

[0040] Furthermore, in S2, the multimodal joint training includes:

[0041] Phase 1: Using pure dialogue subsets and visual question-answering subsets, freeze all parameters of the pre-trained MSI visual encoder and large language model, train only the visual adapter, and use the image-text pairs in the visual question-answering subset as supervision signals to achieve preliminary semantic alignment between the visual feature space and the language embedding space. The training objective is the standard autoregressive language modeling loss.

[0042] The second stage involves introducing a segmentation decoder, removing the freezing restrictions of the MSI visual encoder and the large language model, updating all parameters of the MSI visual encoder with a low learning rate, efficiently fine-tuning the parameters of the large language model using the low-rank adaptive LoRA method, updating the visual adapter and segmentation decoder with all parameters, and performing end-to-end joint instruction fine-tuning training on the entire system using mixed data of three subsets. The joint loss function is composed of a weighted sum of the language modeling loss and the segmentation loss.

[0043] The segmentation loss in the second stage is composed of the equal weighted sum of the binary cross-entropy loss and the Dice loss. For training data samples without segmentation mask labels, the segmentation loss is set to zero.

[0044] Furthermore, in S3, after training, the system saves the full parameter weights of the visual encoder, the full parameter weights of the visual adapter, the original weights and LoRA incremental weights of the large language model, and the full parameter weights of the segment decoder. Before model deployment, the low-rank adaptation incremental weights of the large language model are merged into its original weights, and the weights of the visual encoder, visual adapter, merged large language model, and segment decoder are uniformly organized into a single model weight file.

[0045] Furthermore, in S3, when the user does not provide MSI data, the system skips the visual encoding and segmentation decoding process and directly performs autoregressive decoding on the natural language query command in plain text mode, outputting only a structured text answer.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] First, it lowers the professional threshold for MSI data analysis. This invention allows users to describe their analysis intentions in natural language, and the system automatically completes region identification and segmentation, eliminating the need for professionals to manually delineate regions of interest, thus significantly reducing reliance on the professional background of operators.

[0048] Second, it achieves integrated output of spatial segmentation and semantic interpretation. Through the SEG guiding token mechanism, this invention enables the large language model to trigger the segmentation decoder while generating text responses, and simultaneously outputs pixel-level segmentation masks and structured text interpretations in a single inference, overcoming the defect of existing methods where segmentation results and semantic descriptions are separated.

[0049] Third, this invention addresses the applicability of multimodal large models to high-dimensional scientific data. The physicochemically-aware spectral position encoding designed in this invention explicitly injects the physicochemical priors of the m / z values ​​into the encoder, enabling the model to perceive the chemical identity of spectral channels. The tubular mask self-supervised pre-training strategy enables the encoder to effectively learn the spectral-spatial joint representation of MSI data without the need for labeled data, overcoming the inherent limitation of general multimodal large models in handling high-dimensional non-RGB scientific images.

[0050] Fourth, a low-cost training data construction scheme is provided. This invention proposes an automated data construction method that uses a large language model to assist in generating MSI image-text pairs, and combines an active learning strategy to optimize the priority ranking of segmentation annotations, achieving effective training of the system under the condition of extremely scarce professional annotation data.

[0051] Fifth, it supports flexible and diverse deployment scenarios and output formats. This invention supports both local stand-alone deployment and remote server deployment modes, and supports three precision configurations: full precision, half precision, and quantization inference. The output results support multiple formats such as interactive visualization, image files, numerical arrays, and structured reports, demonstrating good engineering practicality and scalability. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments are briefly introduced below.

[0053] Figure 1 This is an overall architecture diagram of a mass spectrometry imaging intelligent question-answering system according to an embodiment of the present invention.

[0054] Figure 2 This is a flowchart illustrating the overall process of one embodiment of the present invention.

[0055] Figure 3This is a detailed flowchart of step S100 of one embodiment of the present invention.

[0056] Figure 4 This is a structural diagram of the MSI visual feature extraction module according to an embodiment of the present invention.

[0057] Figure 5 This is a detailed flowchart of step S200 of one embodiment of the present invention.

[0058] Figure 6 This is a schematic diagram of a quadruplet training data structure according to an embodiment of the present invention.

[0059] Figure 7 This is a system reasoning flowchart of one embodiment of the present invention. Detailed Implementation

[0060] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0061] like Figure 1 This illustrates the connections and data flow between the four core modules of the MSI visual encoder, visual adapter, large language model, and segmentation decoder, and marks the transmission path of the SEG bootstrap token and its two output branches; Figure 2 The diagram illustrates the hierarchical structure and execution order of the three main steps S100, S200, and S300 and their sub-steps of the method of the present invention.

[0062] This invention provides an intelligent analysis system and method for mass spectrometry imaging based on a multimodal large model. The system includes: an MSI visual encoder, a visual adapter, a large language model, and a segmentation decoder. The MSI visual encoder receives a mass spectrometry imaging MSI data cube and its associated m / z value vector, and outputs a fixed-length visual token sequence and a multi-scale dense feature map that preserves spatial structure information. The visual adapter, connected to the output of the MSI visual encoder, projects the visual token sequence into the embedding space of the large language model to generate a visual-side embedding sequence. The large language model receives the visual-side embedding sequence and a text embedding sequence of a user's natural language query command. The system performs autoregressive decoding to generate a structured text answer containing natural language descriptions. When the current answer requires a segmentation result, a SEG guiding token is generated during the autoregressive decoding process. The segmentation decoder, whose input is connected to the output of the large language model and the MSI visual encoder, responds to the SEG guiding token by extracting the hidden state vector of the large language model corresponding to the token, mapping this vector to a segmentation cue embedding vector, and fusing it with the multi-scale dense feature map. The final output is a pixel-level segmentation mask with the same spatial resolution as the original MSI data. The system can simultaneously output the structured text answer and the pixel-level segmentation mask.

[0063] A multimodal large-scale model-based intelligent analysis method for mass spectrometry imaging includes the following steps: First, a large-scale self-supervised pre-training of the MSI visual encoder is performed, enabling it to extract joint feature representations rich in spectral-spatial semantics from high-dimensional mass spectrometry imaging data. Second, multimodal joint training is performed on the pre-trained visual encoder, trainable adapter, and large language model. This training process is further divided into two sub-stages: The first stage uses MSI image-text pairs for modal alignment pre-training to achieve semantic alignment between the visual feature space and the language embedding space. The second stage uses a quadruple dataset (Q+A+I+M) containing natural language questions, corresponding answers, MSI image data, and pixel-level segmentation masks for instruction fine-tuning, enabling the model to learn to synchronously generate text descriptions and accurate segmentation masks according to user instructions. The segmentation mask is generated by a specially designed segmentation decoder using a special guiding token output by the large language model. Finally, the trained model is deployed on a computing device to receive MSI data and natural language queries uploaded by users, and outputs semantic segmentation results and structured text explanations in real time, thereby realizing intelligent and interactive analysis of mass spectrometry imaging data.

[0064] Specifically, the following steps are included:

[0065] Step S100: Self-Supervised Pre-training of the MSI Visual Encoder: The goal of this step is to enable the MSI visual encoder to fully learn the spectral-spatial joint semantic representation of mass spectrometry imaging (MSI) data through a self-supervised learning paradigm without manual annotation, providing a high-quality visual feature foundation for subsequent multimodal joint training. This step specifically includes the following three sub-steps, as follows: Figure 3 The complete sub-step process of MSI data preprocessing, tubular mask generation, physical sensing spectral-spatial joint encoding, and mask spectral reconstruction self-supervised training is shown.

[0066] S110MSI Data Preprocessing and Tubular Mask Generation

[0067] Before mass spectrometry imaging data is fed into the encoder, it must undergo a standardized preprocessing procedure to eliminate systematic biases between different instruments and different acquisition batches.

[0068] (a) Data preprocessing: For the raw MSI data, the following preprocessing operations are performed in sequence: (1) Baseline correction: Baseline subtraction is performed on the mass spectrometry signal of each pixel to remove the systematic increase of intensity value by instrument noise and chemical background. Common methods include local minimum method and wavelet transform method; (2) Peak alignment: Since there is a slight drift in the mass-to-charge ratio (m / z) axis between different collection batches, peak alignment is performed on all samples in the m / z dimension to ensure that the same metabolite corresponds to the same m / z index in all samples; (3) Intensity normalization: Total ion current (TIC) normalization is performed on the spectral vector of each pixel to eliminate the difference in matrix effect between samples; the normalization formula is:

[0069] j = 1, ..., C;

[0070] in, Let x(i) be the original intensity value of the i-th m / z channel after normalization, x(j) be the original intensity value of the i-th m / z channel, and C be the total number of channels.

[0071] (4) Logarithmic transformation: Apply a logarithmic transformation to the normalized intensity values. ,in A small constant (typically taking the value of ) is used to prevent numerical underflow. This is to compress the intensity dynamic range and alleviate the dominance of high-abundance metabolites on the training process;

[0072] (5) Spatial interpolation (optional): For MSI data with irregular spatial resolution or missing pixels, bilinear interpolation is used to resample the data to a uniform resolution to support batch training.

[0073] After preprocessing, each MSI sample is represented as a three-dimensional tensor I with shape (H, W, C), where H and W are the spatial height and width, respectively, and C is the number of spectral channels (m / z channels). Simultaneously, the m / z value vector v associated with this tensor, with length C, is retained. This vector represents the precise mass-to-charge ratio for the i-th channel and plays a crucial role in subsequent physical sensing spectral position encoding.

[0074] (II) Tube Mask Generation: This invention uses a tube masking strategy to generate the mask signal required for self-supervised training. The core idea is to randomly select a certain proportion of m / z channels and set the selected channels to zero at all spatial pixel positions, thereby forming a mask structure that is spatially consistent in the spectral dimension. Specifically, given the total number of channels C and the mask ratio r (r is 0.75 in this embodiment), the tube mask generation process is as follows: (1) Generate a uniform random noise vector n of length C, with each element taking values ​​in the range [0, 1); (2) Sort the noise vector and take the first floor(r * C) indices to form the set of masked channels. (3) Generate a Boolean mask vector mask with length C, where This indicates that channel i is masked. (4) Apply a mask to the original data tensor I to obtain the masked data. ,in For all i belonging to It is established, while the visible channel retains its original intensity value.

[0075] Tubular masking strategies have significant advantages over random pixel-wise masking: First, full-channel masking requires the model to infer the intensity distribution of the masked channel from the correlation between adjacent m / z channels, thus forcing the encoder to learn the inter-spectral structure rather than relying on local spatial texture; Second, spatial consistency ensures that the same m / z channel is masked simultaneously at all spatial locations, avoiding the model from "taking shortcuts" to complete reconstruction by using the same channel information of adjacent pixels, which significantly improves the learning difficulty of self-supervised signals and the generalization quality of feature representation.

[0076] S120 Physical Sensing Spectral-Spatial Joint Coding:

[0077] This sub-step constitutes the core of the MSI visual encoder, employing multi-level cascaded encoding modules to perform joint feature extraction of the spectral and spatial dimensions from the masked MSI data. The MSI visual encoder designed in this invention consists of the following five functional modules, the structure of which is detailed in the appendix to the specification. Figure 4The diagram illustrates the internal structure and connection method of five functional modules within the encoder: spatial downsampling convolution unit, physical sensing spectral position encoding unit, spatial position encoding injection unit, alternating spectral-spatial attention stacking unit, and dual-branch output unit.

[0078] (a) Spatial downsampling convolutional unit: To solve the problem of computational complexity explosion in pixel-by-pixel spectral modeling on high-resolution MSI data, the MSI visual encoder first compresses the original spatial resolution to 1 / s of the original resolution through a spatial downsampling convolutional unit (in this embodiment, the downsampling step size s=4).

[0079] Specifically, this unit consists of two concatenated convolutional layers: the first layer is a 1x1 convolutional layer that compresses the number of input channels C to an intermediate dimension D / 2, where D is the encoder feature dimension (D=512 in this embodiment); the second layer is an s×s convolutional layer with a stride of s, which reduces the spatial resolution from H×W to (H / s)×(W / s) while expanding the number of channels from D / 2 to D. Each convolutional layer is followed by a GroupNorm normalization layer and a GELU activation function. This operation reduces the spatial pixel sequence length from L=H×W to L'=H / s×W / s, and all subsequent sequence-based attention operations are applied to this compressed spatial sequence, thereby reducing the computational complexity to the original value. times.

[0080] (ii) Physical sensing spectral position encoding unit: This unit is one of the core innovative modules of this invention. It aims to explicitly inject the physicochemical prior information contained in the m / z value into the encoder in the form of a continuous vector, so that the encoder can sense the chemical identity of each spectral channel rather than just relying on its index number.

[0081] Specifically, for the i-th component in the m / z value vector v First, normalization is performed:

[0082] ;

[0083] in and These are the minimum and maximum values ​​of the current sample's m / z value, respectively. This is the numerical stability constant. After normalization... If it belongs to the interval [0, 1], it indicates the relative position of the channel within the entire m / z range.

[0084] Subsequently, a log-uniformly distributed multi-frequency Fourier encoding (referencing the NeRF positional encoding concept) is applied to the normalized m / z value to generate Fourier feature vectors:

[0085] ;

[0086] Where K is the number of frequencies (K=64 in this embodiment). The dimension is 2K. The logarithmically uniform frequency setting allows low-frequency components to capture the global relative position of m / z, while high-frequency components capture the subtle differences between adjacent m / z channels.

[0087] Fourier feature vectors are mapped to D / 2 dimensions via a linear projection layer: ,in Let be the m / z position encoding vector for channel i, with dimension D / 2. The projection weight matrix is ​​a learnable linear projection matrix with dimensions (D / 2) × 2K, and all visible spectral channels share the parameters of this projection layer. This is the corresponding bias vector, with a dimension of D / 2. All visible spectral channels share this projection layer parameter.

[0088] Simultaneously, the intensity values ​​of the visible channel are mapped to D / 2-dimensional intensity features via independent linear projection layers: ,in, This is the intensity encoding vector for the channel, with a dimension of D / 2. Let be the normalized intensity value of the i-th visible channel, with dimension D / 2. The strength projection weight matrix is ​​a learnable matrix with dimensions (D / 2) × 1. This is the corresponding bias scalar.

[0089] Finally, the m / z position encoding and intensity encoding are concatenated and then fused through a projection layer to generate each spectral token: ,in, Let be the spectral token vector for each channel, carrying both the m / z physical position information and ion intensity information for that channel. Its dimension is D, and `concat(·,·)` represents the vector concatenation operation. After performing the above operation on all visible channels, a spectral token sequence T is obtained, with the shape [formula missing]. ,in This represents the number of visible channels.

[0090] In addition, this unit also constructs an m / z correlation bias matrix: for any two visible channels i and j, the normalized m / z difference is calculated. The value is mapped to a scalar bias value via a lightweight multilayer perceptron. This forms the bias matrix B, with the shape of... This matrix is ​​added as a prior bias to the attention weights in subsequent spectral attention calculations, enabling the model to perceive stronger correlations between chemically similar m / z channels from the early stages of learning.

[0091] (III) Spatial Position Encoding Injection Unit: To enable the encoder to perceive the geometric position of each spatial patch within the tissue slice, this unit applies learnable position encoding to the downsampled spatial grid coordinates. Specifically, for each position (h, w) in the downsampled spatial feature map, a normalized coordinate vector is generated:

[0092] ;

[0093] Normalized coordinates are mapped to spatial location encoding vectors through a two-layer multilayer perceptron (hidden layer dimension D / 2, activation function GELU). The dimension is D, and it is superimposed on each spatial location of the spectral token sequence in a broadcast manner, so that the spectral token carries spatial location information at the same time, providing geometric prior for the subsequent spatial attention module.

[0094] (iv) Alternating Spectral-Spatial Attention Stacking Unit: This unit consists of N interleaved attention blocks connected in series (N=4 in this embodiment). Each block contains a spectral self-attention sub-block, a spatial self-attention sub-block, and a feature broadcast fusion unit in sequence. The three sub-blocks work alternately to achieve layer-by-layer deep fusion of spectral and spatial information.

[0095] Spectral self-attention sub-blocks: For each downsampled spatial location (L' in total), the independent visible spectral channel sequence (length) at that location. Multi-head self-attention is performed on the [aspect / time frame]. The m / z correlation bias matrix B is superimposed on the attention weight matrix, enabling the attention mechanism to capture data-driven inter-spectral correlations while incorporating m / z physical prior information. The attention calculation formula is:

[0096] ;

[0097] Where Q, K, and V are the query, key, and value matrices, respectively. Let B be the m / z correlation bias matrix for each attention head. By performing self-attention on the spectral dimension, features at each spectral location are aggregated with contextual information from other m / z channels, thereby learning biochemical patterns such as co-localization relationships and isotopic patterns among metabolites. After spectral attention, [the following text is incomplete and likely refers to a separate process:] The dimension performs mean pooling to aggregate the spectral token sequence at each spatial location into a single pixel feature vector, resulting in a spatial feature sequence with shape (L', D).

[0098] Spatial self-attention sub-blocks: with As input, standard multi-head self-attention operation is performed, enabling features at each spatial location to aggregate contextual information from other spatial locations, thereby capturing long-range spatial dependencies between different anatomical regions in a tissue slice, such as metabolic gradient changes at the boundary between tumor regions and normal tissue.

[0099] Feature broadcast fusion unit: Spatial feature sequence resulting from spatial self-attention output enhanced with spatial relation enhancement. The sequence is expanded back to the spectral dimension via a broadcast operation, residual superposition is performed with the original spectral token sequence, and then normalized using LayerNorm to obtain an updated feature tensor for use in the next alternation block. Each self-attention sub-block adopts a standard Pre-Norm structure (normalization → attention / FFN → residual connection), and Dropout (dropout rate of 0.1) is applied after the attention layer and the feedforward network layer to prevent overfitting.

[0100] After processing through N alternating attention blocks, the final spatial feature map is output, denoted as . Its shape is (D, H / s, W / s).

[0101] (v) Dual-branch output: After processing by N alternating attention blocks, the encoder outputs two features, which are used by different modules in subsequent multimodal training.

[0102] Branch 1 – Visual Token Sequence: Employing a cross-attention pooling mechanism to... A learnable query token (in this embodiment) =256) is the query endpoint, and the final spatial feature map is used as the query endpoint. For the key-value side, a fixed-length visual token sequence is extracted through a multi-head cross-attention operation. , shape is Compared to deformable pooling methods based on grid sampling, cross-attention pooling has fully learnable parameters, more stable gradient flow, and can adaptively focus on the most information-rich spatial regions within the organization. The output of this branch will be projected onto the language embedding space via a visual adapter in step S200 for processing by a large language model.

[0103] Branch Two – Multi-scale Dense Feature Map: This involves converting the final spatial feature map... The feature maps (with shapes of (D, H / s, W / s)) are processed by a Feature Pyramid Network (FPN). Spatial resolution is gradually restored through deconvolutional upsampling, generating dense feature maps at three scales. The resolution is progressively increased along the upsampling direction, corresponding to spatial resolutions of (H / s)×(W / s), (H×2 / s)×(W×2 / s), and (H×4 / s)×(W×4 / s), ultimately approaching the spatial resolution of the original MSI data. This multi-scale feature map preserves the fine spatial structure information of the organization and will work in conjunction with the guiding token in the segmentation decoder in step S200 to generate a high-precision pixel-level segmentation mask.

[0104] S130 Masked Spectral Reconstruction Self-Supervised Training: This sub-step describes the pre-training objective and training strategy of the MSI visual encoder. During the self-supervised pre-training phase, a lightweight reconstruction decoder is attached to the encoder (discarded after pre-training) to form a complete Masked Autoencoder (MAE), with masked spectral reconstruction as the self-supervised learning objective.

[0105] (a) Reconstruction Decoder: The reconstruction decoder consists of two fully connected layers with GELU activation function. The decoder receives the spatially dense feature map from the dual-branch output of the encoder. (with a shape of D×(H / s)×(W / s)), it is first upsampled to the original resolution H×W by bilinear interpolation, and then mapped to the full number of spectral channels C by a fully connected layer. The output reconstructed tensor R has a shape of (H, W, C), which represents the intensity value prediction for all channels of the original MSI data.

[0106] (ii) Reconstruction loss function: This invention uses a combined loss function to supervise reconstruction quality. It only calculates the loss of the masked channel position and does not impose loss constraints on the reconstruction results of the visible channel, so as to force the encoder to infer the masked channel from the visible channel information instead of directly memorizing the input.

[0107] Combination loss The loss is a weighted sum of the mean squared error loss (MSE Loss) and the spectral angle mapping loss (SAM Loss), and the calculation formula is as follows:

[0108] ;

[0109] In this embodiment, the weighting coefficient alpha is set to 0.7.

[0110] Mean squared error loss is calculated to determine the pixel-level mean squared error between the reconstructed value and the original value in the masked channel:

[0111] ;

[0112] in, It is the Frobenius norm. The number of channels being masked.

[0113] Spectral angle mapping loss measures the similarity of spectral morphology by calculating the angle between the reconstructed spectral vector and the original spectral vector. It is insensitive to the absolute value of intensity and focuses on the fidelity of the relative distribution between spectra.

[0114] ;

[0115] in, and These are the intensity vectors of the reconstructed spectrum and the original spectrum at pixel (h,w) in the masked channel, respectively. The combination of MSE loss and SAM loss allows the model to simultaneously optimize intensity accuracy (MSE) and spectral morphology fidelity (SAM). These two types of constraints complement each other and help the encoder learn a more discriminative spectral-spatial joint representation.

[0116] (III) Training Strategy: The optimizer used is AdamW, with a weight decay coefficient set to 0.05; the learning rate is cosine annealing scheduling, with a peak learning rate of 1e. -4 The warm-up phase accounts for 5% of the total training steps; the parameters of the physical perception spectral position encoding unit are updated using 1 / 10 of the peak learning rate to maintain its encoding stability. The batch size is dynamically adjusted based on the GPU memory, and gradient accumulation is used to ensure that the equivalent batch size is not less than 64.

[0117] To further improve the generalization ability of the encoder, the following data augmentation operations are applied to the MSI data during training: random horizontal and vertical flips (each with a flip probability of 0.5); random rotation (rotation angles are uniformly sampled from {0 degrees, 90 degrees, 180 degrees, 270 degrees}); and random m / z channel discarding (randomly setting the intensity of some visible channels to zero with a probability of 0.1 to simulate instrument noise).

[0118] After pre-training, the reconstructed decoder is discarded, and the encoder weights are used as initialization parameters for multimodal joint training in step S200. It is expected that the MSI visual encoder, pre-trained under self-supervision in this step, will provide higher-quality visual feature initialization for subsequent multimodal joint training, helping to accelerate convergence and improve the overall performance of the system on visual question answering and segmentation tasks.

[0119] Step S200: Joint training of mass spectrometry-language multimodal model

[0120] This step begins with the visual encoder weights pre-trained in step S100. Through three sequentially executed sub-steps, it completes the preparation of training data, initial alignment of visual and language modalities, and joint optimization of multiple modules across the entire system. Ultimately, it integrates the visual encoder, visual adapter, large language model, and segmentation decoder into a complete multimodal system with natural language interaction and pixel-level segmentation capabilities (e.g., ...). Figure 5 The execution order, parameter freezing strategy, and data flow of the three sub-steps (training data preparation, visual adapter modality alignment training, and multi-module joint instruction fine-tuning) are illustrated. The logical relationship of the three sub-steps is as follows: S210 completes data preparation without involving model training; S220 completes the first training stage using the data constructed in S210; S230 completes the second training stage based on the weights in S220, introduces the segmentation decoder, and opens up joint optimization of each module.

[0121] S210 Training Data Preparation: This sub-step completes the construction and organization of all training data before the formal training begins, providing a data foundation with controllable quality and a clear structure for the subsequent two training phases. This invention divides the training data into three subsets, with the constituent elements and construction costs of each subset increasing sequentially. The construction scale can be flexibly selected according to actual resource conditions. The structural relationship between the three subsets is detailed in the appendix to the specification. Figure 6 It shows the constituent elements of three types of training subsets (pure dialogue subset, visual question answering subset, and segmentation reasoning subset), the distinction between required and optional fields, and the organization of the three types of subsets converging into the joint training dataset.

[0122] (a) Subset 1: Pure Dialogue Subset

[0123] The pure dialogue subset consists of natural language questions and corresponding answers, without containing any MSI image data or segmentation mask annotations. It represents the lowest-cost and most scalable subset among the three types. This subset originates from two sources: first, question-and-answer pairs related to mass spectrometry, metabolomics, and pathological analysis are selected from existing publicly available general biomedical question-and-answer datasets and used directly after format standardization; second, professional question-and-answer pairs covering metabolite function explanations, mass spectrometry principles, and pathological diagnostic logic are automatically generated using a general large language model, based on core literature and textbooks in the MSI field, and are included in the training set after manual sampling verification.

[0124] The core role of the pure dialogue subset is to maintain the general dialogue capability of the large language model during the joint training phase, preventing the model from degenerating into a specialized system that can only handle fixed-format inputs due to excessive focus on visual understanding tasks, i.e., preventing the so-called "forgetting" phenomenon. This subset accounts for 50% of the total training data during mixed training.

[0125] (ii) Subset 2: Visual Question-and-Answer Subset

[0126] The visual question-answering subset consists of an MSI data cube, a corresponding m / z value vector, natural language questions, and text answers, but does not include a segmentation mask. This subset is the core data source for achieving visual-language modality alignment and is also the main training data for the first training phase of step S220.

[0127] Due to the extreme scarcity of image-text pairs specifically for MSI data in existing publicly available resources, this invention proposes a low-cost automated construction process. First, traditional dimensionality reduction methods such as principal component analysis or nonnegative matrix factorization are performed on the raw MSI data to extract spatial distribution maps of several major components and ion images of the top m / z channels with the highest signal intensity, serving as a rough description of the tissue structure and metabolite distribution of the MSI sample. Then, the above analysis results, along with basic metadata of the sample (including instrument type, tissue origin, m / z value range, etc.), are organized into structured text as prompt words and input into a general large language model, which then generates a natural language description covering tissue region division, metabolite spatial distribution characteristics, and potential biological significance. Finally, the generated text is quality filtered, removing samples with extremely low information content or obvious errors, retaining high-quality image-text pairs. Furthermore, samples already described in literature in publicly available MSI databases (such as METASPACE, PRIDE, etc.) can be directly used as pairing corpora using the original descriptive text, without the need for regeneration. The visual question-answering subset accounts for 30% of the total training data during hybrid training.

[0128] (III) Subset Three: Segmentation Reasoning Subset

[0129] The segmentation inference subset consists of an MSI data cube, a corresponding m / z value vector, a natural language question, a text response containing a SEG guiding token, and a corresponding pixel-level segmentation mask, which is the complete four-tuple structure defined in this invention. This subset is the most costly part to construct among the three subsets. Mask annotation must be completed by domain experts with dual knowledge of mass spectrometry and pathology. For each sample, the target region needs to be manually delineated on the spatial image of the MSI data and a pixel-level binary mask needs to be generated.

[0130] To reduce the cost of manual annotation, this invention proposes an active learning strategy to assist in prioritizing annotations: first, train an initial segmentation model with a small number of labeled samples, and calculate the uncertainty of the model's prediction results for unlabeled samples; prioritize submitting samples with the highest uncertainty for expert annotation, and then update the model with newly labeled samples; iterate in this way to obtain the maximum performance gain with the minimum amount of annotations. The question types in the segmentation inference subset should cover various query formats, such as region identification (e.g., "Please mark the tumor necrosis region"), channel localization (e.g., "Please mark the region with lipid signal enrichment corresponding to m / z=741.5"), and comparative analysis (e.g., "Please mark the region with the most significant metabolic difference from adjacent normal tissue"), to ensure that the model has the ability to segment diverse user commands. This subset accounts for 20% of the total training data during hybrid training.

[0131] (iv) Standardized and uniform data formats

[0132] All three subsets are stored and loaded using the same data format to support mixed processing within the same training framework. Each sample is stored as a dictionary containing the following fields: question text (required), answer text (required; SEG guide token placeholders are inserted after the statements describing the target region in the answer of the segmented sample's answer), MSI data path (optional), m / z value vector (optional, provided along with the MSI data), and mask annotation path (optional). During loading, for samples with missing fields, the corresponding model branch is automatically skipped from calculation, and the corresponding loss term is automatically set to zero, ensuring seamless mixing of the three types of samples in the same batch.

[0133] S220 Visual Adapter Modal Alignment Training: This sub-step is the first training phase. Utilizing subsets one and two constructed in S210, and with both the MSI visual encoder and the large language model in a frozen state, the visual adapter is specifically trained. This trains the adapter to learn to convert the visual token sequences output by the MSI visual encoder into language-side embedding representations that the large language model can directly understand, achieving preliminary semantic alignment between the visual feature space and the language embedding space. This phase does not involve a segmentation decoder, focusing solely on the single objective of modal alignment. Subset one is processed in plain text mode in this phase, without going through the visual adapter. Its role is to maintain the dialogue capability baseline of the large language model and prevent pure visual samples from causing model shifts. The image-text pairs in subset two serve as the core supervision signal for adapter training.

[0134] (i) Parameter Freezing Strategy: In the first training phase, all parameters of the visual encoder are frozen to protect the spectral-spatial joint representation learned in step S100 from being destroyed by stochastic gradient updates. All parameters of the large language model are also frozen, with its pre-trained language understanding and generation capabilities serving as a fixed target space to guide the adapter in learning the correct mapping direction. With both of these large modules frozen, the visual adapter is the only module participating in gradient updates in this phase. Its parameters are initialized randomly and gradually converge to a state that effectively bridges the two modal feature spaces through a large number of image-text pair supervision signals.

[0135] (ii) Visual Adapter Structure: The function of the visual adapter is to transform the visual token sequence output by the encoder (each token has a dimension of D) into an embedding sequence acceptable to the input side of the large language model (each token has a dimension of D). , (This refers to the hidden layer dimension of a large language model). In this invention, the visual adapter adopts a two-layer linear projection structure, connected by a GELU activation function. Specifically, the input linear layer maps the dimension from D to twice its original value. After GELU activation, the output linear layer doubles its dimension. Mapped to Finally, the output is normalized using LayerNorm. This structure has a lightweight parameter set and fast convergence speed, making it suitable for initial alignment with limited image-text pair data. In practical implementations, the adapter structure can be replaced with more complex cross-modal interaction structures such as query Transformer to further improve alignment quality when data is sufficient. This invention does not impose restrictive constraints on the specific structure.

[0136] (III) Training Input Construction: The model input for each training sample is constructed as follows: the MSI data is processed by the frozen visual encoder to extract the visual token sequence, which is then projected onto the language embedding space by the visual adapter to obtain the visual-side embedding sequence; the paired text description is converted into a language-side embedding sequence by the word embedding layer of the large language model; the visual-side embedding sequence is concatenated to the front of the language-side embedding sequence to form a multimodal input sequence, which is then fed into the frozen large language model. For pure dialogue samples in subset one, there is no visual-side embedding, and the text sequence is used directly as input, so that the adapter has no effect on the dialogue samples and maintains the original dialogue capability benchmark of the language model.

[0137] (iv) Training Objectives and Optimization Configuration: The training objective in this stage is to use standard autoregressive language modeling loss, which aims to maximize the conditional probability of the next text token in the target response sequence, given the visual embedding sequence and the generated text tokens. The loss is calculated only at the text token position, without imposing loss constraints on the visual token position.

[0138] The optimizer uses AdamW, and the peak learning rate is set to... Cosine annealing scheduling is employed, with a warm-up step count of 3% of the total training steps and a weight decay coefficient of 0.05. Since this stage only updates the lightweight visual adapter module, the required computational resources are significantly lower than full-parameter fine-tuning, allowing convergence to be completed in a shorter time. After this stage of training, the weights of the visual adapter are used as the initialization parameters for the second training stage of S230, ensuring that joint training starts from a base state with preliminary alignment capabilities, accelerating subsequent convergence and improving final performance.

[0139] S230 Multi-Module Joint Instruction Fine-tuning: This sub-step is the second training phase. Building upon the visual adapter alignment weights completed in S220, a segmentation decoder is introduced, removing the freezing restrictions of each module. End-to-end joint optimization is performed on the entire system using mixed data from three subsets, enabling the model to ultimately possess the complete ability to respond to diverse natural language instructions, simultaneously generate text responses, and display pixel-level segmentation masks. This phase is the most technically complex and module-interactive part of the entire training process. The following details the hierarchical parameter update strategy, the segmentation decoder structure and working mechanism, the joint loss function design, key engineering implementation points, and the training completion determination.

[0140] (i) Hierarchical parameter update strategy: In the second stage, a differentiated parameter update method is adopted based on the parameter scale and functional positioning of each module, so as to maximize the overall system capability while effectively controlling the computational overhead and overfitting risk.

[0141] (1) Visual encoder: Unfreeze and participate in gradient update in full parameter mode, but set the learning rate to 0.1 times the base learning rate. The basis for this design is that after the encoder is fully pre-trained in step S100, it has a stable spectral-spatial joint representation capability; using a smaller learning rate for fine adjustment allows the encoder to further optimize the feature representation based on the feedback from downstream segmentation and question answering tasks, while avoiding "catastrophic forgetting" caused by large gradient updates, which would destroy the spectral structure knowledge learned in pre-training.

[0142] (2) Large Language Model: Low-rank adaptation (LoRA) is used for efficient parameter fine-tuning. Low-rank decomposition matrices are injected only into the self-attention modules (query matrix Q and value matrix V) of each Transformer layer, with a rank of 16 and a scaling factor of 32. The original weights of the large language model are frozen during training, and only the low-rank increment matrices injected by LoRA participate in gradient updates. The number of newly trainable parameters is approximately 0.5% to 1% of the original model's parameters. This approach significantly reduces memory usage and computational overhead while enabling the language model to adapt to the specialized vocabulary and instruction-following style of the MSI domain, while retaining its original general language understanding and generation capabilities.

[0143] (3) Visual adapter: Continue to update in a full-parameter manner, using a base learning rate, and further refine the mapping of visual features to language space in joint training to adapt to the dual needs of instruction compliance and segmentation guidance.

[0144] (4) Segment Decoder: Introduced for the first time in this stage, it starts from random initialization and is trained in a full-parameter manner, using a base learning rate.

[0145] (II) The structure and working mechanism of the segmentation decoder are as follows: The segmentation decoder is one of the core innovative modules of this invention. Its design concept is to transmit the semantic understanding of the user query command by the large language model to the visual side in the form of a hidden state vector, thereby driving the visual features to complete accurate pixel-level segmentation. The segmentation decoder consists of three sequentially connected sub-modules.

[0146] The first submodule is the SEG token extraction and projection module. This invention adds a special SEG guiding token to the vocabulary of the large language model. During autoregressive decoding, when the large language model determines that the current answer requires a segmentation result, it generates this SEG guiding token at the corresponding position in the answer text. After detecting the SEG guiding token, the system extracts the hidden state vector corresponding to the output position of the last Transformer layer of the large language model. The dimension of this hidden state vector is consistent with the hidden layer dimension of the large language model, carrying the model's complete semantic understanding of the target region specified by the user. Subsequently, this hidden state vector is projected through a two-layer multilayer perceptron to become a segmentation cue embedding vector, whose dimension is adjusted to be consistent with the feature dimension D of the visual encoder, serving as the semantic guidance signal input for the segmentation decoder. This projection layer is a crucial bridge connecting language modality understanding and pixel-level visual segmentation.

[0147] The second submodule is the semantic guidance feature fusion module. The segmentation cue embedding vector and the multi-scale dense feature map output by the visual encoder are fused in this module. Specifically, the segmentation cue embedding vector (a one-dimensional vector of dimension D) is first aligned with the number of channels in the finest-grained dense feature map in the channel dimension. Then, it is broadcast and expanded along the spatial dimension to make its shape consistent with the finest-grained dense feature map (shape D×(H / s)×(W / s)). It is then added element-wise to this feature map, and the feature is refined through a lightweight convolutional layer to obtain a spatial feature map that incorporates semantic guidance information. The remaining scale-dense feature maps from the MSI visual encoder FPN are injected progressively during subsequent upsampling via skip connections, ensuring that the segmentation result possesses both high-level semantic accuracy and low-level spatial fineness.

[0148] The third submodule is the mask generation module. The fused feature map is upsampled through multiple levels of deconvolution to gradually restore the spatial resolution of the original MSI data. After each upsampling operation, BatchNorm normalization and ReLU activation function are applied, and finally, a pixel-level segmentation mask is output after a 1x1 convolutional layer and Sigmoid activation function. The value of each pixel in the mask represents the confidence that the pixel belongs to the target region specified by the user query, with a value ranging from 0 to 1. During the inference stage, a confidence threshold of 0.5 is used to segment the data. Figure Two Value-enhanced, the final segmentation result is obtained.

[0149] (III) Joint loss function design: The joint loss function in the second stage consists of a weighted sum of language modeling loss and segmentation loss, which performs end-to-end joint optimization of the entire system.

[0150] The language modeling loss is applied to the text token generation process of the large language model, and its calculation method is consistent with that of standard autoregressive language modeling, which is to maximize the conditional probability of each text token in the target response sequence. For segmentation samples containing SEG guide tokens, the language modeling loss is also applied to the prediction position of the SEG token, ensuring that the model learns to output the SEG guide token instead of ordinary text tokens at the appropriate time, that is, to enable the model to learn the judgment ability of "when to segment".

[0151] The segmentation loss is calculated only for the subset of three samples containing mask annotations and is composed of a weighted sum of binary cross-entropy loss and Dice loss. The binary cross-entropy loss calculates the difference between the predicted confidence and the labeled value pixel-by-pixel, making it sensitive to pixel-level prediction accuracy. The Dice loss is optimized based on the overlap between the predicted and labeled regions, exhibiting stronger robustness in cases with a very small proportion of foreground pixels (such as locating a small abnormal metabolic region within a large area of ​​normal tissue), effectively mitigating class imbalance. Each of the two segmentation losses accounts for 50% of the total segmentation loss. For subsets one and two samples without mask annotations, the segmentation loss term is automatically masked to zero, generating no gradient and not affecting the update direction of modules other than the segmentation decoder.

[0152] The final joint loss is calculated by summing the language modeling loss and segmentation loss with a weight of 1.0. This equal-weight design ensures that language understanding and segmentation generation tasks receive equal attention during joint training. In practice, the weight ratio can be adjusted appropriately based on the different emphases of downstream applications on the two capabilities; this invention does not impose any restrictive constraints on this.

[0153] (iv) Key points of engineering implementation: To ensure the stability and efficiency of the second phase of joint training, the following key design measures are adopted at the engineering implementation level.

[0154] (1) Learning rate grouping settings: The base learning rate is set to The MSI visual encoder uses 0.1 times the base learning rate. The visual adapter, LoRA increment matrix, and segmentation decoder all use the base learning rate. All parameter sets use a uniform cosine annealing schedule, with a warm-up step count of 3% of the total training steps.

[0155] (2) Gradient clipping: Apply global gradient clipping with a maximum norm of 1.0 to all trainable parameters to prevent large gradients backpropagated from the segmentation loss from destroying the existing representations of the visual encoder and adapter in the early stages of training (when the output quality of the segmentation decoder is low).

[0156] (3) Mixed precision training: The BFloat16 mixed precision format is adopted to reduce the memory usage. The normalization layer of the visual encoder and the joint loss calculation maintain Float32 precision to ensure numerical stability.

[0157] (4) Intra-batch mixed sampling: The three types of training subsets are randomly sampled and mixed in each training batch at a ratio of 50%, 30%, and 20%. Different types of samples participate in forward and backward propagation in the same batch. The missing loss terms corresponding to each type of sample are automatically set to zero to ensure the uniformity of the training framework.

[0158] (5) Checkpoint selection strategy: Save the complete model checkpoint every fixed number of training steps (e.g., 500 steps). Use the weighted comprehensive index of visual question answering BLEU-4 score and segmentation average intersection-union ratio (mIoU) on the validation set as the model selection criterion. The best checkpoint is then entered into the deployment process in step S300.

[0159] (v) Second-stage training completion criteria: The second-stage training is considered complete and can proceed to step S300 when the following three conditions are met simultaneously: First, the joint loss decreases by less than 0.1% over 20 consecutive evaluation periods on the validation set, indicating that the model has converged; Second, the average intersection-union ratio of the segmentation task on the validation set reaches the preset performance threshold; Third, the answer quality of the visual question answering task is not lower than the baseline level at the end of stage S220, confirming that joint training has not led to a significant degradation in dialogue ability.

[0160] After the second phase of training is completed, the system saves the following four sets of component weights for subsequent deployment: full parameter weights for the visual encoder; full parameter weights for the visual adapter; original weights and LoRA incremental weights for the large language model, which are merged before deployment—for each weight matrix W injected with LoRA, the product BA of its corresponding low-rank decomposition matrices A and B is directly added to W to obtain the merged complete weight matrix W' = W + BA. Only W' is retained after merging, eliminating the side-channel computation overhead during inference; and full parameter weights for the segment decoder. These four sets of weights together constitute the complete mass spectrometry imaging intelligent question answering system deployed in step S300.

[0161] Step S300: Model Deployment and Intelligent Interactive Analysis: Based on the multimodal model trained in Step S200, this step integrates, deploys, and enables interactive use of the system. This ensures the trained model runs stably in a real-world computing environment, responds to user-uploaded MSI data and natural language query commands, and outputs semantic segmentation results and structured text explanations in real time, achieving intelligent and interactive analysis of mass spectrometry imaging data. This step includes four sub-steps, corresponding to deployment preparation, service-oriented deployment, inference execution, and result presentation. For the overall inference process, please refer to the appendix of the instruction manual. Figure 7 The diagram illustrates the complete data flow of the inference phase, from user input reception, MSI visual encoding, multimodal fusion and autoregressive decoding, SEG token-triggered segmentation decoding, to the final output of text response and pixel-level segmentation mask, and annotates the conditional triggering logic of the visual encoding path and the segmentation decoding path.

[0162] S310 Model Integration and Deployment Preparation: Before deploying the trained model to the inference environment, preparations such as integrating the weights of each module, converting the accuracy, and optimizing the inference process must be completed to ensure that the system balances inference speed and output quality in the actual deployment environment.

[0163] (I) Module weight integration

[0164] After training in step S200, the system saves four independent sets of module weights: the full parameter weights of the visual encoder, the full parameter weights of the visual adapter, the original weights and LoRA incremental weights of the large language model, and the full parameter weights of the segmentation decoder. Before deployment, the following integration operation needs to be performed on the above weights.

[0165] First, the LoRA incremental weights of the large language model are merged into the original weights. Specifically, for each weight matrix W injected with LoRA, the product BA of its corresponding low-rank decomposition matrices A and B is directly added to W, resulting in the merged complete weight matrix W' = W + BA. Only W' is retained after merging. This merging operation eliminates the extra overhead of LoRA bypass computation during inference, making the inference speed exactly the same as before LoRA was introduced, without losing any fine-tuning effects. After merging, the LoRA incremental weight files can be deleted, retaining only the merged complete language model weights.

[0166] Secondly, the visual encoder, visual adapter, merged large language model weights, and segment decoder weights are organized into a single model weight file or a standardized model directory structure, and saved together with the model configuration file (which records information such as network structure hyperparameters, feature dimensions, and number of attention heads for each module) to ensure that the deployment environment can unambiguously load and reconstruct the complete model structure.

[0167] (ii) Inference accuracy configuration: To meet the different requirements of inference speed and numerical accuracy in different deployment scenarios, the present invention supports the following accuracy configuration schemes.

[0168] (1) Full-precision inference (Float32): Suitable for scientific research analysis scenarios with extremely high numerical accuracy requirements. It provides fine segmentation mask boundaries and accurate numerical calculation of metabolite intensity, but has high memory usage and inference latency. It is recommended to use it on workstations or servers equipped with high-memory GPUs (memory not less than 40GB).

[0169] (2) Half-precision inference (BFloat16 or Float16): While maintaining the output quality essentially unchanged in most scenarios, it reduces memory usage by about 50% and increases inference speed by about 1.5 to 2 times. It is suitable for most scientific research and clinical auxiliary analysis scenarios and is the recommended default precision configuration. The normalization layer of the visual encoder maintains Float32 precision to maintain numerical stability.

[0170] (3) Quantization Inference (INT8 or INT4): Integer quantization is performed on the weights of the large language model, which can further reduce the memory usage to 25% to 50% of the original, enabling the system to run on consumer-grade GPUs (with at least 16GB of video memory), suitable for resource-constrained edge deployment scenarios. Quantization is only applied to the large language model, and the visual encoder and segmentation decoder maintain BFloat16 precision to maintain the quality of visual understanding and segmentation output.

[0171] (III) Inference acceleration optimization: To further improve the system's response speed in actual use, the following inference acceleration technologies can be selectively applied.

[0172] (1) Key-Value Cache: During the autoregressive decoding process of large language models, the key-value vectors corresponding to the generated tokens are cached to avoid repeated calculations and significantly reduce the stepwise decoding latency in long text response scenarios. The key-value cache corresponding to the visual token sequence can be reused after the first forward propagation, which has a particularly significant acceleration effect on inference requests containing long visual sequences.

[0173] (2) Parallelization of visual encoding: For scenarios that process multiple MSI samples in batches, the visual encoding stage (encoder forward propagation and adapter projection) can be executed in parallel on the GPU, making full use of the GPU's massive parallel computing capabilities and reducing the average encoding latency of a single sample.

[0174] (3) Computation graph compilation optimization: Use tools such as PyTorch compiler or TensorRT to perform static compilation optimization on the computation graph of visual encoder and segment decoder to eliminate the extra overhead of dynamic graph execution, and obtain additional inference speed improvement in fixed input resolution scenarios.

[0175] S320 System Deployment and Service-Oriented Development: This sub-step deploys the integrated model as an inference service accessible to users, supporting both local stand-alone deployment and remote server deployment modes to meet the needs of different use cases.

[0176] (I) Hardware Environment Requirements: The minimum hardware configuration requirements for system deployment are as follows: Processor: at least an 8-core 64-bit CPU; Memory: at least 32GB; Graphics card: NVIDIA GPU supporting CUDA, with at least 16GB of VRAM (when using INT8 quantization) or at least 24GB (when using BFloat16 configuration); Storage space: at least 100GB (for storing model weight files and temporary data). A dedicated inference server with a high-performance GPU equipped with 80GB of VRAM is recommended to support full-precision inference and large-batch concurrent processing. Operating system support: Linux (Ubuntu 20.04 and above recommended) and Windows 10 and above.

[0177] (II) Software Operating Environment: The system relies on the following software environment: Python 3.9 or later; CUDA 11.8 or later and the corresponding version of cuDNN; PyTorch 2.0 or later, a deep learning framework; and several dependent libraries, including pyimzML or a corresponding format support library for MSI data format parsing, NumPy and SciPy for numerical computation, OpenCV for image processing, and web frameworks such as FastAPI or Flask for service-oriented deployment. These dependencies are installed uniformly using standard Python package management tools, supporting containerized deployment (such as Docker) to ensure environment consistency and portability.

[0178] (III) Local Standalone Deployment Mode: The local standalone deployment mode is suitable for scenarios where researchers conduct offline analysis on their personal workstations. In this mode, users interact with the system through a local graphical interface or command-line interface. MSI data does not need to be uploaded to an external server, ensuring high data privacy and making it suitable for processing sensitive clinical sample data. After system startup, the model weights are loaded into the GPU memory all at once. Subsequent analysis requests execute inference directly on the already loaded model, avoiding startup delays caused by repeatedly loading weights. The local mode supports queued batch processing, allowing users to submit multiple MSI files and corresponding query batch processing tasks. The system automatically completes the analysis in sequence and writes the results to the specified output directory.

[0179] (iv) Remote Server Deployment Mode: The remote server deployment mode is suitable for scenarios involving concurrent access by multiple users or the processing of ultra-large-scale MSI datasets. In this mode, the system provides an inference interface to the outside world in the form of a standard RESTful API service. Users submit MSI data and query commands to the server in a standardized format over the network. After the server completes the inference, it returns the structured results to the client.

[0180] The service interface design follows these specifications: The data upload interface receives upload requests for standard MSI data format files (supporting common formats such as imzML and mzML) and accompanying m / z value files, and returns a unique session identifier; the query interface receives the session identifier and natural language query text, triggers the inference process, and asynchronously returns the analysis results; the result acquisition interface supports returning structured text answers in JSON format, returning segmentation mask overlay images in standard image format (PNG or TIFF), and returning raw mask data in numerical array format for further processing by the user.

[0181] In terms of concurrency control, the server maintains an inference request queue and schedules execution according to the first-in-first-out principle. For scenarios with sufficient computing resources, multi-GPU parallel inference can be configured to improve system throughput. Each inference request is equipped with a timeout protection mechanism, which automatically releases the occupied resources and returns a timeout prompt to the user after the timeout.

[0182] S330 User Interaction and Inference Execution: This sub-step describes the complete inference execution process after the system receives user input. See the attached instruction manual for details. Figure 7 The entire process from input reception to output generation is handled in the inference flow, which consists of three sequentially executed stages: input preprocessing, model forward propagation, and output generation.

[0183] (a) Input Reception and Preprocessing: The system receives two types of user input: MSI data (optional) and natural language query commands (required). The following preprocessing operations are performed on each type of input.

[0184] For MSI data input, the system first parses the user-uploaded MSI file format, extracting the 3D data cube (spatial × spatial × spectral channels) and the corresponding m / z value vector. If the user does not provide an m / z value file, the system attempts to automatically extract the m / z axis information from the MSI file header information. Subsequently, preprocessing operations such as baseline correction, intensity normalization, and logarithmic transformation are performed on the raw MSI data (the processing flow is consistent with step S110 to ensure that the data distribution in the inference stage is consistent with that in the training stage), and the processed data tensors are converted into the model input format. If the spatial resolution of the MSI data exceeds the maximum processing size configured by the system, the system automatically performs sliding window segmentation, processing the large-size data in blocks before stitching the output results, ensuring that the system can process MSI data of any resolution.

[0185] For natural language query input, the system performs length checks and character normalization on the user-input query text. Then, the large language model's word segmenter converts the text into a token sequence, constructing a standardized input format containing system prompts. These system prompts declare the model's role (intelligent analysis assistant for mass spectrometry imaging) and output format specifications (answers must incorporate specific m / z values; SEG guidance tokens are output when segmentation is required), guiding the model to generate standardized and professional analytical responses.

[0186] (II) Visual Encoding Stage: If the user provides MSI data, the system enters the visual encoding stage. The preprocessed MSI data tensor and the m / z value vector are input into the visual encoder. The encoder performs forward propagation according to the physical perception spectral-spatial joint encoding process described in step S120. No tubular mask is applied during the inference stage, and full encoding is performed on all channels. The encoder outputs two features: the visual token sequence is projected onto the language embedding space through the visual adapter to obtain the visual-side embedding sequence; the multi-scale dense feature map is temporarily stored in the system cache and is called by the segmentation decoder when the SEG guiding token is triggered. If the user does not provide MSI data, the system skips the visual encoding stage and directly enters the subsequent process in plain text mode.

[0187] (III) Multimodal Fusion and Autoregressive Decoding: The visual-side embedded sequence (if present) and the text token embedded sequence queried by the user are concatenated to form the multimodal input sequence of the large language model, which is then fed into the large language model for autoregressive decoding. The large language model generates an output sequence token by token. For each generated token, the following judgment is made: if the current token is a regular text token, it is appended to the generated text sequence and decoding continues; if the current token is an SEG guiding token, text decoding is paused, the segmentation decoder is triggered to perform segmentation inference, and text decoding continues until the end-of-sequence marker is encountered after the mask is generated. The autoregressive decoding adopts a bundle search strategy (bundle width set to 4, for example) to improve the language quality of the response, while setting a maximum generation length limit to avoid infinite generation. A typical maximum length is set to 512 tokens.

[0188] (iv) Segment Decoder Inference: When the SEG guiding token is detected during the autoregressive decoding process, the system extracts the hidden state vector corresponding to the output of the last Transformer layer of the large language model. This vector is transformed into a segmentation cue embedding vector by the MLP projection submodule of the segment decoder, and then fused with the multi-scale dense feature map temporarily stored in the visual encoding stage in the semantic guiding feature fusion submodule. Finally, the mask generation submodule outputs a pixel-level confidence map that covers the spatial resolution of the original MSI data.

[0189] In the case where a single query contains multiple SEG guide tokens (i.e., the user requests to segment multiple regions simultaneously), the system independently executes the above segmentation and decoding process for each SEG guide token. Each decoding uses the hidden state vector of the corresponding SEG token as semantic guidance to generate its own independent pixel-level confidence map, ultimately obtaining a segmentation mask set that is the same as the number of SEG tokens.

[0190] S340: Output Post-processing and Result Presentation: This sub-step performs post-processing on the raw output generated during the inference stage and presents the structured results to the user in a standardized format, supporting multiple output formats to meet the needs of different use cases.

[0191] (a) Post-processing of segmentation mask: The original pixel-level confidence map output by the segmentation decoder is processed by the following post-processing steps to obtain the final segmentation result.

[0192] (1) Binarization: The confidence map is converted into a binary mask with a default threshold of 0.5. Pixels with a confidence score greater than or equal to 0.5 are marked as target regions (value 1), and the remaining pixels are marked as background (value 0). The threshold can be adjusted in the range of 0 to 1 according to user needs: increasing the threshold can reduce false detections and improve accuracy; decreasing the threshold can reduce false negatives and improve recall.

[0193] (2) Morphological refinement: Perform morphological opening operation (removing isolated noise pixels) and closing operation (filling small holes) on the binary mask in sequence to make the boundary of the segmented region smoother and more continuous, and remove isolated connected regions with an area lower than the preset minimum threshold.

[0194] (3) Connected region labeling: Perform connected region analysis on the refined binary mask, assign a unique label to each independent connected region, and count the geometric properties of each connected region, such as area (number of pixels), centroid coordinates, and bounding rectangle range, for users to further analyze.

[0195] (4) Coordinate mapping: The pixel coordinates are mapped back to the physical coordinate system of the MSI data (in micrometers). The actual physical size of the segmented region is calculated by combining the spatial resolution parameters of the original MSI data (in micrometers / pixel), so that the segmentation results have direct biological quantitative significance.

[0196] (II) Structured Organization of Results: The system organizes all outputs of this analysis into standardized structured result objects, including the following information fields: Text Response field, storing the complete natural language analysis conclusions generated by the large language model, where the SEG guiding token position is replaced with the reference identifier of the corresponding segmentation mask; Segmentation Result List field, storing the post-processing results of each segmentation mask in the order of SEG guiding token generation, with each record including mask number, corresponding text description fragment, post-processed binary mask data, connected component statistics and physical size information; Analysis Metadata field, recording the basic information of the MSI data used in this analysis (spatial resolution, m / z range, number of channels), system version number, inference timestamp and inference time, facilitating the traceability and reproduction of results.

[0197] (III) Multi-format output support: The system supports the following multiple output formats to adapt to the downstream usage needs of different users.

[0198] (1) Interactive visualization output: In the graphical interface, the segmentation mask is overlaid on the representative ion image or RGB composite image of the MSI data in the form of a semi-transparent color overlay. Different segmentation regions are distinguished by different colors. Users can adjust the overlay transparency, switch the displayed m / z channels, and click on a specific region to view the spectral curve and metabolite intensity distribution of that region, thus realizing interactive exploration of the results.

[0199] (2) Image file output: Export the visualization results of the segmentation mask overlay as PNG or TIFF format image files with the same resolution as the original MSI data spatial resolution; at the same time, export the original binary mask data as a single-channel grayscale image for users to further process in other image analysis software.

[0200] (3) Numerical data output: Export the original pixel-level confidence map as a NumPy array format (.npy file) or a general matrix format (.csv or .mat file) for users to binarize with any threshold or perform other quantitative analysis operations in a custom analysis workflow.

[0201] (4) Structured report output: The text response, segmentation region statistics (area, centroid coordinates, physical dimensions) and analysis metadata are output in JSON format, which makes it convenient for users to programmatically integrate the analysis results of this system with other data processing pipelines or laboratory information management systems.

[0202] (5) Batch report export: For scenarios where multiple samples are processed in batches, the system supports exporting the analysis results of all samples as a structured Excel spreadsheet or PDF report. Each record corresponds to one sample and includes the text summary conclusions and statistical data of key segmentation regions of that sample, which facilitates cross-sample horizontal comparisons by researchers in cohort studies.

[0203] (iv) Anomaly Handling and User Feedback: In the actual operation of the system, the following types of anomalies may be encountered. The system will take corresponding handling strategies for each type of anomaly and provide clear feedback information to the user.

[0204] (1) MSI data format error: If the uploaded MSI file format is not supported or the file is corrupted, the system will detect it in the preprocessing stage and return a clear error message to the user, explaining the list of supported file formats and possible solutions, without proceeding to the reasoning process.

[0205] (2) Unclear query semantics: If the user's query instruction is too vague (such as only entering "analyze this" without specifying the analysis target), the large language model will request the user to supplement information in the text response, list the available analysis directions, and guide the user to provide more specific query instructions, rather than outputting low-quality analysis results.

[0206] (3) Segmentation confidence is too low: If the confidence map output by the segmentation decoder fails to form an effective connected region with an area exceeding the minimum threshold after binarization, the system determines that the confidence of this segmentation result is insufficient and adds a confidence warning to the output, explaining the possible reasons (such as the target region described in the query is not significant in the current MSI data, or the query does not match the content of the MSI data), and suggests that the user adjust the query description or check the quality of the MSI data.

[0207] (4) Excess memory protection: If the MSI data size exceeds the current GPU memory processing limit, the system will automatically switch to sliding window block processing mode and prompt the user to change the processing mode, ensuring that the system can still run stably under resource-constrained conditions instead of crashing.

[0208] (V) Continuous Learning Interface (Optional Extension): As an optional extension function of the system, this invention supports collecting user feedback on the analysis results during the deployment phase, constructing a continuous learning loop to achieve iterative improvement of the system. Specifically, users can manually correct the segmentation mask output by the system through the interactive interface (such as adding missed detection regions, deleting false detection regions, and adjusting boundaries), and score the accuracy of the text responses. After anonymization, the user-corrected segmentation mask and scoring data can be included as new labeled samples in the training set for the next round of model fine-tuning, enabling the system to continuously optimize segmentation accuracy and text response quality over time. Enabling this continuous learning interface requires explicit user authorization and is protected by encryption during data transmission to ensure the security of the original MSI data and user privacy.

[0209] Typical S300 usage example:

[0210] To further illustrate the complete interaction process and output format of this system in actual use, two representative typical use cases are given below, covering two modes: plain text question answering and visual segmentation joint output.

[0211] (I) Typical Case 1: Visual Question Answering Mode

[0212] A user uploads an MSI data cube file of a human colorectal cancer tissue (spatial resolution 100×150 pixels, spectral channels 500, m / z range 400 to 900) and enters the following natural language query: "Please describe the overall spatial distribution characteristics of metabolites in this colorectal cancer tissue slice, focusing on the main metabolic differences between the tumor area and the normal mucosa area."

[0213] After receiving input, the system sequentially performs MSI data preprocessing, visual encoding (extracting visual token sequences and dense feature maps), adapter projection, and multimodal sequence concatenation. Subsequently, the large language model performs autoregressive decoding to generate a structured text response covering tissue region division, metabolite distribution characteristics, and metabolic differences between tumor and normal tissues. In this query, no segmentation instructions were included, and no SEG guiding token was generated during the large language model's decoding process. Therefore, the segmentation decoder was not triggered, and the system ultimately only outputs a text response. The average inference time is approximately 8 to 15 seconds (BFloat16 precision, single A100 GPU).

[0214] (II) Typical Case 2: Visual Question Answering and Segmentation Joint Output Mode

[0215] A user uploaded an MSI data file of rat brain tissue and entered the following query: "Please identify and mark the region in this rat brain slice where the signal of m / z=888.62 (phosphatidylethanolamine PE(38:4)) is significantly enriched, and explain the biological significance of this metabolite in the brain tissue."

[0216] After completing visual encoding, the large language model generates a SEG guiding token after describing the analysis conclusion during the text response generation process. Upon detecting the SEG guiding token, the system extracts the corresponding latent state vector, projects it using an MLP, and fuses it with a dense feature map. The segmentation decoder then generates a pixel-level segmentation mask covering the cerebral cortex and hippocampus. After post-processing, the system simultaneously returns a visual image of the text response and the segmentation mask overlaid on it to the user, along with area statistics of the segmented region (its proportion to the overall slice area) and physical size information. The average inference time for this query was approximately 12 to 20 seconds (BFloat16 precision, single A100 GPU), with the visual encoding stage taking approximately 3 to 5 seconds, the LLM decoding stage taking approximately 6 to 10 seconds, the segmentation decoder stage taking approximately 2 to 4 seconds, and the post-processing stage taking approximately 1 second.

Claims

1. A mass spectrometry imaging intelligent analysis system based on a multimodal large model, characterized in that, include: MSI visual encoder, visual adapter, large language model and segmentation decoder; The MSI visual encoder is used to receive the mass spectrometry imaging MSI data cube and its associated m / z value vector, and output a fixed-length visual token sequence and a multi-scale dense feature map that retains spatial structure information. The visual adapter is connected to the output of the MSI visual encoder and is used to project the visual token sequence into the embedding space of the large language model to generate a visual-side embedding sequence. The large language model is used to receive the visual-side embedding sequence and the text embedding sequence of the user's natural language query instruction, perform autoregressive decoding, generate a structured text answer containing natural language description, and generate an SEG guiding token during the autoregressive decoding process when the current answer needs to output the segmentation result. The segmentation decoder has its input connected to the output of the large language model and the output of the MSI visual encoder. In response to the SEG guiding token, it extracts the hidden state vector of the large language model corresponding to the token, maps the vector to a segmentation cue embedding vector, and fuses it with the multi-scale dense feature map to finally output a pixel-level segmentation mask with the same spatial resolution as the original MSI data. The system is capable of simultaneously outputting the structured text response and the pixel-level segmentation mask. The MSI visual encoder includes: a spatial downsampling convolution unit, a physical perception spectral position encoding unit, a spatial position encoding injection unit, an alternating spectral-spatial attention stacking unit, and a dual-branch output unit; The spatial downsampling convolutional unit is used to compress the spatial height and width of the input MSI data to 1 / s of the original value, and expand the number of channels to the encoder feature dimension D. The physical sensing spectral position encoding unit is used to normalize the m / z value of each visible channel; apply logarithmically uniform multi-frequency Fourier encoding to the normalized m / z value to generate a Fourier feature vector; map the Fourier feature vector to an m / z position encoding vector through linear projection; map the intensity value of the visible channel to an intensity encoding vector through linear projection; concatenate the m / z position encoding vector and intensity encoding vector and then perform fusion projection to generate a spectral token for each visible channel, forming a spectral token sequence; simultaneously, calculate the normalized m / z difference for any two visible channels, map it to a scalar bias value through a multilayer perceptron, and form a spectral token sequence. The m / z correlation bias matrix; the spatial location coding injection unit is used to apply a learnable location code to the downsampled spatial grid coordinates and superimpose the location code onto each spatial location of the spectral token sequence in a broadcast manner; the m / z correlation bias matrix and the spectral token sequence are processed by the alternating spectral-spatial attention stacking unit to obtain the final spatial feature map; the dual-branch output unit includes a cross-attention pooling unit and a multi-scale feature pyramid unit, the cross-attention pooling unit is used to extract a fixed-length visual token sequence from the final spatial feature map through the cross-attention pooling mechanism; the multi-scale feature pyramid unit is used to generate a multi-scale dense feature map from the spatial feature map output by the alternating spectral-spatial attention stacking unit through a feature pyramid network.

2. The intelligent mass spectrometry imaging analysis system based on a multimodal large model according to claim 1, characterized in that, The alternating spectral-spatial attention stacking unit comprises N cascaded alternating attention blocks, each block containing sequentially a spectral self-attention sub-block, a spatial self-attention block, and a feature broadcast fusion unit; The spectral self-attention sub-block is used to independently perform multi-head self-attention operation on the visible spectral channel sequence at each spatial location, and superimpose the m / z correlation bias matrix into the attention weight matrix, and perform mean pooling along the spectral dimension to aggregate the spectral token sequence at each spatial location into a single pixel feature vector to obtain the spatial feature sequence. Spatial self-attention sub-block is used to perform multi-head self-attention operation on spatial feature sequences across all spatial locations, capturing long-range spatial dependencies between different anatomical regions of tissue slices, and obtaining spatial feature sequences enhanced with spatial relationships. The feature broadcast fusion unit is used to broadcast the spatial feature sequence after spatial relation enhancement back to the spectral dimension and perform residual superposition with the spectral token sequence to obtain an updated feature tensor for use by the next alternation block. After processing through N stacked attention blocks, the final spatial feature map is output.

3. The intelligent analysis system for mass spectrometry imaging based on a multimodal large model according to claim 1, characterized in that, The segmentation decoder includes: a SEG token extraction and projection module, a semantically guided feature fusion module, and a mask generation module; The SEG token extraction and projection module is used to extract the hidden state vector at the corresponding position of the output of the last Transformer layer of the large language model after detecting the output of the SEG guiding token of the large language model; and to project the hidden state vector into a segmentation cue embedding vector with the same dimension as the feature dimension D of the MSI visual encoder through two layers of multilayer perceptron. The semantic guidance feature fusion module is used to align the segmentation cue embedding vector with the multi-scale dense feature map in the channel dimension; broadcast and expand the vector along the spatial dimension so that its shape is the same as the finest-grained dense feature map; add the broadcast vector to the finest-grained dense feature map element by element, and refine the features through a lightweight convolutional layer to obtain a spatial feature map that incorporates semantic guidance information; inject other scale dense feature maps from the MSI visual encoder step by step in the subsequent upsampling process through skip connections; The mask generation module is used to upsample the spatial feature map that incorporates semantic guidance information through multiple levels of deconvolution to gradually restore the spatial resolution of the original MSI data; after each level of upsampling, it is followed by BatchNorm normalization and ReLU activation function; finally, it outputs a pixel-level segmentation mask through a 1×1 convolutional layer and Sigmoid activation function, where the value of each pixel represents the confidence that the pixel belongs to the target region specified by the user query.

4. A mass spectrometry imaging intelligent analysis method based on a multimodal large model, applied to the system as described in any one of claims 1 to 2, characterized in that, Includes the following steps: S1. Perform preprocessing on the raw MSI data; generate a tubular mask, and perform physical sensing spectral-spatial joint coding on the masked MSI data; Self-supervised training is performed by reconstructing masked spectra, and the parameters of the MSI visual encoder are optimized by combining loss functions. S2. Construct training data that includes a pure dialogue subset, a visual question-answering subset, and a segmentation reasoning subset; Multimodal joint training is performed on the pre-trained MSI visual encoder, visual adapter, large language model, and segmentation decoder; S3. Deploy the trained system on a computing device to receive MSI data and natural language query commands uploaded by users; Preprocessing and visual encoding are performed on the MSI data to obtain visual token sequences and multi-scale dense feature maps; The visual token sequence is projected onto the language embedding space via a visual adapter to obtain a visual-side embedding sequence; the user's natural language query command is converted into a query text embedding sequence via a large language model; The visual-side embedding sequence and the query text embedding sequence are concatenated and then input into a large language model for autoregressive decoding; when the SEG guiding token is detected, the segmentation decoder is triggered to generate a pixel-level segmentation mask; Synchronously output structured text answers and segmentation masks.

5. The intelligent analysis method for mass spectrometry imaging based on a multimodal large model according to claim 4, characterized in that, In S1, the physical sensing spectral-spatial joint encoding includes: normalizing the m / z value of each visible channel; applying logarithmically uniformly distributed multi-frequency Fourier encoding to the normalized m / z value to generate a Fourier feature vector; mapping the Fourier feature vector to an m / z position encoding vector via linear projection; mapping the intensity value of the visible channel to an intensity encoding vector via linear projection; concatenating the m / z position encoding vector and the intensity encoding vector and then performing fusion projection to generate a spectral token for each visible channel, forming a spectral token sequence; simultaneously, calculating the normalized m / z difference for any two visible channels, mapping it to a scalar bias value via a multilayer perceptron, and forming an m / z correlation bias matrix. The combined loss is composed of a weighted sum of the mean square error loss and the spectral angle mapping loss.

6. The intelligent analysis method for mass spectrometry imaging based on a multimodal large model according to claim 4, characterized in that, In S2, the pure dialogue subset consists of natural language questions and corresponding answers, and does not contain MSI image data or segmentation mask annotations; The visual question-answering subset consists of an MSI data cube, a matching m / z value vector, natural language questions and text answers, but does not include a segmentation mask. The segmentation inference subset includes an MSI data cube, a matching m / z value vector, a natural language question, a text response containing a SEG guiding token, and a corresponding pixel-level segmentation mask.

7. The intelligent analysis method for mass spectrometry imaging based on a multimodal large model according to claim 4, characterized in that, In S2, the multimodal joint training includes: Phase 1: Using pure dialogue subsets and visual question-answering subsets, freeze all parameters of the pre-trained MSI visual encoder and large language model, train only the visual adapter, and use the image-text pairs in the visual question-answering subset as supervision signals to achieve preliminary semantic alignment between the visual feature space and the language embedding space. The training objective is the standard autoregressive language modeling loss. The second stage involves introducing a segmentation decoder, removing the freezing restrictions of the MSI visual encoder and the large language model, updating all parameters of the MSI visual encoder with a low learning rate, efficiently fine-tuning the parameters of the large language model using the low-rank adaptive LoRA method, updating the visual adapter and segmentation decoder with all parameters, and performing end-to-end joint instruction fine-tuning training on the entire system using mixed data of three subsets. The joint loss function is composed of a weighted sum of the language modeling loss and the segmentation loss. The segmentation loss in the second stage is composed of the equal weighted sum of the binary cross-entropy loss and the Dice loss. For training data samples without segmentation mask labels, the segmentation loss is set to zero.

8. The intelligent analysis method for mass spectrometry imaging based on a multimodal large model according to claim 7, characterized in that, In S3, after training, the system saves the full parameter weights of the visual encoder, the full parameter weights of the visual adapter, the original weights of the large language model and the incremental weights of LoRA, and the full parameter weights of the segment decoder. Before model deployment, the low-rank adaptive incremental weights of the large language model are merged into its original weights, and the visual encoder, visual adapter, merged large language model weights, and segment decoder weights are organized into a single model weight file.

9. The intelligent analysis method for mass spectrometry imaging based on a multimodal large model according to claim 4, characterized in that, In S3, when the user does not provide MSI data, the system skips the visual encoding and segmentation decoding process and directly performs autoregressive decoding on the natural language query command in plain text mode, outputting only a structured text answer.

Images