Multimodal mass spectrometry imaging representation learning method for pathological segmentation

Abstract

This invention discloses a multimodal mass spectrometry imaging representation learning method for pathological segmentation, comprising: acquiring and preprocessing multiple mass spectrometry imaging data, establishing spatial transformation relationships for each modality; the preprocessed multimodal data is then subjected to a double-layer mask, followed by a physical perception modal encoder to extract pixel-level spectral-spatial fusion features that fuse mass-to-charge ratio physical information, and then fused using a shared cross-modal Transformer encoder, performing self-supervised pre-training with the goal of reconstructing the spectral intensity of the masked region to obtain a pre-trained encoding system; a cross-modal attention fusion module and a segmentation decoder are introduced to perform supervised fine-tuning on data with segmentation labels to construct a segmentation system; the processed new samples are processed by the segmentation system to obtain a segmentation mask, and mapped back to the pathological image coordinate system according to the spatial transformation relationships to generate segmentation results. This invention efficiently fuses chemical prior information, achieving pixel-level alignment and efficient segmentation of multimodal mass spectrometry imaging data.

Description

Technical Field

[0001] This invention belongs to the field of medical image analysis, specifically relating to a multimodal mass spectrometry imaging representation learning method for pathological segmentation. Background Technology

[0002] Mass spectrometry imaging (MSI) technology can acquire spatial distribution information of hundreds to thousands of metabolites, lipids, proteins, or elements in situ on tissue sections, achieving panoramic visualization of histochemical components without relying on exogenous markers. It is an important tool for disease pathology research, drug distribution analysis, and clinical tissue typing. Depending on the ionization method and applicable molecule types, the mainstream mass spectrometry imaging techniques currently include matrix-assisted laser desorption / ionization mass spectrometry (MALDI-MSI), desorption electrospray ionization mass spectrometry (DESI-MSI), and secondary ion mass spectrometry (SIMS). These three techniques each have their own strengths in spatial resolution, detection quality range, and applicable molecule categories, and complement each other. However, intelligent analysis of mass spectrometry imaging data faces the following technical bottlenecks:

[0003] First, there is the problem of effectively representing high-dimensional spectral data. Mass spectrometry imaging data is essentially a three-dimensional data cube. In the spatial dimension, each pixel corresponds to a sampling position on a tissue slice, and in the spectral dimension, each channel corresponds to an ion intensity value within a mass-to-charge ratio (m / z) range. The number of channels is typically hundreds to thousands. Most existing deep learning methods are designed based on RGB three-channel images, which cannot directly process such high-dimensional spectral data. Dimensionality reduction preprocessing (such as principal component analysis (PCA) and t-distributed random neighborhood embedding (t-SNE)) inevitably loses a large amount of chemical information, which limits the model's ability to learn the inherent chemical laws of mass spectrometry imaging data.

[0004] Second, there is the issue of utilizing the physicochemical information of the mass-to-charge ratio. The mass-to-charge ratio values ​​of each channel in mass spectrometry imaging data are not arbitrary labels, but rather continuous values ​​carrying clear physicochemical meanings. Similar mass-to-charge ratio values ​​often correspond to molecules with related chemical structures, exhibiting an inherent physical correlation. Existing methods based on image convolution or general Transformers treat each spectral channel as an equivalent feature dimension, completely ignoring the prior chemical information contained in the mass-to-charge ratio values. This prevents the model from utilizing physical constraints such as molecular mass conservation and isotope distribution to improve the chemical rationality of feature learning.

[0005] Third, there is the issue of fusing multimodal mass spectrometry data. Different mass spectrometry imaging modalities detect different molecular categories, have varying spatial resolutions and signal-to-noise ratios, and their chemical descriptions of the same tissue region are complementary rather than redundant. However, the number of channels and mass-to-charge ratio coverage differ among the three modalities, making direct alignment of data shapes impossible. Furthermore, existing methods lack cross-modal attention fusion mechanisms, making it difficult to establish cross-modal semantic relationships between the three modalities while preserving the unique chemical information of each modality. This makes it challenging to describe the chemical state of the same tissue location from multiple complementary perspectives.

[0006] Fourth, there is the problem of scarce labeled data. Pathological tissue segmentation tasks require pathologists to perform pixel-level annotation on tissue slides. The annotation process is highly dependent on professional knowledge and is time-consuming and labor-intensive, severely limiting the scale of high-quality labeled data. Existing supervised learning methods are heavily reliant on labeled data and struggle to achieve reliable pixel-level segmentation performance with a limited number of labeled samples. How to fully utilize a large amount of unlabeled mass spectrometry imaging data and learn a general spectral-spatial representation through self-supervised pre-training to reduce the demand for labeled data in downstream tasks is a key problem that urgently needs to be solved in the field of intelligent analysis of mass spectrometry imaging.

[0007] To address the first and second issues mentioned above, a dedicated coding architecture capable of perceiving the physicochemical meaning of mass-to-charge ratio and processing high-dimensional spectral channels is needed to replace the general visual encoder. For the third issue, a unified coding framework supporting joint input of trimodal heterogeneous data needs to be established, and complementary chemical associations between modalities should be learned through cross-modal attention mechanisms. For the fourth issue, a self-supervised pre-training paradigm needs to be introduced, enabling the model to learn general representations from large-scale mass spectrometry imaging data without relying on annotations, thereby significantly reducing the annotation data requirements for downstream tasks.

[0008] In existing research, masked autoencoders (MAEs) have demonstrated powerful self-supervised representation learning capabilities on natural images, point clouds, and multimodal data. By randomly masking parts of the input data and training the model to reconstruct the masked portions, the model is forced to learn the inherent distribution patterns and semantic representations of the data. However, directly applying the masked autoencoder framework to multimodal mass spectrometry imaging data faces the following inherent limitations: First, the masking strategy of standard masked autoencoders is based on pixel patches, which is not suitable for mass spectrometry imaging data acquired pixel by pixel (each pixel is an independent sampling point); second, the encoder of a standard masked autoencoder is not aware of the physical coordinates of the mass-to-charge ratio of spectral channels and cannot utilize prior chemical information; third, existing multimodal masked autoencoder frameworks design completely independent encoders for each modality, lacking cross-modal parameter sharing and interaction mechanisms, and cannot effectively establish cross-modal semantic correspondences between different mass spectrometry imaging modes.

[0009] In view of this, the present invention proposes a multi-modal mass spectrometry imaging representation learning method for pathological segmentation. It is specifically designed to address the physical characteristics of mass spectrometry imaging data and the requirements for multi-modal fusion, thereby solving the above-mentioned technical problems and achieving high-quality pixel-level representation learning and pathological tissue segmentation of multi-modal mass spectrometry imaging data under the condition of scarce labeled data. Summary of the Invention

[0010] Addressing four core shortcomings of existing technologies in the field of intelligent analysis of multimodal mass spectrometry imaging data—lack of effective representation methods for high-dimensional spectral data, unutilized mass-to-charge ratio physicochemical information, absence of multimodal data fusion mechanisms, and scarcity of labeled data limiting the performance of downstream tasks—this invention aims to provide a multimodal mass spectrometry imaging representation learning method for pathological segmentation, specifically achieving the following technical objectives:

[0011] Firstly, a physical-aware modal encoder is provided, which can explicitly perceive the physicochemical meaning of the mass-to-charge ratio values ​​of each channel during the encoding process. It integrates molecular mass information into feature learning by using the mass-to-charge ratio Fourier position encoding and the physical correlation bias matrix, thus overcoming the problem that existing general-purpose visual encoders do not make sufficient use of high-dimensional spectral data and its physical prior information.

[0012] Secondly, a unified pre-training framework for multi-mode mass spectrometry imaging data is provided. By sharing a cross-modal Transformer encoder, the data of three modes—matrix-assisted laser desorption / ionization mass spectrometry imaging, desorption electrospray ionization mass spectrometry imaging, and secondary ion mass spectrometry imaging—are jointly encoded. Under the condition of no manual annotation, a general spectral-spatial unified representation is learned from large-scale mass spectrometry imaging data.

[0013] Third, a dual-layer mask self-supervised pre-training strategy is provided, which applies random masks simultaneously in the spatial pixel dimension and the spectral channel dimension, forcing the model to learn spatial context relationships and inter-channel chemical correlations at the same time, thereby improving the completeness and generalization ability of the pre-trained representation.

[0014] Fourth, a downstream fine-tuning method for pathological tissue segmentation is provided. Using a pre-trained encoding system as the backbone, and through joint fine-tuning of a cross-modal attention fusion module and a lightweight segmentation decoder, high-quality pixel-level pathological tissue segmentation can be achieved with only a small amount of labeled data, and cross-lesion type transfer applications can be supported.

[0015] To achieve the above-mentioned objectives, this invention provides a multi-modal mass spectrometry imaging representation learning method for pathological segmentation, comprising the following steps:

[0016] Step 1: Acquire multiple mass spectrometry imaging data of the same tissue sample, and preprocess the multiple mass spectrometry imaging data with the pathological image of the tissue sample as the reference coordinate system, and establish the spatial transformation relationship between each mode and the pathological image coordinate system.

[0017] Step 2: Using the preprocessed multimodal data as input, the masked multimodal data is obtained through two-layer masking. The pixel-level spectral-spatial fusion features of the fused mass-to-charge ratio physical information are extracted by the physical perception modality encoder, and then the multimodal information is fused by the shared cross-modal Transformer encoder. Self-supervised pre-training is performed with the goal of reconstructing the spectral intensity of the masked region to obtain the pre-trained coding system.

[0018] Step 3: Obtain multimodal alignment data with pixel-level segmentation labels, introduce a cross-modal attention fusion module to fuse the multimodal features extracted by the fusion pre-trained encoding system to obtain enhanced fused features, and obtain pixel-level class probability maps through the segmentation decoder; and perform supervised fine-tuning on the multimodal alignment data with pixel-level segmentation labels to construct the segmentation system;

[0019] Step 4: After processing the newly acquired multimodal mass spectrometry imaging data in Step 1, input it into the segmentation system to obtain a pixel-level segmentation mask; and according to the spatial transformation relationship, map the pixel-level segmentation mask back to the pathological image coordinate system to generate a segmentation result that is spatially fused with the pathological morphological information.

[0020] Optionally, the acquisition of multiple mass spectrometry imaging data of the same tissue sample in step 1 includes: performing continuous slicing of the same tissue sample paraffin block, and acquiring mass spectrometry imaging data of the first mode, the second mode, and the third mode, as well as a pathological image as a registration reference.

[0021] Preferably, the preprocessing of multiple mass spectrometry imaging data includes: using the spatial resolution of the pathological image as a registration reference as a standard, downsampling the high-resolution modal data and upsampling the low-resolution modal data to achieve uniformity of the spatial resolution of each modal data.

[0022] Specifically, nearest neighbor interpolation is used for upsampling of low-resolution modal data, while spatial average pooling is used for downsampling of high-resolution modal data.

[0023] Preferably, establishing the spatial transformation relationship from each modality to the pathological image coordinate system includes:

[0024] The total ion flow map of each modality mass spectrometry imaging data is extracted as a floating image. The grayscale image of the pathological image with the registration benchmark is used as the reference image. Rigid registration and non-rigid registration are performed in sequence to establish the spatial transformation matrix from each modality to the coordinate system of the pathological image.

[0025] The region of interest is delineated in the coordinate system of the pathological image and mapped to the mass spectrometry imaging data of each modality. After cropping, the multimodal alignment intensity data corresponding to the spatial pixel position is obtained, and the spatial transformation matrix is ​​stored simultaneously to complete the establishment of the spatial transformation relationship.

[0026] Preferably, the double-layer mask in step 2 includes:

[0027] Spatial mask is used to randomly mask a preset proportion of pixel positions in the spatial pixel dimension. All channel data of the masked pixels are not included in the encoder calculation.

[0028] Tubular masks are used to perform consistent channel random zeroing at all pixel locations in the spectral channel dimension. The intensity values ​​of the zeroed channels are set to zero, but their mass-to-charge ratio position encoding is preserved.

[0029] The spatial mask and the tubular mask are independent of each other.

[0030] Preferably, the physical sensing modal encoder in step 2 includes:

[0031] The intensity linear projection module is used to map the ion intensity vector of the masked multimodal data pixels into a dense vector to obtain the intensity embedding of each channel.

[0032] The mass-to-charge ratio Fourier position coding module is used to generate continuous position coding vectors based on the mass-to-charge ratio values ​​of each channel, and add them to the intensity embedding of the corresponding channel to obtain the channel feature vector that fuses the mass-to-charge ratio physical information.

[0033] The spectral self-attention module is used to take the sequence of fused feature vectors of each channel as input, perform self-attention operation between different channels of the same pixel, and superimpose a learnable physical correlation bias matrix initialized based on mass-to-charge ratio distance, and output the feature sequence after the interaction between channels.

[0034] Attention-weighted pooling layer is used to generate weights for each channel based on the feature sequence after the interaction between channels, and to perform weighted summation on the feature sequence to obtain pixel-level spectral aggregated features that aggregate all channel information;

[0035] The spatial self-attention module takes pixel-level spectral aggregation features that aggregate information from all channels as input, performs self-attention operations between pixels, and adds two-dimensional spatial position encoding to output pixel-level features that fuse spatial context information.

[0036] Optionally, the physical sensing modal encoder is set independently for each modality, and the parameters of the mass-to-charge ratio Fourier position encoding module are shared among multiple modalities.

[0037] Preferably, step 2 further includes: mapping the pixel-level features with fused spatial context information to a common feature space of uniform dimension through a linear projection adaptation layer.

[0038] Preferably, the shared cross-modal Transformer encoder used in step 2 for multimodal information fusion includes:

[0039] Using features from a common feature space as input, a multimodal feature sequence is obtained by concatenation.

[0040] Based on multimodal feature sequences, cross-modal semantic associations are established through a self-attention mechanism. The parameters are fully shared by multiple modalities, thereby achieving the fusion of multimodal feature information related to chemical correlations and obtaining multimodal information fusion features.

[0041] Preferably, in step 2, a reconstruction loss is established for self-supervised training. The reconstruction loss is composed of a weighted sum of channel adaptive weighted mean square error loss and spectral angle mapping loss. The channel adaptive weights are pre-calculated and fixed based on the signal-to-noise ratio of each channel.

[0042] Furthermore, the reconstruction loss is ,in and The first The reconstructed intensity value and the original intensity value of each channel, For the first Adaptive weights for each channel, Mean square error, These are the weighting coefficients for the spectral angle mapping loss. For spectral angle mapping, To reconstruct the spectral angle, This is the original spectral angle.

[0043] Preferably, step 3 specifically includes:

[0044] Pixel-level segmentation labels on pathological images are obtained, and the pixel-level segmentation labels are mapped to the mass spectrometry imaging pixel coordinate system using a spatial transformation matrix to generate a dense label mask.

[0045] Multimodal data is input into a pre-trained coding system, and pixel-level fused feature maps of each multimodal are extracted as multimodal features.

[0046] Multimodal features are input into the cross-modal attention fusion module. Using a spatial window as the unit, cross-attention is calculated with any modal feature as the query and other modal features as the key and value. The enhanced fused features are then fused with the multimodal mean through residual connections.

[0047] The enhanced fused features are input into the segmentation decoder, and through upsampling and convolution operations, a pixel-level class probability map is output.

[0048] The segmentation loss is calculated using a dense label mask, and the parameters of the cross-modal attention fusion module and the segmentation decoder are updated.

[0049] Furthermore, the segmentation loss is a weighted sum of the cross-entropy loss and the Dice loss, expressed as follows: , The weights for the Dice loss;

[0050] Among them, cross-entropy loss For all Perform pixel-by-pixel classification supervision at each pixel location, represented as , For pixels Category The true label, The corresponding predicted probability, Total number of categories;

[0051] Dice loss This measure assesses the overlap between predicted and ground truth regions for each category, aiming to mitigate the class imbalance problem commonly encountered in segmentation tasks. It is represented as... , To prevent smooth terms with a denominator of zero.

[0052] Preferably, the cross-modal attention fusion module calculates cross-attention within a local window, using the window features of the first modality as the query matrix and the corresponding window features of the second and third modalities as the key and value matrices, to obtain the enhanced features of the first modality; similarly, the enhanced features of the second and third modalities are calculated respectively; finally, the element-wise mean of the enhanced features of multiple modalities is taken to obtain the enhanced fusion features.

[0053] Preferably, step 4 further includes:

[0054] For new samples with the same lesion type as those in the supervised fine-tuning stage, the segmentation system is directly invoked for inference;

[0055] For new samples with new lesion types, the parameters of the pre-trained encoding system and the cross-modal attention fusion module are reused, and the segmentation decoder is fine-tuned using only the labeled data of the new lesion type before inference is performed on the new samples.

[0056] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0057] First, it enables the explicit utilization of physicochemical information related to the mass-to-charge ratio. The proposed physical-sensing modal encoder maps the mass-to-charge ratio values ​​of each channel to continuously differentiable physical embedding vectors through three-modal shared Fourier position encoding of the mass-to-charge ratio. Furthermore, it introduces a physical correlation bias matrix initialized based on the mass-to-charge ratio distance into the spectral self-attention module, allowing the model to explicitly perceive the physical prior of molecular mass when learning inter-channel correlations. This capability is not present in existing general-purpose visual encoders and helps the model learn chemically significant channel correlation patterns such as isotope peaks and adduct ions.

[0058] Second, this invention achieves unified joint encoding of three-mode mass spectrometry imaging data. By sharing mass-to-charge ratio encoding parameters and a shared cross-modal Transformer encoder, this invention enables joint representation learning of heterogeneous data from matrix-assisted laser desorption / ionization mass spectrometry (MALS), desorption / electrospray ionization mass spectrometry (DEES), and secondary ion mass spectrometry (SES) in a unified feature space, eliminating the need for separate encoders for each mode. The shared encoder establishes chemical correlations between the three modes at the mass-to-charge ratio coordinate level through a cross-modal self-attention mechanism, effectively integrating complementary chemical information detected by different modes.

[0059] Third, the completeness of the representation is improved through a dual-layer masking pre-training strategy. The dual-layer masking strategy proposed in this invention applies random masking simultaneously in the spatial pixel dimension and the spectral channel dimension. Compared with the standard masked autoencoder that only applies spatial masking, it applies both spatial context reconstruction pressure and spectral chemical law reconstruction pressure to the model, which helps to learn a more complete spectral-spatial unified representation and improves the feature quality of downstream tasks.

[0060] Fourth, it significantly reduces the dependence of downstream tasks on labeled data. The general representation learned by the pre-trained encoding system on large-scale unlabeled mass spectrometry imaging data can be directly transferred to downstream segmentation tasks as a powerful feature backbone. Experiments show that the method of this invention can achieve reliable pixel-level pathological tissue segmentation performance with only 5 to 20 labeled slice groups, which is significantly lower than the scale of labeled data required for de novo training, effectively alleviating the core bottleneck of scarce labeled data in mass spectrometry imaging.

[0061] Fifth, it supports transfer learning across lesion types. The spectral-spatial general representation learned by the pre-trained encoding system is not specific to any particular lesion type and can be reused across lesion types. When faced with new lesion types, only a small amount of labeled data needs to be fine-tuned to the lightweight segmentation decoder, without retraining the encoding system. This significantly reduces the adaptation cost for new clinical tasks and improves the practicality and scalability of this invention in different clinical scenarios. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0063] Figure 1 This is a flowchart illustrating the multimodal mass spectrometry imaging representation learning method for pathological segmentation provided by the present invention.

[0064] Figure 2 This is a flowchart of the data acquisition and preprocessing process for various mass spectrometry imaging methods.

[0065] Figure 3 This is a schematic diagram of the internal structure of the physical sensing modal encoder provided by the present invention.

[0066] Figure 4 This is an overall architecture diagram of the multimodal mask autoencoder pre-training framework provided by the present invention.

[0067] Figure 5 This is a framework diagram for fine-tuning downstream tasks for pathological tissue segmentation provided by the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and given in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0069] The technical concept of this invention is as follows: Addressing four core shortcomings of existing technologies in the field of intelligent analysis of multi-mode mass spectrometry imaging data—lack of effective representation methods for high-dimensional spectral data, unutilized mass-to-charge ratio physicochemical information, absence of multi-mode data fusion mechanisms, and scarcity of labeled data limiting the performance of downstream tasks—this invention provides a multi-mode mass spectrometry imaging representation learning method for pathological segmentation, such as… Figure 1 Includes the following steps:

[0070] S1. Acquire multiple mass spectrometry imaging data of the same tissue sample, and preprocess the multiple mass spectrometry imaging data using the pathological image of the tissue sample as the reference coordinate system, and establish the spatial transformation relationship between each mode and the pathological image coordinate system.

[0071] This step acquires mass spectrometry imaging data of three modalities for the same tissue sample, and establishes pixel-level correspondence of the three modalities in a unified spatial coordinate system through image registration and resolution unification processing, outputting a multimodal aligned data set for subsequent pre-training and downstream tasks.

[0072] Step 1 of this embodiment requires preparing multiple slices from the same tissue sample paraffin block using a continuous slicing method. This ensures that the tissue regions acquired by each modality are spatially highly consistent, providing a reliable structural correspondence basis for subsequent registration. The pre-training stage requires at least 30 trimodal paired slice groups, preferably at least 100 slice groups, to cover tissue samples from multiple patients. The downstream fine-tuning stage requires at least 5 fully labeled slice groups, preferably 10 to 20 slice groups. For the complete data acquisition and preprocessing workflow, please refer to [link to documentation]. Figure 2 The details are as follows:

[0073] S1.1 Sample Preparation and Multimodal Data Acquisition

[0074] Fresh frozen tissue samples were embedded in an optimal cutting temperature compound (OCT compound). At least four frozen sections, 5-10 micrometers thick, were then continuously cut from the same tissue block using a cryostat. The sections were allocated as follows: section 1 for matrix-assisted laser desorption / ionization mass spectrometry (MALS) imaging; section 2 for desorption / electrospray ionization mass spectrometry (DEES) imaging; section 3 for secondary ion mass spectrometry (DIMS) imaging; and section 4, after hematoxylin-eosin (H&E) staining, was used to obtain a registration reference image. The preferred digital scanning resolution for section 4 was 0.25 micrometers per pixel. After scanning, the section was converted to a grayscale image, serving as a common coordinate system reference for subsequent registration.

[0075] Mass spectrometry imaging was performed on slices 1 to 3 using the corresponding instruments to obtain three sets of raw three-dimensional intensity data. The spatial resolution, mass-to-charge ratio acquisition range, and number of channels for each modality are as follows (the following values ​​are exemplary and do not limit the scope of protection of this invention): Matrix-assisted laser desorption / ionization mass spectrometry imaging has a spatial resolution of 50 micrometers per pixel, a mass-to-charge ratio range of 400-900 Daltons (Da), and a number of channels... The spatial resolution of desorption / electrospray ionization mass spectrometry imaging is 200 micrometers per pixel, with a mass-to-charge ratio ranging from 100 to 1000 Da, and a number of channels. =300; Secondary ion mass spectrometry imaging spatial resolution is 5 micrometers per pixel, number of channels ; , , These are the index variables corresponding to matrix-assisted laser desorption / ionization mass spectrometry (MALS), desorption electrospray ionization mass spectrometry (DES), and secondary ion mass spectrometry (SMS).

[0076] The shapes of the raw output data for the three modes are as follows: , , ,in , , and , , Corresponding to their respective spatial pixel sizes at their respective resolutions, all three cover the same physical spatial range. Each pixel location corresponds to an ion intensity vector on each m / z channel, representing the local chemical composition information at that spatial location, serving as the basic processing unit for subsequent encoders.

[0077] S1.2 Two-step image registration based on hematoxylin-eosin staining images

[0078] Because the three mass spectrometry imaging instruments operate independently, and the slides inevitably undergo translation, rotation, and local nonlinear deformation during preparation and transfer, spatial misalignment exists between the three-modal data and the hematoxylin-eosin staining reference image. This step uses the hematoxylin-eosin staining grayscale image as a common coordinate system reference, independently estimating the spatial transformation of each of the three modalities. Indirect spatial alignment of the three modalities is achieved through bridging the hematoxylin-eosin staining coordinate system, eliminating the need for direct pairwise registration between the three modalities. Specifically:

[0079] S1.2.1 Total Ion Flow Image Extraction. Before registration, the original three-dimensional intensity data of each modality are summed along the channel dimension to obtain the total ion flow (TIC) image of that modality, which is then used as the floating image input to the registration process. The total ion flow image aggregates the spatial structure information of all channels, has a high signal-to-noise ratio, and its tissue boundary grayscale contrast has good spatial correlation with the hematoxylin-eosin staining image, making it suitable for mutual information-based registration optimization.

[0080] S1.2.2 Rigid Registration. Using the total ion current image as the floating image and the hematoxylin-eosin staining grayscale image as the reference image, the rigid transformation, comprising six degrees of freedom (translation, rotation, and isotropic scaling), is optimized. The optimization objective is normalized mutual information (NMI), defined as:

[0081] ,

[0082] in, , For their respective edge entropy, The joint entropy is used. The advantage of using mutual information as a metric lies in its robustness to differences in grayscale distribution between images of different modalities, without assuming a linear grayscale correspondence between the two modalities. Optimization employs a multi-resolution pyramid strategy, progressively refining the transformation parameters from low resolution to high resolution.

[0083] S1.2.3 Non-rigid registration. Using the rigid registration result as the initial state, a B-spline deformation field model is further used to estimate local nonlinear deformation to compensate for tissue shrinkage and local deformation during slide preparation. The optimization objective remains normalized mutual information, with an additional deformation field smoothing regularization term to constrain the continuity of deformation and prevent excessive deformation.

[0084] By sequentially combining rigid transformations with non-rigid deformation fields, we obtain the complete spatial transformation of each mode to the hematoxylin-eosin staining coordinate system. , , The data were then applied to the original three-dimensional intensity data of the corresponding modes, and after bilinear interpolation resampling, spatially aligned three-modal intensity data in the hematoxylin-eosin staining coordinate system were obtained.

[0085] Registration quality is evaluated using normalized mutual information values, requiring that after registration... For slices that do not meet the standards, the registration process is repeated. Slices that still fail to meet the standards after repeated registration are removed from the dataset.

[0086] S1.3 Spatial resolution uniformity

[0087] The spatial resolutions of the three modalities differ significantly, and directly performing joint modeling on multi-resolution data would introduce systematic biases caused by these resolution differences. Therefore, using the spatial resolution of matrix-assisted laser desorption / ionization mass spectrometry (50 micrometers per pixel) as a benchmark, we upsampled the registered desorption / electrospray ionization mass spectrometry data and downsampled the secondary ion mass spectrometry data to ensure that the three modalities are completely consistent in spatial pixel shape.

[0088] S1.3.1 Upsampling of Desorption Electrospray Ionization Mass Spectrometry Imaging. The ratio of the original resolution (200 μm per pixel) to the reference resolution (50 μm per pixel) of desorption electrospray ionization mass spectrometry imaging is 4. The registered desorption electrospray ionization mass spectrometry imaging data is upsampled by 4 times nearest neighbor interpolation in the spatial dimension, changing the data shape from... Expand to Nearest neighbor interpolation is used instead of bilinear interpolation to maintain the discrete independence of intensity values ​​at each measurement location and avoid introducing artificial intensity mixing between adjacent measurement locations.

[0089] S1.3.2 Secondary Ion Mass Spectrometry Imaging Downsampling. The ratio of the original SIMS resolution (5 μm per pixel) to the reference resolution (50 μm per pixel) is 1 / 10. Windowed spatial average pooling downsampling is performed on the registered secondary ion mass spectrometry imaging data in the spatial dimension to change the data shape from... Compress to Average pooling, rather than subsampling, is used to preserve the statistical average information within the original 10×10 pixel region, while suppressing pixel-level random noise during the secondary ion mass spectrometry imaging acquisition process.

[0090] The resolution unification operation only aligns the spatial pixel coordinates and does not change the true spatial resolution of each modality. After processing, the true resolution marker is appended to each of the three modalities. =50 micrometers =200 micrometers =5 micrometers), for use by the subsequent physical sensing coding module.

[0091] S1.4 Region of Interest Selection and Data Output

[0092] Regions of interest (ROIs) were uniformly defined in the hematoxylin-eosin staining coordinate system. The boundaries of the target regions were determined manually or automatically based on tissue detection algorithms. The ROI coordinates were mapped to the pixel coordinates of each modality through a resolution scaling relationship. The three modalities were cropped within the same physical space. The resulting shapes of the three modal data are as follows: , , ,in The total number of pixels is denoted as , and each pixel position is treated as an independent processing unit for the subsequent encoder.

[0093] Perform quality control on the selected data: the normalized mutual information after registration is not less than 0.8; the proportion of effective pixels in the region of interest is not less than 80%; there are no all-zero channels or abnormal channels exceeding the range in each modality, and those that are found are truncated or zeroed out; the size of the dataset meets the pre-training and fine-tuning requirements mentioned at the beginning of this step.

[0094] After the above processing, this step ultimately outputs three types of information: First, trimodal alignment strength data, where the three modalities are completely pixel-level aligned in the H&E coordinate system, requiring no annotation and can be directly used for pre-training in step S2; Second, true resolution labels for each modality ( , , ), for the physical perception encoding module of step S2 to read; third, the spatial transformation matrix of each mode to the hematoxylin-eosin staining coordinate system ( , , These are used in step S4 to map the segmentation results back to the H&E coordinate system. The pixel-level segmentation labels required in step S3 will be obtained separately in step S3.

[0095] S2. Using the preprocessed multimodal data as input, the masked multimodal data is obtained through double-layer masking. The pixel-level spectral-spatial fusion features of the fused mass-to-charge ratio physical information are extracted by the physical perception modality encoder, and then the multimodal information is fused by the shared cross-modal Transformer encoder. Self-supervised pre-training is performed with the goal of reconstructing the spectral intensity of the masked region to obtain the pre-trained coding system.

[0096] This step takes the trimodal aligned data output from step S1 as input and performs joint self-supervised pre-training on the data from the three mass spectrometry imaging modes using a multimodal mask autoencoder framework. This allows the model to learn a universal spectral-spatial unified representation from large-scale mass spectrometry imaging data without relying on any manual annotation. After pre-training, a pre-trained encoding system is output for use in step S3.

[0097] S2.1 Input Data and Two-Layer Masking Strategy

[0098] The input for this step is the trimodal alignment strength data output from step S1, with the following shapes: , , The three modes are fully pixel-level aligned in a unified spatial coordinate system, requiring no annotation. The pre-training data consists of no fewer than 30 pairs of three-modal slices, preferably no fewer than 100 slices. Each pixel position is treated as an independent processing unit (token), with its corresponding value being the ion intensity vector at that position in each mass-to-charge ratio channel.

[0099] To address the two-dimensional structural characteristics of mass spectrometry imaging data, which possess both spatial (pixel location) and spectral (mass-to-charge ratio channels) dimensions, this invention designs a dual-layer masking strategy. This strategy applies independent random masks to the input data in both dimensions, forcing the model to simultaneously learn the spatial contextual relationships between pixels and the chemical correlations between channels. Specifically, this includes:

[0100] 1. Spatial Mask. For each training sample, a mask index is randomly sampled along the spatial pixel dimension, with the mask ratio set to 75% (example value). All channel intensity values ​​at the selected pixel location are set to zero, and only the unmasked visible pixels (approximately 25% of the total pixels) are masked. (Number) input encoders are used to reduce computational load.

[0101] 2. Tubular Mask. Randomly zero out all pixel locations along the spectral channel dimension, with a mask ratio set to 75% (example value). The set of channel mask indices is identical for all pixel locations, meaning the zeroed mass-to-charge ratio channels remain spatially consistent. The intensity values ​​of the zeroed channels are zero, but the corresponding mass-to-charge ratio location encoding remains unaffected, allowing the model to still perceive the chemical identity information of that channel during reconstruction.

[0102] The two types of masks are independent of each other: the tubular mask performs channel zeroing on the raw intensity data before it is input to the encoder; the spatial mask determines which pixels enter the encoder. Together, they constitute a dual masking mechanism, applying reconstruction pressure in both spatial and spectral dimensions, which helps to learn a more complete spectral-spatial unified representation.

[0103] S2.2 Pre-trained Model Structure

[0104] The pre-trained framework consists of five functional modules: a physically-aware modal encoder, a linear projection adaptation layer, a shared cross-modal Transformer encoder, a modal decoder with mask unit padding mechanism, and a reconstruction head. The complete pre-trained architecture is as follows: Figure 3 As shown.

[0105] S2.2.1 Physical Sensing Modal Encoder

[0106] The internal structure of the encoder is as follows Figure 4 As shown, modal encoders with independent parameters are constructed for the three mass spectrometry imaging modes. Each encoder has the same internal structure, using a single pixel as the basic processing unit, and performs the following operations sequentially.

[0107] Intensity linear projection: Performs a channel-wise linear projection on the ion intensity vector of the input pixel, mapping the scalar intensity value of each channel to a dense 64-dimensional vector (example value), with an output shape of... .

[0108] Mass-to-charge ratio (MMR) Fourier position encoding: Sine and cosine functions of different frequencies are applied to the MMR values ​​corresponding to each channel to generate a 64-dimensional (example value) MMR position encoding vector. This continuously differentiable encoding of the physical dimensions of the MMR values ​​into a dense vector allows the model to perceive the differences in MMR values ​​between channels and their physical correlations. The parameters of the MMR encoding modules for the three modalities are completely shared, ensuring the use of a unified physical representation in the MMR coordinate space across the three modalities. This establishes a semantic correspondence at the MMR level for subsequent cross-modal attention.

[0109] Fusion Projection: After element-wise addition of the intensity linear projection result and the mass-to-charge ratio position encoding vector, the dimension is increased to 128 (exemplary value) through a linear projection layer to obtain the fused feature vector of each channel. The output shape is as follows: .

[0110] Spectral self-attention: Multi-head self-attention is performed on the channel feature sequences to model the correlation between channels with different mass-to-charge ratios at the same pixel location. A learnable physical correlation bias matrix is ​​superimposed on the attention scores. ,in Representing the The first channel and the first The prior correlation strength between channels is initialized based on the mass-to-charge ratio distance between channels and updated with gradients during training, enabling the model to learn attention patterns between channel pairs with physicochemical correlations, such as isotope peaks and adduct ions.

[0111] Attention-weighted pooling: Scalar weights are generated from the 128-dimensional feature vector of each channel using a small multilayer perceptron (MLP). After Softmax normalization, the channel feature vectors are weighted and summed, outputting a 128-dimensional (example values) pixel-level spectral aggregated feature. Attention-weighted pooling naturally assigns higher weights to high signal-to-noise ratio channels, while the weights of noisy channels approach zero, exhibiting stronger adaptability compared to mean pooling.

[0112] Spatial self-attention: Perform a multi-head self-attention operation on the visible pixel feature sequence, and add a two-dimensional spatial location code to preserve pixel spatial coordinate information, outputting... The coding features.

[0113] S2.2.2 Linear Projection Adaptation Layer

[0114] For the output of each modal encoder ( Each feature is upscaled to 256 dimensions (example value) through independent linear projection layers, outputting pixel-level feature sequences for each modality, with the shape being... The linear projection adaptation layer maps the outputs of each modal encoder to a common feature space of uniform dimension, providing a sequence of processing units with consistent dimension for the mixed inputs of subsequent cross-modal Transformers.

[0115] S2.2.3 Shared cross-modal Transformer encoder

[0116] The pixel-level feature sequences of the three modalities, after being projected through the linear projection adaptation layer, are concatenated along the sequence dimension to form... A hybrid processing unit sequence is generated, and modality type embedding and 2D spatial location encoding are added to each processing unit before inputting it into a shared cross-modal Transformer encoder. The encoder consists of 6 stacked Transformer encoder layers (exemplary value), with 8 attention heads (exemplary value). All processing units of the three modalities perform self-attention operations within the same sequence, enabling processing units of different modalities to perceive each other and establish cross-modal semantic associations. The parameter-sharing mechanism of mass-to-charge ratio location encoding gives cross-modal processing units with similar mass-to-charge ratios higher attention weights to nature, achieving cross-modal information fusion based on chemical correlation. The encoder parameters are completely shared by the three modalities, and the output shape is... .

[0117] S2.2.4 Modal decoder and mask processing unit filling

[0118] The encoder output sequence ( The embedding is divided into three parts according to modal type, and the shape of each part is as follows: For pixel locations obscured by spatial masks and channel locations zeroed out by tubular masks, learnable mask processing units are used for placeholder embedding. Spatial mask locations and tubular mask locations use separate mask processing units. After filling, the shape of each modal sequence is restored. ( (Total number of pixels). Each complete modal sequence is input into its corresponding modal decoder. The decoder consists of two (example value) Transformer decoder layers, and the output shape is... .

[0119] S2.2.5 Reconstruction Head

[0120] Output of each modal decoder ( Each signal is projected onto the corresponding modality's channel number dimension via an independent linear projection reconstruction head, outputting the intensity reconstruction value of the masked location, with shapes as follows: , , The reconstruction head is a lightweight linear layer that does not introduce additional nonlinear transformations.

[0121] S2.3 Self-Supervised Pre-training

[0122] For each mode, the reconstruction loss is calculated only at locations occluded by spatial or tubular masks; visible locations are not included in the loss calculation. The reconstruction loss for each mode is a weighted sum of the channel adaptive weighted mean square error loss and the spectral angle mapping (SAM) loss.

[0123] ,

[0124] in, and The first The reconstructed intensity value and the original intensity value of each channel, For the first Adaptive weights for each channel, Mean square error, Weighting coefficients for spectral angle mapping loss (example values) =0.1), For spectral angle mapping, To reconstruct the spectral angle, Original spectral angle. Channel adaptive weighting. The signal-to-noise ratio (SNR) of each channel is pre-calculated and fixed during training. Channels with high SNR receive higher reconstruction weights, while the weights of noise-dominant channels approach zero.

[0125] The spectral angle mapping loss is defined as the complement of the cosine similarity between the reconstructed spectral vector and the original spectral vector:

[0126] .

[0127] The mean squared error loss constrains the absolute error of the intensity values ​​of each channel, while the spectral angle mapping loss constrains the normalized relative peak structure. The two complement each other, enabling the model to maintain the accuracy of the spectral shape while reconstructing the absolute intensity values.

[0128] The total reconstruction loss of the three modes is the sum of the losses of each mode:

[0129] ,

[0130] in, To compensate for the loss in matrix-assisted laser desorption / ionization mass spectrometry imaging, To compensate for the loss in desorption electrospray ionization mass spectrometry imaging, This is the loss in secondary ion mass spectrometry imaging.

[0131] The gradient is backpropagated from the total loss to update all parameters of the encoder, linear projection adaptation layer, shared cross-modal Transformer encoder, decoder, and reconstruction head.

[0132] The training configuration is as follows (the following values ​​are all exemplary and do not limit the scope of protection of this invention): The optimizer uses the decoupled weight decay Adam optimizer (AdamW), with an initial learning rate of 1×10⁻⁶. -4 Cosine annealing learning rate scheduling, 200 training epochs, batch size of 32 slices, weight decay coefficient of 0.05; feature dimension configuration: intensity projection dimension 64, mass-to-charge ratio encoding dimension 64, fused dimension 128, linear projection adaptation layer output dimension 256, Transformer hidden layer dimension 256, feedforward network dimension 1024.

[0133] S2.4 Pre-training Result Processing

[0134] After pre-training is completed, all modal decoders and reconstruction heads are discarded, while the three-modal encoder, each modal linear projection adaptation layer, and the shared cross-modal Transformer encoder are retained. The parameter set of the retained modules constitutes the pre-trained encoding system, which is called by step S3.

[0135] S3. Obtain multimodal alignment data with pixel-level segmentation labels, introduce a cross-modal attention fusion module to fuse the multimodal features extracted by the fusion pre-trained encoding system to obtain enhanced fused features, and obtain pixel-level category probability maps through the segmentation decoder; and perform supervised fine-tuning on the multimodal alignment data with pixel-level segmentation labels to construct the segmentation system.

[0136] This step uses the pre-trained encoding system output from step S2 as the feature extraction backbone, combined with a small amount of densely labeled trimodal aligned data, and achieves pixel-level segmentation of pathological tissues through joint fine-tuning of the cross-modal attention fusion module and the lightweight segmentation decoder. This step sequentially executes three stages: feature extraction, cross-modal attention fusion and segmentation decoding, and supervised training. The downstream fine-tuning framework is as follows: Figure 5 As shown.

[0137] S3.1 Input Data and Supervision Label Acquisition

[0138] S3.1.1 Trimodal Fine-tuning Data

[0139] The input trimodal data for this step is the trimodal alignment strength data after preprocessing in step S1, and the shapes are as follows: , , During the fine-tuning stage, the input data is not masked and is input with all three modal channels intact. The data required for fine-tuning should be at least 5 sets of fully labeled slices, preferably 10 to 20 sets. When the labeled data is insufficient, data augmentation operations such as random rotation, mirror flipping, and spectral intensity perturbation can be performed on the three modal intensity data to expand the effective training samples.

[0140] S3.1.2 Dense Supervision Label Acquisition

[0141] The pixel-level segmentation labels required for the fine-tuning stage originate from the manual annotations of pathologists on H&E stained images. First, a pathologist with diagnostic qualifications performs pixel-level segmentation annotations on the digitized hematoxylin-eosin stained images. The annotation categories are defined according to the specific clinical task; for example, the categories are set as follows: 0 represents normal tissue, 1 represents tumor area, and 2 represents necrotic or stromal area. The number of categories can be adjusted according to the specific task. Subsequently, the spatial transformation matrix established in step S1 (…) is used… , , The pixel-level label mask on the hematoxylin-eosin stained image was transformed to the mass spectrometry imaging pixel coordinate system via coordinate mapping, resulting in a dense label mask with the same spatial size as the three-modal data, and its shape is as follows. Each pixel position corresponds to an integer category label, all Each pixel has a valid label, enabling dense supervision.

[0142] S3.2 Feature Extraction

[0143] Load the pre-trained encoding system weights saved in step S2, and perform forward propagation on the complete trimodal input data described in step S3.1. No mask is applied during the inference phase; all... Each pixel enters the encoder (i.e., ... Each modal data is processed by its corresponding modal encoder and linear projection adaptation layer to obtain the modal data. The pixel-level feature sequence; the three-modal feature sequences are concatenated into The input is a shared cross-modal Transformer encoder, and the output is... The mixed processing unit sequence; the mixed sequence is split into three parts by modal type embedding, each part having the following shape. Reconstructed into From the spatial feature map, pixel-level fused feature maps of the three modalities are obtained, denoted as... , , All shapes .

[0144] In this step, the parameters of the pre-trained encoding system are frozen by default, and gradients are not calculated to protect the general representation learned in the pre-training stage and reduce the computational overhead of the fine-tuning stage.

[0145] S3.3 Model Structure

[0146] This section introduces two learnable modules: a cross-modal attention fusion module and a segmentation decoder. The parameters of both are updated with gradients during the fine-tuning phase.

[0147] S3.3.1 Cross-modal attention fusion module

[0148] The pre-trained encoding system has established preliminary cross-modal semantic associations through a shared cross-modal Transformer, but its optimization objective is reconstruction loss, leading to general representation learning. To further establish cross-modal fusion features for segmentation tasks, this module... , , Perform cross-modal attention fusion oriented towards segmentation tasks.

[0149] The merging operation is performed on a per-window basis, and The spatial dimension is divided into several non-overlapping local windows (exemplary values). =8), cross-attention calculations are performed independently within each window to control computational complexity.

[0150] Taking the enhancement of matrix-assisted laser desorption / ionization mass spectrometry imaging mode as an example, in each Inside the window, with The features within the window are used as the shape of the query matrix. ),by The corresponding window features are used as the key matrix and value matrix (shape) Perform cross-attention calculation:

[0151] , , ,

[0152] ,

[0153] in, , , The projection matrix is ​​learnable. For the attention head dimension, Softmax is applied along the window of the bond matrix. Each position is normalized, meaning that each query matrix position corresponds to all positions within the window. The sum of the attention weights at each position in the key matrix is ​​1. The same calculation is performed. Enhanced features of the matrix-assisted laser desorption / ionization mass spectrometry imaging mode were obtained through residual linking polymerization:

[0154] ,

[0155] right and Perform the same operation on each of the three modes to obtain their respective enhanced features. , , All shapes The element-wise mean of the three-modal enhanced features is taken to obtain the fused feature map:

[0156] ,

[0157] The output shape is All learnable parameters in this module are updated with gradients during the fine-tuning phase.

[0158] S3.3.2 Segment Decoder

[0159] The segment decoder is a lightweight upsampling structure, with ( Taking as input, perform the following operations sequentially to output a pixel-level category probability map consistent with the spatial resolution of the input mass spectrometry imaging data.

[0160] 1×1 convolution dimensionality reduction: The fused feature map is reduced to dimensionality using a 1×1 convolution. This reduces the computational load of subsequent upsampling modules.

[0161] Upsampling Module 1: The spatial resolution is doubled through bilinear interpolation, followed by 3×3 convolution, batch normalization, and ReLU activation, resulting in the output. The feature map.

[0162] Upsampling module two: The spatial resolution is doubled again by bilinear interpolation, followed by 3×3 convolution, batch normalization, and ReLU activation, and the output is... The feature map.

[0163] Output convolution and resolution restoration: 1×1 convolution maps the number of channels to the number of classes. The output pixel-level class probabilities are activated by Softmax and then restored to their original values ​​using bilinear interpolation. ,get The pixel-level category probability map is perfectly aligned with the spatial dimensions of the supervision labels.

[0164] The above upsampling factor is an example value. In practical applications, it can be flexibly adjusted according to the ratio between the spatial resolution of the input MSI data and the required output resolution.

[0165] S3.4 Supervision and Fine-tuning

[0166] During the fine-tuning phase, only the parameters of the cross-modal attention fusion module and the segmentation decoder are updated; the parameters of the pre-trained encoding system are frozen by default. If the data scale allows, a lightweight fine-tuning can be performed on the last few layers of the shared cross-modal Transformer encoder with a very small learning rate (ideally no more than one-tenth of the normal learning rate). In this case, an independent learning rate is set for these parameters, and they are managed in groups with the learning rates of the fusion module and the decoder.

[0167] The segmentation loss is a weighted sum of the cross-entropy loss and the Dice loss:

[0168] ,

[0169] in, Weights for Dice loss (example values) =0.5);

[0170] Cross-entropy loss For all Perform pixel-by-pixel classification supervision at each pixel location:

[0171] ,

[0172] in, For pixels Category The true label (unique hot encoding). This represents the corresponding predicted probability.

[0173] Dice loss Measuring the overlap between predicted and real regions for each category effectively alleviates the class imbalance problem commonly encountered in mass spectrometry imaging segmentation tasks.

[0174] ,

[0175] in, To prevent smooth terms with a denominator of zero.

[0176] The training configuration is as follows (all values ​​below are illustrative): The optimizer used is the Adam optimizer, with an initial learning rate of 10× -3 Cosine annealing learning rate scheduling, 100 training epochs, batch size of 8 slices, weight decay coefficient of 0.01.

[0177] After fine-tuning, the parameters of the pre-trained encoding system, the cross-modal attention fusion module, and the segmentation decoder together constitute a complete segmentation system for inference in step S4.

[0178] S4. After processing the newly acquired multimodal mass spectrometry imaging data in step 1, input it into the segmentation system to obtain a pixel-level segmentation mask; and according to the spatial transformation relationship, map the pixel-level segmentation mask back to the pathological image coordinate system to generate a segmentation result that is spatially fused with the pathological morphological information.

[0179] This step, based on the complete segmentation system fine-tuned in step S3, performs inference on newly acquired trimodal mass spectrometry imaging data, outputs a pixel-level pathological tissue segmentation mask, and maps the segmentation results back to the hematoxylin-eosin staining coordinate system, providing multimodal metabolomics auxiliary information for clinical pathological diagnosis. This step can directly perform inference on new samples with the same type of lesions without retraining; for new lesion types, only a small amount of labeled data is needed to perform minor re-fine-tuning on the segmentation decoder, and the parameters of the pre-trained encoding system and the cross-modal attention fusion module can be fully reused.

[0180] S4.1 New Sample Data Preprocessing

[0181] For the newly acquired trimodal mass spectrometry imaging data, data acquisition, image registration, resolution unification, and region of interest selection are performed according to the same preprocessing procedure described in step S1. The trimodal aligned intensity data are output, with the following shapes: , , Simultaneously, the spatial transformation matrix from each modality to the hematoxylin-eosin staining coordinate system is obtained. , , This data is used for subsequent segmentation result mapping. The preprocessing procedure for new samples is completely consistent with the training phase to ensure that the distribution of the inference input data matches that of the training data.

[0182] S4.2 Reasoning Flow

[0183] Load the complete segmentation system weights saved in step S3, perform forward inference on the new sample trimodal data, without applying any mask, keeping all module parameters fixed, and without performing gradient calculations. The inference process then performs the following operations sequentially.

[0184] S4.2.1 Feature Extraction. Let... The complete three-modal data of the new sample are processed by a modal encoder and a linear projection adaptation layer to obtain the modal data. Pixel-level feature sequences; three-modal sequences concatenated into The input is then a shared cross-modal Transformer encoder, which outputs a sequence of hybrid processing units. These units are then split and reconstructed modally to obtain a three-modal pixel-level fused feature map. , , All shapes .

[0185] S4.2.2 Cross-modal attention fusion. The three-modal fusion feature map is processed by the cross-modal attention fusion module, which performs in-window cross-attention calculation, residual aggregation, and three-modal mean fusion as described in step S3.3, and outputs the fused feature map. , shape .

[0186] S4.2.3 Segmentation and Decoding. After forward propagation by the segment decoder, the output is... The pixel-level class probability map is obtained by taking the maximum value along the class dimension for each pixel position. Integer segmentation mask.

[0187] S4.3 Post-processing and coordinate mapping of segmentation results

[0188] The integer segmentation mask of the inference output can be optionally subjected to morphological post-processing operations, including performing connected component filtering on isolated noise pixels and performing morphological closing operations on segmentation boundaries to fill small holes; the above post-processing operations are optional and do not affect the core technical solution of the present invention.

[0189] Using the spatial transformation matrix obtained in step S4.1 ( , , The segmentation mask in the mass spectrometry imaging pixel coordinate system is mapped back to the hematoxylin-eosin staining image coordinate system. After bilinear interpolation and resampling to the hematoxylin-eosin staining image resolution, a high-resolution segmentation mask spatially aligned with the hematoxylin-eosin staining image is obtained. This mapped segmentation mask can be superimposed on the hematoxylin-eosin staining image to achieve spatial fusion visualization of multimodal chemical segmentation information and pathological morphological information, providing a valuable reference for clinicians.

[0190] S4.4 Two Typical Application Scenarios

[0191] Scenario 1: Direct inference from new samples of the same type of lesion

[0192] When the lesion type of a new sample is the same as that of the data used in the fine-tuning stage of step S3, the complete segmentation system weights saved in step S3 are directly loaded, and inference is performed according to the process described in step S4.2. No additional training or parameter adjustment is required, and a pixel-level segmentation mask can be directly output. Subsequent new samples of the same lesion type can be directly inferred without repeated labeling or training.

[0193] Scenario 2: Application of new lesion types

[0194] When the lesion type of the new sample differs from that of the data used in the fine-tuning stage of step S3, the parameters of the pre-trained encoding system and the cross-modal attention fusion module can be directly reused. Only a small amount of labeled data (no less than 5 labeled slice groups) needs to be used to fine-tune the segmentation decoder for the new lesion type. This fine-tuning is then performed according to the training method described in step S3.4. After completion, inference can be performed for the new lesion type. The general mass spectrometry imaging feature representation learned by the pre-trained encoding system on large-scale unlabeled data can be reused across lesion types, thereby reducing the scale of labeled data and computational cost required for applying new lesion types.

[0195] S4.5 output

[0196] The final output of this step includes the following three types of information: First, a pixel-level segmentation mask in the mass spectrometry imaging coordinate system, with the shape of... First, each pixel corresponds to an integer category label; second, a high-resolution segmentation mask mapped to the hematoxylin-eosin staining coordinate system, with spatial resolution consistent with the H&E reference image, can be directly used for pathology report generation and clinical decision support; third, a category probability map for each pixel location ( This output can be used to assess prediction confidence or perform threshold screening. The above outputs can provide multimodal metabolomics-level quantitative auxiliary information for clinicopathological tasks such as tumor region identification, tumor boundary delineation, tissue subtype classification, and metabolic heterogeneity analysis.

[0197] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal mass spectrometry imaging representation learning method for pathological segmentation, characterized in that, Includes the following steps: Step 1: Acquire multiple mass spectrometry imaging data of the same tissue sample, and preprocess the multiple mass spectrometry imaging data with the pathological image of the tissue sample as the reference coordinate system, and establish the spatial transformation relationship between each mode and the pathological image coordinate system. Step 2: Using the preprocessed multimodal data as input, the masked multimodal data is obtained through double-layer masking. The pixel-level spectral-spatial fusion features of the fused mass-to-charge ratio physical information are extracted by the physical sensing modality encoder, and then the multimodal information is fused by the shared cross-modal Transformer encoder. A self-supervised pre-training method is used to reconstruct the spectral intensity of the masked region, resulting in a pre-trained encoding system. The two-layer mask includes: a spatial mask, used to randomly mask a preset proportion of pixel positions in the spatial pixel dimension, where all channel data of the masked pixels are not included in the encoder calculation; and a tubular mask, used to uniformly and randomly zero out all pixel positions in the spectral channel dimension, where the intensity value of the zeroed channels is returned to zero but their mass-to-charge ratio position encoding is retained. The spatial mask and the tubular mask are independent of each other. Step 3: Obtain multimodal alignment data with pixel-level segmentation labels, introduce a cross-modal attention fusion module to fuse the multimodal features extracted by the fusion pre-trained encoding system to obtain enhanced fused features, and obtain pixel-level class probability maps through the segmentation decoder; and perform supervised fine-tuning on the multimodal alignment data with pixel-level segmentation labels to construct the segmentation system; Step 4: After processing the newly acquired multimodal mass spectrometry imaging data in Step 1, input it into the segmentation system to obtain a pixel-level segmentation mask; and according to the spatial transformation relationship, map the pixel-level segmentation mask back to the pathological image coordinate system to generate a segmentation result that is spatially fused with the pathological morphological information.

2. The multi-modal mass spectrometry imaging representation learning method for pathological segmentation according to claim 1, characterized in that, The preprocessing of multiple mass spectrometry imaging data includes: using the spatial resolution of the pathological image as a registration reference as a standard, downsampling the high-resolution modal data and upsampling the low-resolution modal data to achieve uniformity of the spatial resolution of each modal data. Specifically, nearest neighbor interpolation is used for upsampling of low-resolution modal data, while spatial average pooling is used for downsampling of high-resolution modal data.

3. The multi-modal mass spectrometry imaging representation learning method for pathological segmentation according to claim 1, characterized in that, The aforementioned method of establishing spatial transformation relationships from each modality to the pathological image coordinate system includes: The total ion flow map of each modality mass spectrometry imaging data is extracted as a floating image. The grayscale image of the pathological image with the registration benchmark is used as the reference image. Rigid registration and non-rigid registration are performed in sequence to establish the spatial transformation matrix from each modality to the coordinate system of the pathological image. The region of interest is delineated in the coordinate system of the pathological image and mapped to the mass spectrometry imaging data of each modality. After cropping, the multimodal alignment intensity data corresponding to the spatial pixel position is obtained, and the spatial transformation matrix is ​​stored simultaneously to complete the establishment of the spatial transformation relationship.

4. The multi-modal mass spectrometry imaging representation learning method for pathological segmentation according to claim 1, characterized in that, The physical sensing modal encoder in step 2 includes: The intensity linear projection module is used to map the ion intensity vector of the masked multimodal data pixels into a dense vector to obtain the intensity embedding of each channel. The mass-to-charge ratio Fourier position coding module is used to generate continuous position coding vectors based on the mass-to-charge ratio values ​​of each channel, and add them to the intensity embedding of the corresponding channel to obtain the channel feature vector that fuses the mass-to-charge ratio physical information. The spectral self-attention module is used to take the sequence of fused feature vectors of each channel as input, perform self-attention operation between different channels of the same pixel, and superimpose a learnable physical correlation bias matrix initialized based on mass-to-charge ratio distance, and output the feature sequence after the interaction between channels. Attention-weighted pooling layer is used to generate weights for each channel based on the feature sequence after the interaction between channels, and to perform weighted summation on the feature sequence to obtain pixel-level spectral aggregated features that aggregate all channel information; The spatial self-attention module takes pixel-level spectral aggregation features that aggregate information from all channels as input, performs self-attention operations between pixels, and adds two-dimensional spatial position encoding to output pixel-level features that fuse spatial context information.

5. The multi-modal mass spectrometry imaging representation learning method for pathological segmentation according to claim 1, characterized in that, Step 2 also includes mapping the pixel-level features with fused spatial context information to a common feature space of uniform dimension through a linear projection adaptation layer.

6. The multi-modal mass spectrometry imaging representation learning method for pathological segmentation according to claim 5, characterized in that, Step 2, which involves using a shared cross-modal Transformer encoder to perform multimodal information fusion, includes: Using features from a common feature space as input, a multimodal feature sequence is obtained by concatenation. By establishing cross-modal semantic associations through a self-attention mechanism, the parameters of which are fully shared by multiple modalities, the fusion of multimodal feature information of chemical correlation is achieved, resulting in multimodal information fusion features.

7. The multi-modal mass spectrometry imaging representation learning method for pathological segmentation according to claim 1, characterized in that, Step 3 specifically includes: Pixel-level segmentation labels on pathological images are obtained, and the pixel-level segmentation labels are mapped to the mass spectrometry imaging pixel coordinate system using a spatial transformation matrix to generate a dense label mask. Multimodal data is input into a pre-trained coding system, and pixel-level fused feature maps of each multimodal are extracted as multimodal features. Multimodal features are input into the cross-modal attention fusion module. Using a spatial window as the unit, cross-attention is calculated with any modal feature as the query and other modal features as the key and value. The enhanced fused features are then fused with the multimodal mean through residual connections. The enhanced fused features are input into the segmentation decoder, and through upsampling and convolution operations, a pixel-level class probability map is output. The segmentation loss is calculated using a dense label mask, and the parameters of the cross-modal attention fusion module and the segmentation decoder are updated.

8. The multi-modal mass spectrometry imaging representation learning method for pathological segmentation according to claim 7, characterized in that, The cross-modal attention fusion module uses the features within the window of the first modality as the query matrix and the corresponding window features of the second and third modalities as the key and value matrices to calculate cross-attention and obtain the enhanced features of the first modality. Similarly, the enhanced features of the second and third modes are calculated separately; finally, the element-wise mean of the enhanced features of multiple modes is taken to obtain the enhanced fusion features.

9. The multi-modal mass spectrometry imaging representation learning method for pathological segmentation according to claim 1, characterized in that, Step 4 also includes: For new samples with the same lesion type as those in the supervised fine-tuning stage, the segmentation system is directly invoked for inference; For new samples with new lesion types, the parameters of the pre-trained encoding system and the cross-modal attention fusion module are reused, and the segmentation decoder is fine-tuned using only the labeled data of the new lesion type before inference is performed on the new samples.

Images