A Super-Resolution Reconstruction Method for Mass Spectrometry Images Based on Transfer Learning

Abstract

The present invention discloses a method for super-resolution reconstruction of mass spectrometry images based on transfer learning, comprising the following steps: training an optical adversarial neural network model to obtain a trained optical adversarial neural network model; transferring parameters from the trained optical adversarial neural network model to an MSI neural network model to obtain an initial MSI neural network model; training the initial MSI neural network model to obtain a trained MSI neural network model; and using the trained MSI neural network model to output a target MSI image corresponding to the MSI image to be processed. The present invention first learns a partial mapping relationship from an optical image, and then transfers the model parameters learned from the optical image to the MSI model, thereby eliminating the need for a large amount of MSI data to be learned from scratch. This effectively solves the technical problem of requiring a large amount of MSI data to improve the spatial resolution of mass spectrometry images during deep learning training.

Description

Technical Field

[0001] The present invention belongs to the technical field of biochemical image processing, and in particular to a mass spectrometry image super-resolution reconstruction method based on transfer learning. Background Art

[0002] The spatial resolution of mass spectrometry imaging (MSI) is crucial for biological research. Excellent spatial resolution enables researchers to detect metabolic differences at a finer structural scale, leading to novel and significant discoveries. Current approaches to improving spatial resolution primarily focus on improving existing instrumentation and experimental conditions, such as developing nanospray desorption electrospray ionization (nanoDESI) mass spectrometry imaging devices and, for matrix-assisted laser desorption ionization (MALDI) mass spectrometry imaging, laser spot size, matrix selection, and application. However, while these instruments can produce images with relatively high spatial resolution, they require high-end instrumentation and the exploration of diverse experimental conditions. In recent years, deep learning has made significant progress in super-resolution reconstruction of biological images, such as fluorescence microscopy, electron microscopy (EM), and computed tomography (CT). However, due to dataset size limitations, its application in MSI has been limited.

[0003] In mass spectrometry imaging experiments, higher spatial resolution typically requires longer imaging times, and it is difficult to optimize both parameters simultaneously. Currently, some studies have utilized non-deep learning methods to improve the pixel resolution of mass spectrometry images through post-acquisition processing, such as bilinear interpolation, image fusion, and oversampling image deconvolution. However, these non-deep learning methods all have some technical issues, such as bilinear interpolation, which can blur the image and produce edge halos, image fusion, which requires high-precision image registration between the two imaging modes, and oversampling scanning, which is very time-consuming and labor-intensive. While using deep learning models can address these issues and improve pixel resolution and restore realistic texture details, the limited number of available datasets means that deep learning models lack sufficient data to train, making model training difficult. Consequently, deep learning has been rarely used to improve the resolution of mass spectrometry imaging.

[0004] Currently, no effective solutions have been proposed for the problems in related technologies. Summary of the Invention

[0005] Purpose of the invention: To provide a mass spectrometry image super-resolution reconstruction method based on transfer learning to overcome the above-mentioned technical problems existing in the existing related technologies.

[0006] Technical solution: A mass spectrometry image super-resolution reconstruction method based on transfer learning, the method comprising the following steps:

[0007] S1. Using a training set based on optical microscope images, the optical adversarial neural network model is trained until the model converges to obtain a trained optical adversarial neural network model;

[0008] S2. Using the fine-tuning method, the parameters of the trained optical adversarial neural network model are transferred to the MSI neural network model to obtain the initial MSI neural network model;

[0009] S3, training the initial MSI neural network model using a training set based on the MSI image until the model converges, thereby obtaining a trained MSI neural network model;

[0010] S4. Use the trained MSI neural network model to output a target MSI image corresponding to the MSI image to be processed.

[0011] In a further embodiment, the step of training the optical adversarial neural network model using the optical microscope images in the training set until the model converges to obtain the trained optical adversarial neural network model comprises the following steps:

[0012] S11. Using several Nissl-stained sagittal brain images, a training set based on optical microscopy images was created, and an optical adversarial neural network model was constructed.

[0013] S12, using the high-resolution optical microscope images and low-resolution optical microscope images in the training set to train the constructed optical adversarial neural network model, and outputting a predicted high-resolution optical microscope image;

[0014] S13. Compare the predicted high-resolution optical microscope image with the actual high-resolution optical microscope image, and use the first comprehensive loss function to optimize the optical adversarial neural network model to obtain a trained optical adversarial neural network model.

[0015] In a further embodiment, the optical microscope images in the training set include original high-resolution optical microscope images and low-resolution optical microscope images obtained by downsampling the original high-resolution optical microscope images.

[0016] In a further embodiment, the calculation formula of the first comprehensive loss function is:

[0017]

[0018]

[0019]

[0020] Where, L percepIndicates perceived loss;

[0021] λ and η represent hyperparameters, and λ = 5 × 10 -3 ,η=1×10 -2 ;

[0022] It means fighting against loss;

[0023] L1 represents L1-norm loss;

[0024] and Represents the operation of taking the average of all real data and fake data in the mini-batch;

[0025] X r and x f represent real images and images generated by the generator respectively;

[0026] D Ra represents the relativistic average discriminator;

[0027] G(xi) and y represent the image restored by the model and the real image respectively;

[0028] Represents the operation of taking the sum of absolute differences over all images in the mini-batch.

[0029] In a further embodiment, the fine-tuning method is used to transfer the parameters of the trained optical adversarial neural network model to the MSI neural network model to obtain the initial MSI neural network model, which includes the following steps:

[0030] S21. Save the parameters and network structure of the trained optical adversarial neural network model;

[0031] S22. Build a neural network model for mass spectrometry imaging based on the network structure of the optical adversarial neural network model, while keeping the structure of each layer unchanged;

[0032] S23. Add a classifier consisting of a convolutional layer and a fully connected layer to the end of the constructed mass spectrometry imaging neural network model and calculate the cross entropy loss.

[0033] S24. Copy all parameters in the saved optical adversarial neural network model to the corresponding network structure positions in the mass spectrometry imaging neural network model, and use random numbers to replace the parameters of the classifier to generate an initial MSI neural network model.

[0034] In a further embodiment, the cross entropy loss is used to exploit the intermolecular relationships of the MSI images to enable the initial MSI neural network model to distinguish different classes of biomolecules.

[0035] In a further embodiment, the training of the initial MSI neural network model using the training set based on the MSI image until the model converges to obtain the trained MSI neural network model comprises the following steps:

[0036] S31, obtaining MSI images of the sagittal plane of several mouse brain tissues, and creating a training set based on the MSI images;

[0037] S32, using the high-resolution MSI image and the low-resolution MSI image in the training set to train the initial MSI neural network model, and output a predicted super-resolution MSI image;

[0038] S33. Compare the predicted super-resolution MSI image with the actual super-resolution MSI image, and optimize the MSI neural network model using the second comprehensive loss function to obtain a trained MSI neural network model.

[0039] In a further embodiment, the MSI images in the training set include high-resolution MSI images and low-resolution MSI images obtained by downsampling the high-resolution MSI images.

[0040] In a further embodiment, the step of obtaining a plurality of sagittal MSI images of mouse brain tissue and preparing a training set based on the MSI images further comprises the following steps:

[0041] The SciPy, NumPy, and matplotlib Python packages are used to simulate two-dimensional analytical probe sampling on a point source to calculate a specific point spread function.

[0042] The high-resolution MSI image is convolved with a specific point spread function and then downsampled. The convolved image is downsampled using the bicubic kernel function in MATLAB to obtain the corresponding synthetic low-resolution MSI image.

[0043] In a further embodiment, the calculation formula of the second comprehensive loss function is:

[0044]

[0045] Where, represents a hyperparameter, and

[0046] L c represents the cross entropy loss.

[0047] Beneficial effects: By using optical microscope images to train the adversarial neural network model, and transferring the parameters of the trained adversarial neural network model to the MSI neural network model using the idea of transfer learning, the learning of a small sample size target data set (MSI image) can be enhanced, so that only a small number of single ion image pairs are needed to complete the training of the MSI neural network model. In the training stage, the first comprehensive loss function and the second comprehensive loss function are used respectively to optimize the optical adversarial neural network model and the MSI neural network model, so that the model can learn the detailed texture information of the image and improve the prediction effect of the model. Finally, in the output stage, the learned mapping relationship is used to perform higher pixel resolution and more realistic texture on the new MSI image. In order to predict the details of the brain, compared with the traditional mass spectrometry image super-resolution reconstruction method, the present invention introduces transfer learning, and uses the idea that the distribution of most metabolites in the brain is closely related to the anatomical structure, and it is easier to obtain a large number of optical images than MSI images. First, some mapping relationships are learned from the optical images, and then the model parameters learned from the optical images are transferred to the MSI model, so that there is no need for a large amount of MSI data to learn from scratch. It has the advantages of superior performance and fast training, and effectively solves the problems of limited spatial resolution of mass spectrometry imaging images, time-consuming mass spectrometry image acquisition process, and no large data set for training, thereby effectively solving the technical problem of requiring a large amount of MSI data for learning to improve the spatial resolution of mass spectrometry images in deep learning training. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 is a flow chart of a mass spectrometry image super-resolution reconstruction method based on transfer learning according to an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of a mass spectrometry image super-resolution reconstruction method based on transfer learning according to an embodiment of the present invention;

[0051] Figure 3 This is an intuitive comparison diagram of the super-resolution effect in a mass spectrometry image super-resolution reconstruction method based on transfer learning according to an embodiment of the present invention;

[0052] Figure 4 This is a statistical comparison chart of indicators of super-resolution effects in a mass spectrometry image super-resolution reconstruction method based on transfer learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In the following description, numerous specific details are provided to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without one or more of these details. In other instances, certain technical features well known in the art are not described to avoid confusion with the present invention.

[0054] According to an embodiment of the present invention, a mass spectrometry image super-resolution reconstruction method based on transfer learning is provided.

[0055] like Figures 1-4 As shown, according to the mass spectrometry image super-resolution reconstruction method based on transfer learning according to an embodiment of the present invention, the method includes the following steps:

[0056] S1. Using a training set based on optical microscope images, the optical adversarial neural network model is trained until the model converges to obtain a trained optical adversarial neural network model;

[0057] The method of training the optical adversarial neural network model using the optical microscope images in the training set until the model converges to obtain the trained optical adversarial neural network model includes the following steps:

[0058] S11. Using several Nissl-stained sagittal brain images, a training set based on optical microscopy images was created, and an optical adversarial neural network model was constructed.

[0059] Specifically, the optical microscope images in the training set include original high-resolution optical microscope images and low-resolution optical microscope images obtained by downsampling the original high-resolution optical microscope images.

[0060] In addition, in this embodiment, the process of obtaining Nissl-stained sagittal brain images to create a training set based on optical microscope images and obtaining MSI images of the sagittal plane of several mouse brain tissues and creating a training set based on MSI images also includes preprocessing the obtained optical microscope images or MSI images, including:

[0061] Denoising: Optical microscope images or MSI images often contain noise (such as cluttered background pixels and sudden brightness changes), so their effects need to be removed or reduced. For example, wavelet denoising algorithms, mean filtering, median filtering, or non-local mean filtering can be used to achieve this.

[0062] Normalization: If multiple optical microscopy or MSI images are to be combined and compared, the images need to be normalized to achieve uniform image quality and color range. Linear scaling or nonlinear normalization methods can be used.

[0063] Labeling and Segmentation: Optical microscopy or MSI images may require manual or automated labeling or image segmentation to distinguish different objects or structures and reduce errors in image analysis. Common algorithms include threshold segmentation, region growing, active contour models, or deep learning.

[0064] Parameter adjustment: By adjusting the contrast, brightness, saturation, and sharpness of the image, the clarity and readability of the image can be improved. Specifically, in this embodiment, a pre-built neural network model is used to output adjustment parameters corresponding to the position information of the Nissl-stained sagittal brain image, and the output adjustment parameters are used to adjust the acquired Nissl-stained sagittal brain image. Specifically, the following steps are included:

[0065] Data collection: Nissl-stained sagittal brain images and their corresponding positional information were collected from the sample brain.

[0066] Data processing: De-noise and pre-process the data, and mark or classify the location information of the samples to ensure that the data is error-free.

[0067] Neural network model training: Using machine learning or deep learning methods, a neural network model is constructed and trained using the collected data to obtain adjustment parameter outputs corresponding to the position information of Nissl-stained sagittal brain images.

[0068] Adjustment parameter output: Using a pre-trained neural network model and new Nissl-stained sagittal brain images, with the help of a computer program, position information is input into the neural network model to obtain the corresponding adjustment parameter outputs such as contrast, brightness, saturation, and sharpness.

[0069] Nissl-stained sagittal brain image adjustment: Apply the adjustment parameters output by the neural network to adjust the acquired Nissl-stained sagittal brain images. Common correction and adjustment algorithms include contrast enhancement, brightness adjustment, saturation adjustment, and sharpness enhancement.

[0070] Feedback and Adjustment: Observe the processed Nissl-stained sagittal brain images and provide verification and feedback. If the results are not satisfactory, the neural network model and data need to be modified and retrained to obtain more accurate adjustment parameters and processing results.

[0071] S12, using the high-resolution optical microscope images and low-resolution optical microscope images in the training set to train the constructed optical adversarial neural network model, and outputting a predicted high-resolution optical microscope image;

[0072] S13. Compare the predicted high-resolution optical microscope image with the actual high-resolution optical microscope image, and use the first comprehensive loss function to optimize the optical adversarial neural network model to obtain a trained optical adversarial neural network model.

[0073] In this embodiment, a large number (about 15,000) of Nissl-stained sagittal brain image pairs were used to train the optical model (optical adversarial neural network model). These images can be easily obtained from the Allen Brain Atlas database and can clearly show the anatomical structure of brain slices. In this embodiment, the difference between the model prediction output and the real image was measured, and the model was optimized using three loss functions (i.e., the first comprehensive loss function), including L1-norm loss L1, adversarial loss L2, and α-valued loss L3. GRa and perceptual loss L percep . Adversarial loss and perceptual loss allow the model to learn detailed texture information of the image. In general, the calculation formula of the first comprehensive loss function is:

[0074]

[0075]

[0076]

[0077] Where, L percep represents the perceptual loss, which is the L1 loss of the features calculated using the VGG network;

[0078] λ and η represent hyperparameters, and λ = 5 × 10 -3 ,η=1×10 -2 ;

[0079] represents the adversarial loss, which is used to distinguish the predicted image from the target image;

[0080] L1 represents L1-norm loss, which is defined as the sum of the absolute differences between the predicted image and the true image, and is used to evaluate the 1-norm distance between the super-resolution reconstructed image and the target image;

[0081] and Represents the operation of taking the average of all real data and fake data in the mini-batch;

[0082] X r and x f represent real images and images generated by the generator respectively;

[0083] D Ra represents the relativistic average discriminator;

[0084] G(xi) and y represent the image restored by the model and the real image respectively;

[0085] Represents the operation of taking the sum of absolute differences over all images in the mini-batch.

[0086] S2. Using the fine-tuning method, the parameters of the trained optical adversarial neural network model are transferred to the MSI neural network model to obtain the initial MSI neural network model;

[0087] The method of using the fine-tuning method to transfer the parameters of the trained optical adversarial neural network model to the MSI neural network model to obtain the initial MSI neural network model includes the following steps:

[0088] S21. Save the parameters and network structure of the trained optical adversarial neural network model;

[0089] S22. Build a neural network model for mass spectrometry imaging based on the network structure of the optical adversarial neural network model, while keeping the structure of each layer unchanged;

[0090] S23. Add a classifier consisting of a convolutional layer and a fully connected layer to the end of the constructed mass spectrometry imaging neural network model and calculate the cross entropy loss.

[0091] Specifically, the cross entropy loss is used to utilize the intermolecular relationships of the MSI image to enable the initial MSI neural network model to distinguish different categories of biomolecules.

[0092] S24. Copy all parameters in the saved optical adversarial neural network model to the corresponding network structure positions in the mass spectrometry imaging neural network model, and use random numbers to replace the parameters of the classifier to generate an initial MSI neural network model.

[0093] In this example, transfer learning is introduced to transfer parameters learned from a large reference dataset (optical images) to the target model to enhance learning of the smaller target dataset (MSI images). In transfer learning, only a few (approximately 10) existing single-ion MSI image pairs are needed to train the MSI model.

[0094] In this example, fine-tuning is used to transfer parameters, transferring all parameters from the optical model to the MSI model intact while preserving the structure of each layer. Training is then re-used using the MSI images without freezing the parameters of any layer.

[0095] The additional cross-entropy loss, Lc, leverages the intermolecular relationships in MSI images, enabling the model to distinguish between different classes of biomolecules. Because MSI images contain intermolecular relationships, all MSI images in the training dataset are classified based on these relationships, with those with strong intermolecular relationships and similar distributions grouped together. Furthermore, a classifier consisting of convolutional and fully connected layers is added to the end of the original ESRGAN model, and a cross-entropy loss is calculated to enhance model learning. This cross-entropy loss can be optimized to enable the model to distinguish between different classes of biomolecules.

[0096] S3, training the initial MSI neural network model using a training set based on the MSI image until the model converges, thereby obtaining a trained MSI neural network model;

[0097] The method of training the initial MSI neural network model using a training set based on MSI images until the model converges to obtain the trained MSI neural network model comprises the following steps:

[0098] S31, obtaining MSI images of the sagittal plane of several mouse brain tissues, and creating a training set based on the MSI images;

[0099] Specifically, the MSI images in the training set include high-resolution MSI images and low-resolution MSI images obtained by downsampling these high-resolution MSI images.

[0100] This embodiment also includes data augmentation of random horizontal flipping and 90-degree rotation of the training dataset (including the training set based on optical microscope images and the training set based on MSI images), and the magnification ratio of the training dataset is 100. The mini-batch size is set to 16. A larger patch size is conducive to training deeper networks because a larger receptive field helps capture more semantic information. However, it increases training time and consumes more computing resources. In order to balance model performance and training time, this implementation sets the spatial patch size of the ground truth image to 128×128. This embodiment uses a single graphics processing unit (GPU, NVIDIA RTX 3090, 24GB memory) to accelerate the training process of the deep learning model. It only takes a few seconds to predict a new MSI image and improve its spatial resolution through the trained network. The training and testing process can be further accelerated by using a more powerful GPU or parallel computing on multiple GPUs.

[0101] S32, using the high-resolution MSI image and the low-resolution MSI image in the training set to train the initial MSI neural network model, and output a predicted super-resolution MSI image;

[0102] S33. Compare the predicted super-resolution MSI image with the actual super-resolution MSI image, and optimize the MSI neural network model using the second comprehensive loss function to obtain a trained MSI neural network model.

[0103] Specifically, the calculation formula of the second comprehensive loss function is:

[0104]

[0105] Where, represents a hyperparameter, and

[0106] L c represents the cross entropy loss.

[0107] In this embodiment, in order to simulate the imaging process of MSI, a LR MSI image generator was made, which can generate LR images by simulating specific point spread functions (PSFs) under different instrument parameters (such as the characteristic width and pixel resolution of the probe). The PSF was calculated by simulating two-dimensional analytical probe sampling on a point source using SciPy, NumPy and matplotlib Python packages. First, the HR MSI image was convolved with the PSF and then downsampled. Then, these convolved images were downsampled using the bicubic kernel function in MATLAB to obtain the corresponding synthetic LR images. MOSR was trained based on synthetic LR and measured HR MSI image pairs, rather than measured LR and HR image pairs from the same brain slice. These processes can reduce the height and width of the image to 1 / 4 of the original size, indicating a scaling factor of 4. If the height or width of the image before downsampling is not a multiple of 4, the image will be cropped from the upper left corner as the origin to ensure that the height and width are both the maximum integer multiples of 4. The cropped image is then used as the HR image, and the corresponding LR image is obtained by downsampling. In this way, each HR image has a corresponding LR image with a height and width of 1 / 4, respectively.

[0108] S4. Use the trained MSI neural network model to output a target MSI image corresponding to the MSI image to be processed.

[0109] In addition, in order to better understand the above technical solutions of the present invention, the design principles and original intentions of the present invention are described below:

[0110] 1) Design principle:

[0111] like Figure 2The figure shows the principle diagram of MOSR. MOSR consists of three main stages: (i) the training stage of the optical model: using rich optical images to train an artificial neural network (ANN); (ii) the training stage of the mass spectrometry imaging (MSI) model: by introducing transfer learning, another ANN is trained using existing MSI images; (iii) the inference stage of the MSI model: super-resolution (SR) reconstruction is performed on new low-resolution MSI images. a. Obtain high-resolution (HR) optical images from the Allen Brain Atlas database, and downsample the HR images to obtain the corresponding low-resolution (LR) images using the point spread function and bicubic kernel function in MATLAB. The optical model is trained by using the synthetic LR optical image as input and the corresponding HR optical image as the target image. During the training process, the output of the optical model is compared with the target image through three loss functions: (1) "L1 norm loss L1" evaluates the 1-norm distance between the SR reconstructed image and the target image; (2) "adversarial loss" Use another ANN (called “discriminator”) to try to distinguish between the predicted image and the target image; (3) “perceptual loss L percep "Use the trained VGG19 model to distinguish the predicted image from the target image in the feature space. Optimization uses the Adam algorithm, and the generator and discriminator networks are updated alternately until the model converges. b. Transfer the parameters of the optical model to the MSI model through fine-tuning for transfer learning. The MSI model is trained by using LR MSI images as input and the corresponding HR MSI images as target images, and is optimized using the above three loss functions and the additional "cross entropy loss Lc" and the Adam algorithm. c. A new LR MSI image that has never been seen by the MSI model is taken as input, and the MSI model outputs an SR predicted MSI image.

[0112] The main network architecture of MOSR adopts ESRGAN, which can achieve better visual quality and more realistic textures. It is a GAN-based network consisting of a generator and a discriminator.

[0113] The generator can output a super-resolution predicted image from the LR image through nonlinear calculations. In the MOSR network of the present invention, most of the generator's calculations are performed in the LR feature space. First, the features of the LR image are extracted through the convolutional layer, and then the residual dense block (RRDB) is used as the basic block to input into the repeating unit. More layers and connections can always improve performance. Therefore, RRDB adopts a deeper and more complex structure than the original residual block in SRGAN, combining a multi-level residual network and dense connections. Specifically, RRDB has a residual-in-residual structure, and residual learning is used at different levels. An RRDB block consists of 3 Dense blocks, each of which consists of 5 convolutional layers and 4 LeakyReLU layers in turn. The present invention stacks a total of 23 RRDB blocks in the repeating unit and outputs the calculation results through the convolutional layer. Then, skip connections are used between the input and output of the repeating unit to avoid gradient explosion and gradient vanishing during deep network training. Finally, two 2x upsampling layers are used to increase the height and width of the LR image by 4 times, outputting a super-resolution predicted image with rich texture.

[0114] To help the generator output super-resolution predicted images with better visual quality, the discriminator typically calculates an adversarial loss by determining whether the image comes from a real dataset or from the generator. Our MOSR model uses a relativistic discriminator based on the VGG network, which can predict the probability that a real image is more realistic than a fake image and helps learn sharper edges and more detailed textures.

[0115] 2) Design intention:

[0116] Due to limitations in high-end equipment and sampling efficiency, such as long imaging time, cumbersome sample preparation, and extensive experimental parameter optimization, high-quality HSR-MSI images are difficult to obtain. In contrast, many high-resolution optical images are available in public databases. Therefore, the present invention constructs a MOSR framework that can transfer knowledge learned from optical images to MSI images, thereby generating predicted HSR-MSI images. The MOSR model consists of three stages: a training stage for the optical and MSI models and an inference stage for the MSI model. Briefly, during the training stage, a large number (approximately 15,000) of Nissl-stained sagittal brain image pairs are used to train the optical model. These images are easily available from the Allen Brain Atlas database and clearly show the anatomical structure of brain slices. Then, the difference between the model output and the true HR images is measured, and the model is optimized using three loss functions: L1-norm loss, adversarial loss, and perceptual loss. The adversarial loss and perceptual loss allow the model to learn detailed texture information of the image. Next, transfer learning is introduced to transfer parameters learned from a large reference dataset (optical images) to the target model to enhance learning on a smaller target dataset (MSI images). Through transfer learning, only a small number (approximately 10) of existing single-ion image pairs are needed to train the MSI model. The goal of the inference phase is to leverage the learned mapping to predict new MSI images with higher pixel resolution and more realistic texture detail.

[0117] 3) Comparison of super-resolution effects:

[0118] like Figure 3 The following is a visual comparison of the existing method and the MOSR super-resolution effect. The reference image is a high-resolution real MSI image. The PSNR (peak signal-to-noise ratio) indicator is shown in the upper left corner of the figure, and the profile intensity is shown in the upper right corner.

[0119] like Figure 4 The figure shows the MOSR indicator statistics. In this embodiment, the existing method is compared with the MOSR super-resolution effect, and the relative standard deviation, peak signal-to-noise ratio and resolution scaling error are respectively counted.

[0120] In summary, with the help of the above technical solution of the present invention, by using optical microscope images to train the adversarial neural network model, and transferring the parameters of the trained adversarial neural network model to the MSI neural network model using the idea of transfer learning, the learning of the target data set (MSI image) with a small sample size can be enhanced, so that only a small number of single ion image pairs are needed to complete the training of the MSI neural network model. In the training stage, the first comprehensive loss function and the second comprehensive loss function are used respectively to optimize the optical adversarial neural network model and the MSI neural network model, so that the model can learn the detailed texture information of the image and improve the prediction effect of the model. Finally, in the output stage, the learned mapping relationship is used to perform higher pixel resolution on the new MSI image. Compared with the traditional mass spectrometry image super-resolution reconstruction method, the present invention introduces transfer learning, taking advantage of the fact that the distribution of most metabolites in the brain is closely related to the anatomical structure, and obtaining a large number of optical images is easier than MSI images. First, part of the mapping relationship is learned from the optical image, and then the model parameters learned from the optical image are transferred to the MSI model, thereby eliminating the need for a large amount of MSI data for learning from scratch. It has the advantages of superior performance and fast training, effectively solving the problems of limited spatial resolution of mass spectrometry imaging images, time-consuming mass spectrometry image acquisition process, and lack of large data sets for training, thereby effectively solving the technical problem of requiring a large amount of MSI data for learning to improve the spatial resolution of mass spectrometry images in deep learning training.

[0121] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A mass spectrometry image super-resolution reconstruction method based on transfer learning, characterized in that: The method comprises the following steps: S1. Using a training set based on optical microscope images, the optical adversarial neural network model is trained until the model converges to obtain a trained optical adversarial neural network model; The optical adversarial neural network model is optimized using the first comprehensive loss function to obtain a trained optical adversarial neural network model; The calculation formula of the first comprehensive loss function is: Where, L percep Indicates perceived loss; λ and η represent hyperparameters, and λ = 5 × 10 -3 ,η=1×10 -2 ; It means fighting against loss; L1 represents L1-norm loss; and Represents the operation of taking the average of all real data and fake data in the mini-batch; x r and x f represent real images and images generated by the generator respectively; D Ra represents the relativistic average discriminator; G(x i ) and y represent the image restored by the model and the real image respectively; represents the operation of taking the sum of absolute differences of all images in the mini-batch; S2. Using the fine-tuning method, the parameters of the trained optical adversarial neural network model are transferred to the MSI neural network model to obtain the initial MSI neural network model; The MSI neural network model is optimized using the second comprehensive loss function to obtain a trained MSI neural network model; The calculation formula of the second comprehensive loss function is: Where, represents a hyperparameter, and L c represents the cross entropy loss; S3, training the initial MSI neural network model using a training set based on the MSI image until the model converges, thereby obtaining a trained MSI neural network model; Acquiring several sagittal MSI images of mouse brain tissue and preparing a training set based on the MSI images further includes the following steps: The SciPy, NumPy, and matplotlib Python packages are used to simulate two-dimensional analytical probe sampling on a point source to calculate a specific point spread function. The high-resolution MSI image is convolved with a specific point spread function and then downsampled. The convolved image is downsampled using the bicubic kernel function in MATLAB to obtain the corresponding synthetic low-resolution MSI image. S4. Use the trained MSI neural network model to output a target MSI image corresponding to the MSI image to be processed.

2. The mass spectrometry image super-resolution reconstruction method based on transfer learning according to claim 1, characterized in that: The optical adversarial neural network model is trained using the optical microscope images in the training set until the model converges. The trained optical adversarial neural network model is obtained by the following steps: S11. Using several Nissl-stained sagittal brain images, a training set based on optical microscopy images was created, and an optical adversarial neural network model was constructed. S12, using the high-resolution optical microscope images and low-resolution optical microscope images in the training set to train the constructed optical adversarial neural network model, and outputting a predicted high-resolution optical microscope image; S13. Compare the predicted high-resolution optical microscope image with the actual high-resolution optical microscope image, and use the first comprehensive loss function to optimize the optical adversarial neural network model to obtain a trained optical adversarial neural network model.

3. The mass spectrometry image super-resolution reconstruction method based on transfer learning according to claim 2, characterized in that: The optical microscope images in the training set include original high-resolution optical microscope images and low-resolution optical microscope images obtained by downsampling the original high-resolution optical microscope images.

4. The mass spectrometry image super-resolution reconstruction method based on transfer learning according to claim 1, characterized in that The method of using the fine-tuning method to transfer the parameters of the trained optical adversarial neural network model to the MSI neural network model to obtain the initial MSI neural network model includes the following steps: S21. Save the parameters and network structure of the trained optical adversarial neural network model; S22. Build a neural network model for mass spectrometry imaging based on the network structure of the optical adversarial neural network model, while keeping the structure of each layer unchanged; S23. Add a classifier consisting of a convolutional layer and a fully connected layer to the end of the constructed mass spectrometry imaging neural network model and calculate the cross entropy loss. S24. Copy all parameters in the saved optical adversarial neural network model to the corresponding network structure positions in the mass spectrometry imaging neural network model, and use random numbers to replace the parameters of the classifier to generate an initial MSI neural network model.

5. The mass spectrometry image super-resolution reconstruction method based on transfer learning according to claim 4, characterized in that: The cross entropy loss is used to exploit the intermolecular relationships of MSI images to enable the initial MSI neural network model to distinguish different classes of biomolecules.

6. The mass spectrometry image super-resolution reconstruction method based on transfer learning according to claim 1, characterized in that: The method of training the initial MSI neural network model using the training set based on the MSI image until the model converges to obtain the trained MSI neural network model comprises the following steps: S31, obtaining MSI images of the sagittal plane of several mouse brain tissues, and creating a training set based on the MSI images; S32, using the high-resolution MSI image and the low-resolution MSI image in the training set to train the initial MSI neural network model, and output a predicted super-resolution MSI image; S33. Compare the predicted super-resolution MSI image with the actual super-resolution MSI image, and optimize the MSI neural network model using the second comprehensive loss function to obtain a trained MSI neural network model.

7. The mass spectrometry image super-resolution reconstruction method based on transfer learning according to claim 6, characterized in that: The MSI images in the training set include high-resolution MSI images and low-resolution MSI images obtained by downsampling these high-resolution MSI images.

Images