A method for registering mass spectrometry imaging data with whole-slide pathology sections

Abstract

The present invention discloses a method for registering mass spectrometry imaging data with whole-slide pathology sections. By performing deep learning interpolation on the mass spectrometry imaging data and using affine transformation, the interpolation method based on deep learning is introduced into the research related to whole-slide pathology sections (WSIs), providing a reliable magnification factor for the whole-slide pathology section (WSI) data. The registration method of the present invention can present morphological features and chemical distributions in one image, enabling more direct observation of the morphological features of WSIs and the molecular distribution characteristics of WSIs, greatly improving the accuracy and precision of the registered images.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image multimodal registration, and specifically relates to a registration method for mass spectrometry imaging data and whole-slide pathology sections. Background Art

[0002] Mass spectrometry imaging (MSI) is a powerful molecular imaging technique that can perform untargeted mass spectrometry analysis on different sites on a section and visualize them individually, allowing in-situ analysis of multiple known or unknown metabolites simultaneously and providing spatially resolved chemical information. Due to its ability to detect multiple molecular species simultaneously, it has been used in clinical fields such as drug discovery and personalized medicine. However, the disadvantage of mass spectrometry imaging (MSI) is its relatively low resolution and less capture of the morphological features of the section itself.

[0003] The characteristics of whole-slide pathology sections (WSI) are high resolution, which can observe finer morphological features in tissues and the texture of cells in the sections, but cannot capture molecular-level features. Due to its high resolution for examining minute morphological details, it is possible to distinguish regions of interest (ROI) in the sample. However, the judgment of pathologists largely depends on their prior knowledge and experience, and thus is inevitably subjective.

[0004] To reduce the subjective judgment of pathologists, the concept of computer vision has been introduced into this field, and the main application means is: combining mass spectrometry imaging (MSI) and whole-slide pathology sections (WSI) through machine learning algorithms. The common method is image alignment, by mapping the molecular information of mass spectrometry imaging (MSI) to the corresponding regions of whole-slide pathology sections (WSI) in order to combine molecular information with morphological features. In previous studies, many researchers only aligned the relative positions of mass spectrometry imaging (MSI) and whole-slide pathology sections (WSI). This form of image alignment has the following defects:

[0005] (1) Most studies directly magnify the low-resolution mass spectrometry imaging (MSI) and use the magnified mass spectrometry imaging (MSI) to be consistent with the whole-slide pathology section (WSI). This direct magnification of mass spectrometry imaging (MSI) may cause the relative positions in the mass spectrometry imaging (MSI) to shift, resulting in poor final alignment;

[0006] (2) Some relative position offsets may occur during sectioning, and even when using two closely cut sections, it is impossible to perfectly match the two sections;

[0007] The instability of biological samples and the huge difference in resolution have led to the lack of a mature and highly implementable registration method for mass spectrometry imaging (MSI) and whole-slide pathology sections (WSI). Summary of the Invention

[0008] Objective of the Invention: To solve the problems of low accuracy and poor precision existing in the registration methods of existing mass spectrometry imaging (MSI) and whole-slide pathology (WSI), the present invention proposes a registration method for mass spectrometry imaging data and whole-slide pathology sections. By performing deep learning interpolation on the mass spectrometry imaging data and using affine transformation, the relative error caused by sectioning is reduced, and the accuracy and precision of the registered images are greatly improved.

[0009] Technical Solution: A registration method for mass spectrometry imaging data and whole-slide pathology sections includes the following steps:

[0010] Obtain sections of a histological sample to be measured, and produce corresponding whole-slide pathology sections and matrix-assisted laser desorption / ionization mass spectrometry imaging (MALDI-MSI) images; the MALDI-MSI images are obtained by performing matrix-assisted laser desorption / ionization on adjacent sections; the adjacent sections are sections adjacent to the whole-slide pathology sections;

[0011] Use the rotating calipers algorithm to obtain the minimum bounding rectangles of the whole-slide pathology sections and the MALDI-MSI images; based on the minimum bounding rectangles, perform position correction on the whole-slide pathology sections and the MALDI-MSI images to obtain corrected whole-slide pathology sections and corrected MALDI-MSI images;

[0012] Use the Canny edge operator to extract the edge contours of the corrected whole-slide pathology sections and the edge contours of the corrected MALDI-MSI images to obtain the edge contours of the corrected whole-slide pathology sections and the edge contours of the corrected MALDI-MSI images;

[0013] Input the corrected MALDI-MSI images into a super-resolution deep learning neural network, and output mass spectrometry images with the same resolution as the corrected whole-slide pathology sections;

[0014] Use affine transformation to perform primary registration on the corrected whole-slide pathology sections and the output mass spectrometry images to obtain primarily registered images;

[0015] Use the moving least squares method to perform secondary registration on the primarily registered images to obtain the final registered images.

[0016] Further, the obtaining of sections of the histological sample to be measured and producing corresponding whole-slide pathology sections and MALDI-MSI images specifically includes:

[0017] Perform cryosectioning on the histological sample to be measured to obtain multiple histological sections;

[0018] Select a section including histological features from multiple histological sections, perform histological staining on the section including histological features to obtain a stained section;

[0019] Scan the stained section through a digital slide scanning system to obtain a whole-slide pathology image;

[0020] Select a histological section adjacent to the section including histological features, denote it as an adjacent section, and perform matrix-assisted laser desorption / ionization on the adjacent section to obtain a matrix-assisted laser desorption / ionization mass spectrometry imaging image.

[0021] Further, based on the minimum bounding rectangle, perform position correction on the whole-slide pathology image and the matrix-assisted laser desorption / ionization mass spectrometry imaging image to obtain a corrected whole-slide pathology image and a corrected matrix-assisted laser desorption / ionization mass spectrometry imaging image, specifically including:

[0022] Obtain corresponding position correction angles according to the four vertices of the minimum bounding matrix;

[0023] Rotate the whole-slide pathology image and the matrix-assisted laser desorption / ionization mass spectrometry imaging image according to the corresponding position correction angles to obtain a corrected whole-slide pathology image and a corrected matrix-assisted laser desorption / ionization mass spectrometry imaging image.

[0024] Further, the super-resolution deep learning neural network includes:

[0025] A bicubic interpolation sub-network for performing bicubic interpolation on the input corrected matrix-assisted laser desorption / ionization mass spectrometry imaging image to obtain an interpolated low-resolution image ILR;

[0026] A feature extraction sub-network for extracting multiple image patches from the interpolated low-resolution image ILR, performing convolution operations on each image patch to obtain feature vectors, and all feature vectors form a feature matrix;

[0027] A non-linear mapping sub-network for performing non-linear mapping on the feature matrix through convolution operations to obtain a new feature matrix;

[0028] An upsampling sub-network for performing upsampling operations on the new feature matrix to obtain a mass spectrometry imaging image with the same resolution as the corrected whole-slide pathology image.

[0029] Further, use affine transformation to perform initial registration on the corrected whole-slide pathology image and the output mass spectrometry image to obtain an initially registered image, specifically including:

[0030] Denote the four vertex coordinates of the output mass spectrometry imaging image as (x M1 ,y M1 ), (xM2 , y M2 ), (x M3 , y M3 ), (x M4 , y M4 ); Denote the four vertex coordinates of the corrected whole - field pathological section as (x W1 , y W1 ), (x W2 , y W2 ), (x W3 , y W3 ), (x W4 , y W4 );

[0031] Through the affine transformation equations \(x\) Wn = a1x Mn + b1y Mn + c1 and \(y\) Wn = a2x Mn + b2y Mn + c2, transform the four vertex coordinates of the output mass spectrometry imaging image to (x W1 , y W1 ), (x W2 , y W2 ), (x W3 , y W3 ), (x W4 , y W4 ); where a1, a2, b1, b2, c1, c2 respectively represent the parameters for different geometric transformations.

[0032] Furthermore, the use of the moving least - squares method to perform secondary registration on the initially registered image specifically includes:

[0033] Use the moving least - squares method to construct a corresponding deformation function \(f\) v (v) for each pixel point \(v\) on the initially registered image, so that the edge contour of the mass spectrometry imaging image after affine transformation deforms towards the edge contour of the corrected whole - field pathological section. Calculate the positions of each pixel point after deformation through the deformation function \(f\) v (v):

[0034]

[0035] In the formula:

[0036]

[0037]

[0038]

[0039]

[0040]

[0041] In the formula, p i represents the i-th deformation control point in the horizontal axis direction, and p j represents the j-th deformation control point in the vertical axis direction. The deformation control points correspond to the edge contour information of the mass spectrometry imaging image; q i represents the i-th target control point in the horizontal axis direction, and q j represents the j-th target control point in the vertical axis direction. The target control points correspond to the edge contour information of the whole-slide pathology section; w i represents the weight; α represents a parameter for adjusting the deformation effect; based on the positions of the pixels after deformation, the initially registered image is subjected to secondary registration.

[0042] The present invention also discloses a computer storage medium storing a program for a method of registering mass spectrometry imaging data with a whole-slide pathology section. When the program for the method of registering mass spectrometry imaging data with a whole-slide pathology section is executed by at least one processor, the steps of a method of registering mass spectrometry imaging data with a whole-slide pathology section disclosed above are implemented.

[0043] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0044] (1) By performing deep learning interpolation on the mass spectrometry imaging image and registering the interpolated mass spectrometry imaging image with the whole-slide pathology section, the present invention has small fluctuations in the quality of the registered image;

[0045] (2) By correcting the mass spectrometry imaging image and the whole-slide pathology section, the present invention can effectively reduce the relative position deviation occurring during the experiment, including: reducing the morphological structure shift occurring in adjacent sections during cross-section production, or the displacement occurring due to experimental reasons during the ionization process;

[0046] (3) By image correction and affine transformation, the present invention realizes the preliminary registration of the mass spectrometry imaging image and the whole-slide pathology section, and then uses the moving least squares method to align the edges of the mass spectrometry imaging image and the whole-slide pathology section, deforming the edge of the mass spectrometry imaging image towards the whole-slide pathology section to completely align the mass spectrometry imaging image with the whole-slide pathology section, minimizing the deviation generated at the edge between the two types of images, and having the advantages of reducing the edge overlap area, improving the image alignment accuracy, and having clear edges. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a schematic flowchart of a method for registering mass spectrometry imaging data with a whole-slide pathology section;

[0048] Figure 2 It is the MSI pre - processing process of a registration method for mass spectrometry imaging data and whole - slide pathology sections;

[0049] Figure 3 It is the WSI pre - processing process of a registration method for mass spectrometry imaging data and whole - slide pathology sections;

[0050] Figure 4 It is a schematic diagram for comparing the image registration results before and after image correction;

[0051] Figure 5 It is a schematic diagram for comparing the results of different registration methods;

[0052] Figure 6 It is a comparison graph of the registration results of using affine transformation and moving least - squares method under different mass spectrometry imaging images and WSI images. Detailed implementation manners

[0053] The present invention has established a workflow capable of integrating and synchronously analyzing mass spectrometry imaging data (WSI) and whole - slide pathology sections (MSI), and shown good results. The interpolation method based on deep learning is first introduced into the research related to whole - slide pathology sections (MSI), providing a reliable magnification factor for whole - slide pathology section (MSI) data. The image registration method of the present invention can present morphological features and chemical distributions in one image, and can more directly observe the morphological features of WSI and the molecular distribution features of MSI.

[0054] Now, the technical solution of the present invention will be further elaborated in combination with the accompanying drawings and embodiments.

[0055] Embodiment 1:

[0056] This embodiment discloses a registration method for mass spectrometry imaging data and whole - slide pathology sections, including the following steps:

[0057] Obtain sections of the tissue sample to be measured, and make corresponding whole - slide pathology sections and matrix - assisted laser desorption / ionization mass spectrometry imaging images; the matrix - assisted laser desorption / ionization mass spectrometry imaging images are obtained by performing matrix - assisted laser desorption / ionization on adjacent sections; the adjacent sections are the sections adjacent to the whole - slide pathology sections;

[0058] Use the rotating calipers algorithm to obtain the minimum bounding rectangle of the whole - slide pathology section and the matrix - assisted laser desorption / ionization mass spectrometry imaging image; based on the minimum bounding rectangle, correct the positions of the whole - slide pathology section and the matrix - assisted laser desorption / ionization mass spectrometry imaging image to obtain the corrected whole - slide pathology section and the corrected matrix - assisted laser desorption / ionization mass spectrometry imaging image;

[0059] Using the Canny edge operator, extract the edge contours of the corrected whole-slide pathology section and the edge contours of the corrected matrix-assisted laser desorption / ionization mass spectrometry imaging (MALDI-MSI) image, obtaining the edge contours of the corrected whole-slide pathology section and the edge contours of the corrected MALDI-MSI image;

[0060] Input the corrected MALDI-MSI image into a super-resolution deep learning neural network to output a mass spectrometry imaging image with the same resolution as the corrected whole-slide pathology section;

[0061] Use affine transformation to perform primary registration on the corrected whole-slide pathology section and the output mass spectrometry image to obtain a primarily registered image;

[0062] Use the moving least squares method to perform secondary registration on the primarily registered image to obtain the final registered image.

[0063] Example 2:

[0064] Refer to Figure 1 , this example proposes a registration method for mass spectrometry imaging data and whole-slide pathology sections based on machine learning, including the following steps:

[0065] Step 1: Prepare a mass spectrometry imaging image set and a whole-slide pathology section image set; in this example, the mass spectrometry imaging image set and the whole-slide pathology section image set are made through the following sub-steps:

[0066] S110: Perform cryosectioning on the tissue sample to be tested to obtain multiple histological sections;

[0067] S120: Select the sections that can show histological features from the multiple histological sections, and stain the sections that can show histological features to obtain stained sections;

[0068] S130: Use a digital slide scanning system to scan the stained sections to obtain high-resolution whole-slide pathology section images. Whole-slide pathology sections are usually stored in a multi-resolution pyramid structure, with downsampled versions of multiple original images. Different layers of the pyramid correspond to different resolutions, and the number of frames corresponding to different resolutions is also different; the bottom layer of the pyramid model corresponds to the highest-resolution image, called level 0. The middle layers of the pyramid model are thumbnails of the bottom-layer image, that is, the other resolution images are called level 1 to level n from bottom to top, respectively, thus constructing a whole-slide pathology section image set; S140: Select the histological sections adjacent to the sections that can show histological features, denoted as adjacent sections, and perform matrix-assisted laser desorption / ionization (MALDI-MSI) on the adjacent sections to obtain multiple mass spectrometry imaging images;

[0069] S150: Convert and process the mass spectrometry imaging image through a format and save it as an imzML file, thereby constructing a mass spectrometry imaging image set for subsequent image registration;

[0070] Step 2: The preparation step of Step 1 may cause an offset in the slice scanning angle. Therefore, the whole-slide pathology slice image and the mass spectrometry imaging image need to be preprocessed through image correction to obtain the corrected whole-slide pathology slice image and mass spectrometry imaging image.

[0071] In this embodiment, the preprocessing includes but is not limited to the following sub-steps: First, gray-scale the image and remove noise using Gaussian blur; then use erosion and dilation to remove the remaining noise in the image to obtain a binary image; finally, use the rotating calipers algorithm to calculate the minimum bounding rectangle of the binary image, calculate the image correction angle based on the four vertex coordinates of the minimum bounding rectangle, and rotate the image according to the image correction angle. Finally, use the Canny edge operator to extract the contour of the rotated image to obtain the preprocessed image.

[0072] The images mentioned in this step refer to the whole-slide pathology slice image and the mass spectrometry imaging image, and both types of images need to undergo the above preprocessing.

[0073] Now, the calculation process of using the rotating calipers algorithm to calculate the minimum bounding rectangle of the binary image and calculating the image correction angle based on the four vertex coordinates of the minimum bounding rectangle is described as follows:

[0074] Each binary image is a convex polygon structure, and the following operations are performed on each binary image.

[0075] S210: Take any side of the polygon to make a line l1 that coincides with it, select a point on the polygon that is farthest from l1 to make a parallel line l2; among the two parallel lines, select a set of two points with the largest distance, and respectively make a parallel tangent perpendicular to l1. The rectangle formed by the four lines is the bounding rectangle, and the area of this rectangle is saved as the current minimum value;

[0076] S220: Rotate the bounding rectangle clockwise until one side of the rectangle coincides with one side of the binary image. This side is denoted as the parallel line l3. Select a point on the polygon that is farthest from the parallel line l3 to make a parallel line l4; among the two parallel lines, select a set of two points with the largest distance, and respectively make a parallel tangent perpendicular to l3. Calculate the area of the new rectangle formed by the four lines, compare the area of the new rectangle with the current minimum value. If it is less than the current minimum value, then save the area of the new rectangle as the current minimum value to update the current minimum value, and simultaneously save the rectangle information corresponding to the current minimum value; if it is not less than, then do not update;

[0077] S230: Repeat S220 until the clockwise rotation angle of the line is greater than 90 degrees, then execute S240;

[0078] S240: Obtain the minimum circumscribed matrix [w, h] of the binary image, where w represents the matrix width and h represents the matrix height; the four vertex coordinates of the minimum circumscribed matrix [w, h] of the binary image are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), and the corrected four vertex coordinates are (x1, y1), (x1 + w, y1), (x1, y1 + h), and (x1 + w, y1 + h); calculate the rotation angle θ required for the minimum circumscribed matrix through w = (x2 - x1)cosθ - (y2 - y1)sinθ.

[0079] As Figure 2 and Figure 3 shown, preprocess the mass spectrometry imaging image and the whole-slide pathology section image respectively to obtain an image with only the required section information in the field of view, reducing the influence of other noises on the registration result;

[0080] Step 3: Select the preprocessed whole-slide pathology section with an appropriate resolution. Specifically: The preprocessed whole-slide pathology section is stored in a multi-resolution pyramid structure. In this embodiment, the image of the fourth layer is selected, and its resolution is 7936 * 4768;

[0081] Step 4: Due to the reason of the MADLI device, the resolution of the obtained mass spectrometry imaging image is (90, 60); therefore, before image registration, first interpolate the mass spectrometry imaging image; however, the effect of conventional linear interpolation is not good, so in this embodiment, a super-resolution deep learning neural network (VDSR) is used to interpolate the mass spectrometry imaging image to obtain the interpolated mass spectrometry imaging image.

[0082] The super-resolution deep learning neural network adopted in this embodiment includes:

[0083] A bicubic interpolation sub-network, which is used to perform bicubic interpolation on the input corrected matrix-assisted laser desorption / ionization mass spectrometry imaging image to obtain the interpolated low-resolution image ILR (Low resolution image after interpolation);

[0084] A feature extraction sub-network, which is used to extract multiple image patches (patches) from the interpolated low-resolution image ILR, and represent each image patch (patch) as a multi-dimensional vector through convolution operations, and all vector features form a feature matrix;

[0085] A non-linear mapping sub-network, which is used to perform non-linear mapping on the feature matrix through convolution operations to obtain a new feature matrix;

[0086] The upsampling sub-network is used to perform upsampling operations on the new feature matrix to obtain a mass spectrometry imaging image with a resolution close to that of the whole-field pathological section selected in step 3.

[0087] Specifically, the super-resolution deep learning neural network adopted in this embodiment includes 19 convolutional layers, 19 Relu activation function layers, and a residual network layer; the size of each convolutional layer is 3×3, the number of channels of each convolutional kernel is 64, and 0 is used as the edge padding in each convolutional layer to keep the resolution unchanged before and after convolution; through 19 Relu activation function layers, more complex non-linear mappings can be generated, and more complex models can be fitted; the initial learning rate is 0.1, and it decays by 10 times after every 20 rounds of training, so there are a total of four learning rates after 80 rounds of training; the residual network layer includes a 3x3 convolutional kernel with 1 channel.

[0088] Before interpolating the mass spectrometry imaging image using the super-resolution deep learning neural network, the deep learning neural network needs to be trained with the prepared low-resolution image and the corresponding high-resolution image. When the mean square error between the high-resolution image output by the super-resolution deep learning neural network and the original high-resolution image is less than a pre-set threshold, the iteration is stopped and the training is completed.

[0089] Input the corrected mass spectrometry imaging image into the trained super-resolution deep learning neural network to output an image with a resolution close to that of the corrected WSI image; for example: perform 80-fold interpolation on the corrected mass spectrometry imaging image, and the resolution of the corresponding output image is (7200, 4800).

[0090] Step 5: Perform an affine transformation on the interpolated mass spectrometry imaging image to obtain the mass spectrometry imaging image after the affine transformation;

[0091] Denote the four vertex coordinates of the interpolated mass spectrometry imaging image as (x M1 , y M1 ), (x M2 , y M2 ), (x M3 , y M3 ), (x M4 , y M4 ); denote the four vertex coordinates of the WSI image as (x W1 , y W1 ), (x W2 , y W2 ), (x W3 , y W3 ), (x W4 , y W4 );

[0092] Through the affine transformation equation xWn = a1x Mn + b1y Mn + c1 and y Wn = a2x Mn + b2y Mn + c2, where a1, a2, b1, b2, c1, c2 respectively represent the parameters during different geometric transformations of the image, such as rotation, translation, scaling, etc. Transform the four vertex coordinates of the interpolated mass spectrometry imaging image to (x W1 , y W1 ), (x W2 , y W2 ), (x W3 , y W3 ), (x W4 , y W4 ), that is, correspond the four vertex coordinates of the mass spectrometry imaging image with the four vertex coordinates of the WSI image.

[0093] According to the edge contour of the corrected whole-slide pathology section and the edge contour of the mass spectrometry imaging image after affine transformation, perform the initial registration on the corrected whole-slide pathology section and the mass spectrometry imaging image after affine transformation to obtain the initially registered image.

[0094] Step 6: The section corresponding to the mass spectrometry imaging image is an adjacent section to the section corresponding to the WSI image. After affine transformation, due to section preparation, there will be a certain error in the edge contour of the initially registered image, as Figure 4 shown. a is the registration result of the mass spectrometry imaging image and the WSI image without image correction, and b and c are the registration results of the mass spectrometry imaging image and the WSI image with prior image correction. It can be seen that there is an obvious image displacement inside the registration result of a, and the registration result is not accurate enough.

[0095] In this embodiment, the moving least squares method is used to reduce or eliminate the error of the edge contours of these two types of images, so that the registration between them is accurate. That is, use the moving least squares method to perform secondary registration on the initially registered image to obtain the final registered image.

[0096] Use the moving least squares method to construct a corresponding deformation function f v (v) for each pixel point v on the original image, so that the edge contour of the mass spectrometry imaging image after affine transformation deforms towards the edge contour of the corrected whole-slide pathology section, and calculate the position of the image after deformation through the following formula:

[0097]

[0098] Where:

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] In the formula, the edge contour information of mass spectrometry imaging is used as the deformation control vertex p, the edge contour information of the whole-field pathological section is used as the target control vertex q, and the coordinate points of other pixels in the mass spectrometry imaging image are v and w. i represents the weight, that is, the reciprocal of the distance from v to the control point p, and α is a parameter for adjusting the deformation effect, usually 1. i and j represent all control points in the horizontal and vertical coordinate directions of the image.

[0105] Based on the positions of the pixels after deformation, the initially registered image is subjected to secondary registration.

[0106] Such as Figure 5 shown, from top to bottom are the detail map in the WSI image, the detail map of the registration result obtained only by affine transformation (AT), and the detail map of the registration result obtained by affine transformation and moving least squares (MLS) (AT+MLS); Figure 6 is the registration result of using affine transformation and moving least squares under different mass spectrometry imaging images and WSI images. As can be seen from Figure 5 and Figure 6 It can be seen that the method for registering mass spectrometry imaging images and WSI images proposed in this embodiment by using deep learning interpolation and combining affine transformation with moving least squares has good results.

[0107] Embodiment 3:

[0108] This embodiment proposes a registration system (i.e., a computer device) for mass spectrometry imaging data and whole-field pathological sections. The system includes a network interface, a memory, and a processor; wherein, the network interface is used for receiving and sending signals during the process of receiving and sending information with other external network elements; the memory is used for storing computer program instructions that can run on the processor; the processor is used for executing the steps of a method for registering mass spectrometry imaging data and whole-field pathological sections disclosed above when running the computer program instructions.

[0109] The registration system (i.e., computer device) for mass spectrometry imaging data and whole-slide pathology sections includes a processor, a memory, and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal via a network connection.

[0110] Embodiment 4:

[0111] This embodiment provides a computer storage medium that stores a program for a method of registering mass spectrometry imaging data and whole-slide pathology sections. When the program for the method of registering mass spectrometry imaging data and whole-slide pathology sections is executed by at least one processor, the steps of a method of registering mass spectrometry imaging data and whole-slide pathology sections disclosed above are implemented.

[0112] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0113] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0114] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A registration method for mass spectrometry imaging data and whole-slide pathology sections, characterized in that: Including the following steps: Obtain a section of the histological sample to be measured, and produce corresponding whole-slide pathology sections and matrix-assisted laser desorption / ionization mass spectrometry imaging (MALDI-MSI) images; The MALDI-MSI image is obtained by performing matrix-assisted laser desorption / ionization on adjacent sections; the adjacent sections are sections adjacent to the whole-slide pathology section; Using the rotating calipers algorithm, obtain the minimum bounding rectangle of the whole-slide pathology section and the MALDI-MSI image; based on the minimum bounding rectangle, perform position correction on the whole-slide pathology section and the MALDI-MSI image to obtain a corrected whole-slide pathology section and a corrected MALDI-MSI image; Using the Canny edge operator, extract the edge contours of the corrected whole-slide pathology section and the edge contours of the corrected MALDI-MSI image to obtain the edge contours of the corrected whole-slide pathology section and the edge contours of the corrected MALDI-MSI image; Input the corrected MALDI-MSI image into a super-resolution deep learning neural network, and output a mass spectrometry imaging image with the same resolution as the corrected whole-slide pathology section; Using affine transformation, perform primary registration on the corrected whole-slide pathology section and the output mass spectrometry image to obtain a primarily registered image; Using the moving least squares method, perform secondary registration on the primarily registered image to obtain a final registered image.

2. The registration method of mass spectrometry imaging data and whole slide pathology sections according to claim 1, wherein: The obtaining of a section of the histological sample to be measured and producing corresponding whole-slide pathology sections and MALDI-MSI images specifically includes: Perform cryosectioning on the histological sample to be measured to obtain multiple histological sections; Select a section including histological features from the multiple histological sections, and stain the section including histological features to obtain a stained section; Scan the stained section through a digital slide scanning system to obtain a whole-slide pathology section image; Select a histological section adjacent to the section including histological features, denoted as the adjacent section, and perform matrix-assisted laser desorption / ionization on the adjacent section to obtain a MALDI-MSI image.

3. A method for registering mass spectrometry imaging data with whole slide pathology sections according to claim 1, characterized in that: The performing of position correction on the whole-slide pathology section and the MALDI-MSI image based on the minimum bounding rectangle to obtain a corrected whole-slide pathology section and a corrected MALDI-MSI image specifically includes: Obtain corresponding position correction angles according to the four vertices of the minimum bounding matrix; Rotate the whole-slide pathology section and the MALDI-MSI image according to the corresponding position correction angles to obtain a corrected whole-slide pathology section and a corrected MALDI-MSI image.

4. A method for registering mass spectrometry imaging data with a whole-slide pathology section according to claim 1, characterized in that: The super-resolution deep learning neural network includes: A bicubic interpolation sub-network for performing bicubic interpolation on the input corrected MALDI-MSI image to obtain an interpolated low-resolution image ILR; A feature extraction sub-network, which is used to extract multiple image patches from the interpolated low-resolution image \(I_{LR}\), perform convolution operations on each image patch to obtain feature vectors, and all the feature vectors form a feature matrix; A non-linear mapping sub-network, which is used to perform non-linear mapping on the feature matrix through convolution operations to obtain a new feature matrix; An upsampling sub-network, which is used to perform upsampling operations on the new feature matrix to obtain a mass spectrometry imaging image with the same resolution as the corrected whole-field pathological section.

5. A method for registering mass spectrometry imaging data with a whole-slide pathology section according to claim 1, characterized in that: Using the affine transformation to perform primary registration on the corrected whole-field pathological section and the output mass spectrometry image to obtain a primarily registered image, specifically including: Record the four vertex coordinates of the output mass spectrometry imaging image as (x M1 , y M1 ), (x M2 , y M2 ), (x M3 , y M3 ), (x M4 , y M4 ); record the four vertex coordinates of the corrected whole - field pathological section as (x W1 , y W1 ), (x W2 , y W2 ), (x W3 , y W3 ), (x W4 , y W4 ); Through the affine transformation equations x Wn = a1x Mn + b1y Mn + c1 and y Wn = a2x Mn + b2y Mn + c2, the four vertex coordinates of the output mass spectrometry imaging image are transformed to (x W1 , y W1 ), (x W2 , y W2 ), (x W3 , y W3 ), (x W4 , y W4 ); where a1, a2, b1, b2, c1, and c2 respectively represent the parameters for different geometric transformations.

6. The registration method of mass spectrometry imaging data and whole-slide pathology sections according to claim 1, characterized in that: Using the moving least squares method to perform secondary registration on the primarily registered image, specifically including: Construct a corresponding deformation function \(f\) for each pixel point \(v\) on the initially registered image using the moving least squares method v (v) to deform the edge contour of the mass spectrometry imaging image after affine transformation towards the edge contour of the corrected whole-field pathological section. The positions of the deformed pixel points are calculated through the deformation function \(f\) v (v): where: where p i represents the i-th deformation control point in the horizontal axis direction, and p j represents the j-th deformation control point in the vertical axis direction. The deformation control points correspond to the edge contour information of the mass spectrometry imaging image; q i represents the i-th target control point in the horizontal axis direction, and q j represents the j-th target control point in the vertical axis direction. The target control point corresponds to the edge contour information of the whole-slide pathology section; w i represents the weight; α represents the parameter for adjusting the deformation effect; Based on the positions of the pixels after deformation, perform secondary registration on the primarily registered image.

7. A system for implementing a method for registering mass spectrometry imaging data with a whole-slide pathology section, characterized in that, The system includes a network interface, a memory, and a processor; wherein, The network interface is used for receiving and sending signals during the process of transceiver information with other external network elements; The memory is used for storing computer program instructions that can run on the processor; The processor is used for executing the steps of the registration method of the mass spectrometry imaging data and the whole-field pathological section according to any one of claims 1 to 6 when running the computer program instructions.

8. A computer storage medium, characterized in that, The computer storage medium stores a program for implementing the registration method of the mass spectrometry imaging data and the whole-field pathological section. When the program for implementing the registration method of the mass spectrometry imaging data and the whole-field pathological section is executed by at least one processor, the steps of a registration method of the mass spectrometry imaging data and the whole-field pathological section according to any one of claims 1 to 6 are implemented.

Images