Spatial molecular phenotype super-resolution prediction method and device based on pathological whole section image and storage medium

Abstract

The application discloses a kind of based on pathological whole section image's spatial molecular phenotype super-resolution prediction method, device and storage medium, the method includes: obtaining the pathological whole section image of sample to be analyzed;Pathological whole section image is detected and is cut to block processing to the organization area, generate multiple image blocks containing spatial coordinates;Using trained prediction network, multiple image blocks are handled, obtain pixel level or spot level gene expression / pathway activity dense prediction feature map;According to its spatial position, dense prediction feature map is filled back to whole section coordinate system and is spliced, obtains whole section level super-resolution spatial molecular atlas.The present application can be widely used in tumor microenvironment spatial phenotype analysis, virtual molecular staining, risk stratification and prognosis evaluation etc.Scenarios, with no additional experiment, low cost, high resolution, deployability is strong and the like advantages.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and biomedical informatics, and in particular to a method, apparatus and storage medium for super-resolution prediction of spatial molecular phenotypes based on pathological whole-section images. Background Technology

[0002] Cutting-edge space omics technologies such as spatial transcriptomics and spatial proteomics can measure the molecular expression profiles of genes or proteins in a precise spatial coordinate system while preserving tissue structure, providing revolutionary tools for understanding the spatial heterogeneity of life processes. However, these technologies typically face significant application barriers: they are extremely expensive, involving specialized instruments, complex reagents, and deep sequencing or large-scale imaging; throughput is limited, with a small number of samples processed per experiment and lengthy procedures; and spatial resolution is fundamentally limited—for example, mainstream sequencing-based spatial transcriptomics technologies often contain data points from multiple cells, making it difficult to resolve molecular differences at the single-cell level.

[0003] On the other hand, hematoxylin-eosin stained pathological slides, widely generated in clinical diagnosis and research, are the gold standard for pathological diagnosis. They are routinely and cost-effectively available in most hospitals worldwide, resulting in a vast and valuable sample library closely linked to patient prognostic information. However, while these slides contain extremely rich information on tissue morphology, cell structure, and disease phenotypes, they are inherently "molecularly blind"—key molecular information such as the specific gene expression status, protein activity, or functional subtypes of cells in the slides cannot be directly read. This information must be indirectly obtained through additional, often destructive, molecular biology experiments (such as laser microdissection followed by sequencing and immunohistochemistry), which greatly limits the possibility of systematically uncovering deep biological mechanisms from a vast amount of clinical resources.

[0004] This contradiction—the low accessibility of high-information-depth spatial omics technologies and the insufficient information depth of routine clinical slides with extremely high accessibility—constitutes a core bottleneck in current translational medicine research. Therefore, there is an urgent need for a computational method that can automatically infer spatial molecular phenotypes and support super-resolution output based solely on routine HE whole-slide images, in order to reduce costs and improve clinical usability. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art by providing a method, device, and storage medium for super-resolution prediction of spatial molecular phenotypes based on pathological whole-slice images. With the optional introduction of spatial gene expression constraints, it can achieve pixel-level dense prediction of molecular signals such as gene expression or pathway activity, and achieve seamless reconstruction at the whole-slice level through weighted fusion, thereby providing a foundation for virtual molecular staining, spatial subregion analysis, and clinical risk assessment.

[0006] The objective of this invention can be achieved through the following technical solutions: A method for super-resolution prediction of spatial molecular phenotypes based on whole-section pathological images includes the following steps: Obtain pathological whole-section images of the sample to be analyzed; The pathological whole-section image is subjected to tissue region detection and segmentation processing to generate multiple image blocks containing spatial coordinates; The trained prediction network is used to process the multiple image patches to obtain dense prediction feature maps; The dense predicted feature maps are spliced ​​and backfilled into the full slice coordinate system according to their spatial positions to obtain a full slice-level super-resolution spatial molecular map. The prediction network includes a pathological baseline model, a Transformer backbone network, and a decoder. The pathological baseline model encodes each image patch to obtain an image token sequence, which is then fused with preset start and end tokens to form a fused token sequence. The Transformer backbone network performs multi-layer feature learning on the fused token sequence to extract a multi-scale representation set. The decoder reassembles the multi-scale representation set into a spatial feature map and fuses them step by step to output the dense prediction feature map.

[0007] Furthermore, the slicing process specifically involves sliding window slicing that allows adjacent image blocks to overlap, and recording the coordinates and overlap relationship of each image block in the full slice; When the dense predicted feature map is spliced ​​back to the full slice coordinate system according to its spatial location, a weighted fusion strategy with linear edge decay is adopted for the overlapping area.

[0008] Furthermore, the weighted fusion strategy with linear edge decay specifically includes: Construct a weight matrix for each image patch, such that the weight of the central region of the image patch is 1, and the weight of the region near the edge decreases by a linear function; The predicted values ​​of each image block are weighted and summed according to the weight matrix, and then normalized using the weights to obtain the final stitching result. The weights in the weight matrix are determined by the overlap width and the image block size, and the weight of pixel (i,j) is the minimum value of the attenuation weights in the horizontal and vertical directions.

[0009] Furthermore, the step of performing multi-layer feature learning on the fused token sequence and extracting a multi-scale representation set specifically involves: Multiple sets of tokens are extracted from the multi-layered Transformer backbone network at equal intervals or with a preset layer index to form a multi-scale representation set.

[0010] Furthermore, the reorganization into a spatial feature map specifically involves: The multi-scale representation set is rearranged into a two-dimensional feature map in spatial order, and upsampling or downsampling is performed through convolution or transposed convolution to obtain a spatial feature map with a preset resolution.

[0011] Furthermore, during the training of the prediction network, the parameters of the basic pathological model are frozen, and only the parameters of the Transformer backbone network and decoder are updated; The prediction network is trained using a composite loss function, which simultaneously constrains the consistency between the average value of the predicted output within the blot region and the gene expression in the target space, as well as the consistency between the staining texture reconstructed from the output features through convolution and the texture of the original pathological whole-slice image.

[0012] Furthermore, the method also includes: The spatial gene expression or its derived pathway scores, which are spatially registered with the pathological whole slice image, are obtained, normalized / standardized, and then gene tokens are generated through Fourier feature embedding. The gene token and image token sequences are concatenated with preset start and end tokens to form the fusion token sequence.

[0013] Furthermore, the method also includes: The region of interest (ROI) is marked in the whole pathological slide image. When performing the slicing process, slicing is only performed on the marked ROI.

[0014] The present invention also provides a spatial molecular phenotypic super-resolution prediction device based on pathological whole-section images, comprising: The image acquisition module is used to acquire pathological whole-section images of the sample to be analyzed; The slicing and coordinate management module is used to perform tissue region detection and slicing processing on the pathological whole slice image, generate multiple image blocks containing spatial coordinates, and record the coordinates and overlap relationship of each image block in the whole slice; The pathological basic model encoding module is used to encode each image block to obtain an image token sequence; The molecular input encoding module responds when it receives a spatial gene expression or its derived pathway score that is spatially registered with the pathological whole slice image. It is used to normalize / standardize the spatial gene expression or its derived pathway score, and then generate a gene token through Fourier feature embedding. The Transformer fusion module is used to fuse the image token sequence with a preset start and end token to form a fusion token sequence, or to concatenate the gene token and image token sequences with preset start and end tokens respectively to form a fusion token sequence, and to perform multi-layer feature learning on the fusion token sequence to extract a multi-scale representation set. The decoding and output module is used to reorganize the multi-scale representation set into a spatial feature map, and fuse them step by step to output the dense prediction feature map; The full-slice weighted stitching module is used to stitch the dense predicted feature map back to the full-slice coordinate system according to its spatial position to obtain a full-slice-level super-resolution spatial molecular map. The storage module is used to store model parameters and intermediate results.

[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention requires no additional molecular experiments. It can predict multi-level spatial molecular phenotypes such as gene expression, pathway activity, and spatial functional prototypes using only HE-stained pathological whole slide images that are routinely available in clinical settings. This significantly reduces experimental costs, simplifies the operation process, and makes it easy to promote and apply in clinical settings.

[0017] 2. This invention allows for flexible selection of whether to introduce spatial gene expression or its derived pathway scores as constraint information. Through a multimodal fusion strategy, it effectively improves the accuracy of molecular phenotype prediction, suppresses false positive prediction results, and enhances the reliability of prediction results.

[0018] 3. This invention supports pixel-level super-resolution output. Experiments show that it can generate spatial molecular maps with a resolution of up to 0.25μm. At the same time, through the edge linear attenuation weighted fusion strategy, it achieves seamless stitching of overlapping image blocks, avoids block artifacts during the stitching process, and fully preserves tissue morphology details and molecular distribution characteristics.

[0019] 4. The output results of this invention are compatible with existing spatial omics analysis workflows and can be directly used for downstream research such as cell clustering, molecular difference analysis, and spatial neighborhood relationship analysis without additional data format conversion, thereby improving analysis efficiency and expanding the application scenarios of the technology.

[0020] 5. This invention supports local inference based on regions of interest annotated by pathologists, which can specifically focus on key tissue areas, reduce invalid calculations, and improve inference efficiency; at the same time, it supports local deployment, effectively protects the privacy data of clinical samples, and meets the dual needs of privacy protection and computational efficiency in clinical applications.

[0021] 6. The prediction network of this invention adopts a composite loss function to achieve dual supervision during the training phase, taking into account both molecular prediction consistency and tissue texture consistency, further improving prediction accuracy, and ensuring that the prediction results not only conform to molecular expression rules but also match tissue morphological characteristics, providing more accurate technical support for clinical diagnosis, risk stratification, and prognostic assessment. Attached Figure Description

[0022] Figure 1 This is a flowchart of the present invention; Figure 2 This is a block diagram of the spatial molecular phenotype super-resolution prediction device of the present invention; Figure 3 This is a schematic diagram of the edge linear decay weighted fusion and normalization of overlapping patches in this invention; Figure 4 This is a schematic diagram illustrating the dual supervision and composite loss function calculation process of the prediction network during the training phase of this invention. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0024] Example 1 This embodiment discloses a virtual cell spatial molecular phenotypic super-resolution prediction method based on pathological whole-section images. The method acquires a pathological whole-section image of the sample to be analyzed; performs tissue region detection and segmentation on the pathological whole-section image to generate multiple image patches containing spatial coordinates; a trained prediction network processes the multiple image patches to obtain pixel-level or spot-level dense prediction feature maps of gene expression / pathway activity; the dense prediction feature maps are then stitched back to the whole-section coordinate system according to their spatial positions to obtain a whole-section-level super-resolution spatial molecular map. Specifically, the prediction network includes a pathological basic model, a Transformer backbone network, and a decoder. The pathological basic model encodes each image patch to obtain an image token sequence, which is then fused with preset start and end tokens to form a fused token sequence. The Transformer backbone network performs multi-layer feature learning on the fused token sequence to extract a multi-scale representation set. The decoder reassembles the multi-scale representation set into a spatial feature map and fuses them step-by-step to output the dense prediction feature map.

[0025] like Figure 1 As shown, the specific steps of this method are as follows: Step 1: Obtain whole-slice images (WSI) of the sample to be analyzed. Specifically, in this embodiment, images of whole-slice pathological sections stained with hematoxylin and eosin (HE) are acquired.

[0026] In an optional implementation, spatial gene expression (SGE) data spatially registered with the pathological whole slide image or pathway scores derived therefrom can be obtained. The SGEs can come from spatial omics platforms such as Visium, Visium-HD, Xenium, or other equivalent platforms.

[0027] During the training phase of the prediction network, training data pairs containing WSI and its registered SGE need to be collected. SGE can be z-score normalized, or optionally ssGSEA can be used to generate pathway scores as the training objective.

[0028] Step 2: Perform tissue region detection and segmentation on the acquired pathological whole-section images to generate multiple image patches with spatial coordinates.

[0029] During the training phase of the prediction network, background removal and tissue region detection are performed on WSI. Sliding window segmentation is performed based on the preset patch size p and stride s, allowing adjacent patches to overlap. The top-left corner coordinates (x0, y0) and the level / magnification information of each patch in the WSI coordinate system are recorded.

[0030] In an optional implementation, if a region of interest (ROI) is marked by a pathologist, the ROI marking of the whole pathological slide image is obtained, and when performing the slicing process, only the area covered by the marked ROI is sliced.

[0031] Step 3: Input each image patch into the Pathological Basic Model (PFM) to extract multiple image tokens; where PFM is a pre-trained network based on Visual Transformer, the output token dimension is D, and the number of tokens is consistent with the patch partitioning rule; during the training phase, this step is carried out synchronously with the subsequent token construction, providing a foundation for the construction of fusion sequences.

[0032] Step 4: When spatial gene expression data or pathway scores are available, normalize / standardize them, encode them into gene tokens using Fourier feature embedding, and concatenate the image tokens with the tokens according to preset start and end tokens to form a fusion token sequence; when spatial gene expression data is unavailable, only the image tokens and preset start and end tokens are used to form a fusion token sequence; during the training phase, the molecular input (spatial gene expression data or pathway scores) needs to be projected into gene tokens through Fourier feature embedding, and fixed start and end tokens are added before and after the image tokens to form a fusion token sequence.

[0033] Specifically, the Fourier feature embedding step is as follows: the normalized / standardized molecular input g (scalar or vector) is mapped to a high-dimensional spectral feature γ(g) = [g, sin(2kπg), cos(2kπg)], k = 0…Nf-1, where Nf is a preset number of frequency bands (a positive integer, usually set to 128), used to control the frequency resolution and feature dimension of the Fourier embedding; then, layer normalization is performed on γ(g) and linearly projected to the hidden dimension D, thereby obtaining a gene token with the same dimension as the image token. This encoding can be used for gene expression, pathway scoring, or other molecular features.

[0034] Step 5: Input the fused token sequence into the Transformer backbone network for multi-layer representation learning, and extract the token set from the preset multi-layer network; during the training phase, the fused token sequence needs to be input into the Transformer, and the token set needs to be extracted from four equally spaced layers for subsequent decoding and fusion.

[0035] Step 6: The decoder performs recombination and multi-scale fusion on the extracted multi-layer token set to obtain a dense prediction feature map and output the prediction results of spatial gene expression or pathway activity. During the training phase, the recombination operation performed by the decoder on the tokens includes splicing and projection of classification token CLS and image token, spatial rearrangement, convolution / transposed convolution up / downsampling, and multi-scale stepwise fusion to output a dense prediction feature map. During the inference phase, you can choose between speckle-level output (aligned with the patch size) or pixel-level super-resolution output (up to 0.25μm resolution).

[0036] Specifically, the token fusion and multi-scale decoding process of the Transformer backbone network and decoder mentioned above includes: (1) Token fusion: Preset start and end tokens are added to both ends of the image token and the optional gene token sequence, and a classification token CLS is added so that the model can learn global semantics.

[0037] (2) Multi-layer token extraction: Extract multiple sets of tokens at equal intervals or with preset layer indices in the multi-layer of the Transformer backbone network to form a multi-scale representation set.

[0038] (3) Reassemble: The tokens are rearranged into two-dimensional feature maps in spatial order; convolution / transposed convolution is used to upsample or downsample the feature maps of different scales; and dense feature maps of the same size as the input patch are obtained through stepwise fusion (e.g., convolution after splicing, summation or attention fusion), and finally the target molecule prediction map is obtained through the output head.

[0039] Specifically, during the training of the aforementioned prediction network, the parameters of the basic pathology model are frozen, and only the parameters of the Transformer backbone network and decoder are updated. Furthermore, it supports replacing visual Transformer backbones or visual-language pathology models of different sizes to adapt to different deployment scenarios. During training, a composite loss function is used for constraints. This loss function simultaneously supervises the consistency between predicted values ​​within the blotchy region and gene expression in the target space, as well as the consistency between the hematoxylin and eosin texture reconstructed from the output features and the original image. The training objective is to minimize this composite loss function, and training is performed using the Adam optimizer, learning rate warm-up and decay strategies, data augmentation, and early stopping strategies.

[0040] like Figure 4 As shown, the dual supervision and composite loss during the training phase are specifically as follows: (1) Molecular consistency loss: The pixel-level prediction map is averaged in each spatial spot / region to obtain the spot-level prediction value; the regression loss is calculated with the target SGE (or target pathway score).

[0041] (2) Image reconstruction loss: The dense predicted feature map is input into the convolutional reconstruction network to obtain the reconstructed HE image, and the mean square error loss is calculated with the input HE image.

[0042] (3) Molecular input drop rate: The drop rate is introduced to randomly empty the SGE / pathway input with a preset probability during training, so that the model has stable reasoning ability in both "image input only" and "image + molecular constraint input" modes, and is also suitable for reasoning scenarios without molecular constraints.

[0043] In summary, the training process of the prediction network in this embodiment may include: Step 201: Data Preparation. Collect training data pairs containing WSI and its registered SGE; z-score normalization can be performed on the SGE, or optionally, ssGSEA can be used to generate pathway scores as the training objective.

[0044] Step 202: Segmentation and Token Construction. The WSI is segmented into fixed-size patches and the coordinates are recorded; the patches are input into PFM to obtain image tokens; the molecular input is projected into gene tokens through Fourier feature embedding, and fixed start and end tokens are added before and after the image tokens to form a fusion sequence.

[0045] Step 203: Transformer and Multi-Layer Extraction. Input the fused sequence into the Transformer; extract a set of tokens from four equally spaced layers for subsequent decoding and fusion.

[0046] Step 204: Decoding and Dense Prediction. The decoder performs reconstruction on the token, including CLS and image token splicing projection, spatial rearrangement, convolution / transposed convolution up / downsampling, and multi-scale stepwise fusion to output a dense prediction feature map.

[0047] Step 205: Loss Function and Optimization. The training objective is to minimize the loss between the blob region and the target SGE, and the image reconstruction loss between the output feature map and the input HE image. The Adam optimizer, learning rate warm-up and decay strategies, data augmentation, and early stopping strategies are used for training.

[0048] Step 7: In the inference stage, the prediction results of each image patch are stitched together according to their spatial position in the whole pathological slice image. The overlapping areas are processed using an edge linear attenuation weighted fusion strategy to finally generate a whole slice-level super-resolution spatial molecular map.

[0049] like Figure 3 As shown, the specific splicing process is as follows: (1) Construct a weight matrix W for each patch. The weight of the central region is 1, and the weights of the edges and overlapping regions decay according to a linear function. Specifically, the weights are determined by the overlap width o and the patch size p. The weight of pixel (i,j) is the minimum value of the decay weights in the horizontal and vertical directions. That is, the weight of pixel (i,j) located in the patch is W(i,j)=min{w̃(i), w̃(j)}, where w̃(·) is a one-dimensional linear decay function. The weight of the non-overlapping region is 1. (2) Multiply the patch prediction by W, put it back into the full slice coordinates and sum them up to obtain the weighted cumulative image S(x,y): S += P k (x,y) ⊙ W k (x,y); Simultaneously accumulate the weighted graph M(x,y): M += Wk(x,y); (3) Obtain the normalized output I(x,y): I = S(x,y) / M(x,y), and get the seamless splicing result.

[0050] Specifically, super-resolution spatial molecular maps are used to characterize spatial molecular phenotypes. The output spatial molecular phenotypes include at least gene expression maps, pathway activity maps, and spatial functional archetype (SFA) maps derived from the pathway activity maps. The pathway activity maps are obtained by weighted summation of predicted gene expression from a preset gene set or by enrichment calculation based on a single sample gene set. The SFA maps are obtained by performing prototype decomposition / prototype analysis on multi-pathway spatial features.

[0051] This embodiment fully implements the method described in this invention, without relying on additional experiments, and can achieve high-precision, high-resolution spatial molecular phenotypic prediction while maintaining tissue morphology.

[0052] If the above methods are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0053] Example 2 This embodiment provides a virtual cell spatial molecular phenotype super-resolution prediction method based on pathological whole-section images, which further includes downstream phenotype derivation and application based on the dense prediction feature map obtained from the prediction. Specifically, it may include: (1) Pathway activity: Calculate pathway scores for the predicted gene expression map according to the preset gene set. Weighted summation, ssGSEA / GSVA and other single-sample enrichment algorithms or equivalent methods can be used.

[0054] (2) Spatial Functional Prototype (SFA): Input the vector field composed of multi-path spatial features into the prototype decomposition algorithm (e.g., prototype analysis, non-negative decomposition or cluster-prototype joint decomposition) to obtain several spatial functional prototypes and their spatial distribution map.

[0055] (3) Risk scoring and stratification: Calculate the SFA score for each sample based on the SFA chart and combine it with the preset Cox regression coefficient to obtain the risk score; divide the samples into high-risk / low-risk groups according to the threshold (such as the median) and conduct prognostic assessment.

[0056] (4) Single-cell aggregation: Optionally, pixel-level prediction results can be aggregated within the cell range using cell / nuclear segmentation masks to obtain molecular phenotypes at single-cell resolution for cell type annotation, spatial neighborhood analysis, etc. Cell segmentation can be performed on WSI using cell / nuclear segmentation (e.g., CellViT) to obtain masks, or the segmentation results provided by the spatial single-cell platform can be used directly; cell-level aggregation specifically involves averaging the pixel-level prediction values ​​within each cell mask to obtain cell-level gene expression or pathway scores, and the output can be used as single-cell spatial transcriptome data in the standard analysis process.

[0057] The rest is the same as in Example 1.

[0058] Example 3 This embodiment provides a virtual cell spatial molecular phenotypic super-resolution prediction device based on pathological whole-section images, such as... Figure 2 As shown, it includes the following modules: Image acquisition module: used to acquire pathological whole-section images of the sample to be analyzed, as well as optional spatial gene expression data or pathway score data; during the training phase, it is also used to collect training data pairs containing WSI and its registered SGE, and supports z-score normalization or ssGSEA processing of SGE to generate pathway scores.

[0059] The segmentation and coordinate management module is used to perform tissue region detection and segmentation processing on whole pathological slide images, generate image patch with spatial coordinates, and manage the position information of each image patch. During the training phase, it supports segmenting WSI into fixed-size patches and recording coordinates. During the inference phase, it supports reading GeoJSON / JSON format ROI annotations exported by QuPath and performing segmentation processing only on selected regions to reduce computational overhead.

[0060] The pathological basic model encoding module is used to input image blocks into the pathological basic model, extract image tokens, and provide an image-based foundation for subsequent fusion sequence construction.

[0061] Molecular input encoding module: used to normalize and encode spatial gene expression or pathway scores using Fourier feature embedding to generate gene tokens; during the training phase, it supports projecting molecular inputs into gene tokens via Fourier feature embedding, and constructing fusion sequences in conjunction with image representation units.

[0062] The Transformer fusion module is used to concatenate image tokens and gene tokens to form a fused token sequence, and completes multi-layer representation learning and multi-scale representation extraction through the Transformer backbone network. During the training phase, it supports inputting the fused sequence into the Transformer and extracting a set of tokens from four equally spaced layers, which are then transmitted to the decoding and output module.

[0063] The decoding and output module is used to recombine and fuse multi-layer tokens at multiple scales, outputting dense predicted feature maps and spatial gene expression or pathway activity prediction results. During the training phase, it supports recombination operations such as CLS and image token splicing projection, spatial rearrangement, convolution / transposed convolution up / downsampling, etc., to achieve multi-scale stepwise fusion and output dense predicted feature maps. During the inference phase, it supports two output modes: speckle-level and pixel-level (up to 0.25μm resolution). It also supports transmitting pixel-level prediction results to downstream cell-level aggregation processing. At the same time, this module integrates loss function calculation and optimization functions. During the training phase, it uses a composite loss function for constraint, combined with the Adam optimizer, learning rate warm-up and decay, data augmentation, and early stopping strategies to complete model training.

[0064] The full-slice weighted stitching module is used to stitch together the prediction results of each image patch according to its spatial location. It adopts an edge linear attenuation weighted fusion strategy for overlapping areas to generate a full-slice-level super-resolution spatial molecular map. Specifically, it constructs a weight matrix W for each patch, multiplies the patch prediction value by W, backfills it into the full-slice coordinates and accumulates them, and simultaneously accumulates the weight map. Finally, it divides the pixel-level accumulated value by the weight sum to obtain a seamless stitching result.

[0065] Storage module: Used to store model parameters, intermediate calculation results and program instructions; it also stores training data, image patches, prediction results, weight maps and other related data during inference, and can also store downstream analysis data such as cell segmentation masks and cell-level aggregation results.

[0066] This device consists of one or more processors, memory, GPU acceleration units, and the aforementioned software modules. These modules work together to automatically complete the entire process from pathological whole slide image input, model training, inference prediction to downstream analysis and whole slide stitching, realizing all the functions described in this invention. A computer program is stored on the corresponding computer-readable storage medium. When the program is executed by the processor, it implements all the steps described in this invention, such as WSI slicing, token encoding, multimodal fusion, decoding prediction, and weighted stitching.

[0067] The rest is the same as in Example 1.

[0068] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A super-resolution prediction method for spatial molecular phenotypes based on whole-section pathological images, characterized in that, Includes the following steps: Obtain whole-section images of the sample to be analyzed; The pathological whole-section image is subjected to tissue region detection and segmentation processing to generate multiple image blocks containing spatial coordinates; The trained prediction network is used to process the multiple image patches to obtain dense prediction feature maps; The dense predicted feature maps are spliced ​​and backfilled into the full slice coordinate system according to their spatial positions to obtain a full slice-level super-resolution spatial molecular map. The prediction network includes a pathological baseline model, a Transformer backbone network, and a decoder. The pathological baseline model encodes each image patch to obtain an image token sequence, which is then fused with preset start and end tokens to form a fused token sequence. The Transformer backbone network performs multi-layer feature learning on the fused token sequence to extract a multi-scale representation set. The decoder reassembles the multi-scale representation set into a spatial feature map and fuses them step by step to output the dense prediction feature map.

2. The method for super-resolution prediction of spatial molecular phenotypes based on pathological whole-section images according to claim 1, characterized in that, The slicing process specifically involves sliding window slicing that allows adjacent image blocks to overlap, and recording the coordinates and overlap relationship of each image block in the full slice. When the dense predicted feature map is spliced ​​back to the full slice coordinate system according to its spatial location, a weighted fusion strategy with linear edge decay is adopted for the overlapping area.

3. The method for super-resolution prediction of spatial molecular phenotypes based on pathological whole-section images according to claim 2, characterized in that, The weighted fusion strategy with linear edge decay specifically includes: Construct a weight matrix for each image patch, such that the weight of the central region of the image patch is 1, and the weight of the region near the edge decreases by a linear function; The predicted values ​​of each image block are weighted and summed according to the weight matrix, and then normalized using the weights to obtain the final stitching result. The weights in the weight matrix are determined by the overlap width and the image block size, and the weight of pixel (i,j) is the minimum value of the attenuation weights in the horizontal and vertical directions.

4. The method for super-resolution prediction of spatial molecular phenotypes based on pathological whole-section images according to claim 1, characterized in that, The step of performing multi-layer feature learning on the fused token sequence and extracting a multi-scale representation set specifically involves: Multiple sets of tokens are extracted from the multi-layered Transformer backbone network at equal intervals or with a preset layer index to form a multi-scale representation set.

5. The method for super-resolution prediction of spatial molecular phenotypes based on pathological whole-section images according to claim 1, characterized in that, The reorganization into a spatial feature map specifically refers to: The multi-scale representation set is rearranged into a two-dimensional feature map in spatial order, and upsampling or downsampling is performed through convolution or transposed convolution to obtain a spatial feature map with a preset resolution.

6. The method for super-resolution prediction of spatial molecular phenotypes based on pathological whole-section images according to claim 1, characterized in that, During the training of the prediction network, the parameters of the basic pathological model are frozen, and only the parameters of the Transformer backbone network and decoder are updated. The prediction network is trained using a composite loss function, which simultaneously constrains the consistency between the average value of the predicted output within the blot region and the gene expression in the target space, as well as the consistency between the staining texture reconstructed from the output features through convolution and the texture of the original pathological whole-slice image.

7. The method for super-resolution prediction of spatial molecular phenotypes based on pathological whole-section images according to claim 1, characterized in that, The method also includes: The spatial gene expression or its derived pathway scores, which are spatially registered with the pathological whole slice image, are obtained, normalized / standardized, and then gene tokens are generated through Fourier feature embedding. The gene token and image token sequences are concatenated with preset start and end tokens to form the fusion token sequence.

8. The method for super-resolution prediction of spatial molecular phenotypes based on pathological whole-section images according to claim 1, characterized in that, The method also includes: The region of interest (ROI) is marked in the whole pathological slide image. When performing the slicing process, slicing is only performed on the marked ROI.

9. A spatial molecular phenotypic super-resolution prediction device based on pathological whole-section images, characterized in that, include: The image acquisition module is used to acquire pathological whole-section images of the sample to be analyzed; The slicing and coordinate management module is used to perform tissue region detection and slicing processing on the pathological whole slice image, generate multiple image blocks containing spatial coordinates, and record the coordinates and overlap relationship of each image block in the whole slice; The pathological basic model encoding module is used to encode each image block to obtain an image token sequence; The molecular input encoding module responds when it receives a spatial gene expression or its derived pathway score that is spatially registered with the pathological whole slice image. It is used to normalize / standardize the spatial gene expression or its derived pathway score, and then generate a gene token through Fourier feature embedding. The Transformer fusion module is used to fuse the image token sequence with a preset start and end token to form a fusion token sequence, or to concatenate the gene token and image token sequences with preset start and end tokens respectively to form a fusion token sequence, and to perform multi-layer feature learning on the fusion token sequence to extract a multi-scale representation set. The decoding and output module is used to reorganize the multi-scale representation set into a spatial feature map, and fuse them step by step to output the dense prediction feature map; The full-slice weighted stitching module is used to stitch the dense predicted feature map back to the full-slice coordinate system according to its spatial position to obtain a full-slice-level super-resolution spatial molecular map. The storage module is used to store model parameters and intermediate results.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1-8.

Images