An automated method for discovering phenotypic biomarkers in cross-modal pathological images

By employing a dual-branch Transformer architecture for cross-modal learning and multi-scale modeling, the problems of information loss and manual dependence in phenotypic biomarker discovery are solved, enabling automated, low-cost, and efficient phenotypic biomarker discovery and outputting hierarchical biomarkers.

CN123118563APending Publication Date: 2026-11-17NINGBO SHENWEI VISION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611364975.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-04
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies for the discovery of phenotypic biomarkers suffer from problems such as reliance on wet experimental verification, long cycles, high costs, limited sample size, severe information loss, one-sided single-scale analysis, reliance on manual interpretation for visualization methods, and inability of fixed-scale neighborhoods to adapt to tissue heterogeneity.

Method used

A dual-branch Transformer architecture is used for cross-modal representation learning, aligning H&E cell embeddings and mIF cell embeddings into a unified latent space. Through multi-scale hierarchical modeling, the neighborhood is dynamically expanded to identify local distribution differences, thereby achieving automated screening of phenotypic biomarkers.

Benefits of technology

It enables automated discovery of phenotypic biomarkers without wet laboratory validation, shortening the cycle, reducing costs, comprehensively covering multi-scale phenotypic differences, ensuring objectivity and reproducibility, and outputting hierarchical biomarkers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118563A_ABST
    Figure CN123118563A_ABST
Patent Text Reader

Abstract

This invention discloses an automatic discovery method for phenotypic biomarkers in cross-modal pathological images. It extracts multiple spatially corresponding patch pairs and performs spatial registration; performs cell instance segmentation separately, taking the intersection of two mask sets as a shared cell set; constructs a dual-branch Transformer architecture, aligning H&E cell embeddings and mIF cell embeddings into a unified latent space, enabling the H&E branch to extract H&E cell embeddings and aggregation tokens with protein semantics, which are then translated into cell-level, microenvironment-level, tissue-level, and tissue-cell-level phenotypic vectors, with tissues representing a set of instances with spatial metadata; uses the phenotypic vector of each instance as the query center, adaptively determining the optimal radius by expanding the neighborhood and based on the trend of logarithmic multiple changes, identifying local distribution differences between different prognostic populations; filters candidate differential regions, and outputs hierarchical phenotypic biomarkers; this invention achieves fully automatic, multi-scale phenotypic biomarker discovery.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of digital pathological image processing technology, and in particular to an automatic discovery method for phenotypic biomarkers in cross-modal pathological images. Background Technology

[0002] Prognostic biomarkers are crucial tools in precision oncology. Based on their acquisition methods, prognostic biomarkers can be categorized into molecular biomarkers and phenotypic biomarkers. Molecular biomarkers capture the intrinsic state of tumors and guide targeted interventions, but their validation relies on wet experimental techniques such as mIF (multiplex immunofluorescence) staining and spatial transcriptome sequencing, which suffer from long validation cycles, high costs, and limited sample sizes. Phenotypic biomarkers, derived from histopathological morphology, reflect higher-order tissue architecture and microenvironment characteristics closely related to patient prognosis. Hematoxylin and eosin (H&E) staining, as the gold standard for clinical diagnosis, offers advantages such as low cost and high accessibility; however, systematically identifying phenotypic biomarkers from H&E images still faces significant challenges. Currently, computational pathology-based phenotypic biomarker discovery protocols mainly include the following categories: (1) Methods based on clustering (such as k-means, hierarchical clustering) or dimensionality reduction (such as UMAP, VAE) compress high-dimensional pathological features into low-dimensional embeddings, and then perform differential analysis. These methods are computationally efficient, but compressing high-dimensional pathological features into low-dimensional embeddings leads to the irreversible loss of a large amount of fine-grained phenotypic variation information, especially the subtle differences in the boundary regions between clusters and the characteristics of rare cell subpopulations. This results in subsequent differential analysis being conducted only at the coarse-grained level, making it difficult to discover biomarkers with fine discriminative power.

[0003] (2) Visualization-driven methods based on heatmaps, attention maps or feature importance scores are essentially descriptive rather than discriminative in their output. They require subjective posterior interpretation by experts, are labor-intensive and highly subjective, and have extremely poor repeatability and scalability, making it difficult to achieve automated large-scale biomarker discovery.

[0004] (3) Differential abundance analysis-based methods (such as diffcyt) use high-resolution clustering combined with statistical tests to perform differential abundance analysis. These methods are limited to analysis at a single biological scale (such as only cell-level differential abundance analysis) and lack the ability to perform hierarchical joint modeling from cells to microenvironment, then to tissues and spatial contexts, resulting in the discovery of biomarkers that are often one-sided.

[0005] (4) Spatial neighborhood analysis-based methods (such as Squidpy) discover local cell aggregation patterns through spatial maps and neighborhood enrichment analysis. These methods use fixed neighborhood radii or k-nearest neighbor definitions, which cannot adapt to the huge differences in cell density and tissue structure in different regions, severely limiting the resolution and biological interpretability of spatial neighborhood analysis. Summary of the Invention

[0006] To address the shortcomings of existing technologies, such as reliance on wet experimental verification, information loss due to high-dimensional feature compression, lack of systematicity in single-scale analysis, reliance on manual interpretation for visualization methods, and the inability of fixed-scale neighborhoods to adapt to tissue heterogeneity, this invention aims to provide a fully automated, multi-scale, and highly interpretable method for the automatic discovery of phenotypic biomarkers in cross-modal pathological images. This method can directly start from patient cohorts with different prognoses, compare phenotypic distribution differences between cohorts, and inversely infer the cellular phenotypic factors leading to prognostic differences, thus eliminating the dependence on wet experimental verification. Through a two-branch Transformer architecture for cross-modal representation learning, the optimized low-dimensional H&E cell embedding and optimized mIF cell embedding are aligned into a unified latent space, enabling the trained H&E branch to learn from H&E... Patch extracts optimized low-dimensional H&E cell embeddings and H&E aggregation tokens with protein semantics; the optimized low-dimensional H&E cell embeddings are translated step by step into cell-level phenotypic vectors, microenvironment-level phenotypic vectors, tissue-level phenotypic vectors, and tissue-cell-level phenotypic vectors to achieve multi-scale hierarchical joint modeling; using the phenotypic vector of each instance as the query center, the optimal radius is adaptively determined by dynamically expanding the neighborhood and based on the changing trend of logarithmic multiple values, and the local distribution differences between different prognostic populations are identified without prior dimensionality reduction; fully automated phenotypic biomarker screening is achieved, and hierarchical phenotypic biomarkers are output.

[0007] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: an automatic discovery method for cross-modal pathological image phenotypic biomarkers, comprising the following steps: S1: Obtain paired H&E and mIF images of the same tissue slice, extract multiple H&E patches and mIF patches corresponding to the spatial location, and perform spatial registration on each patch pair; S2: Perform cell instance segmentation on the spatially registered H&E patch and mIF patch respectively to obtain the H&E instance segmentation mask of each cell in the H&E patch and the mIF instance segmentation mask of each cell in the mIF patch, and take the intersection of the two mask sets as the shared cell set. S3: Construct a dual-branch Transformer architecture, including an H&E branch and a mIF branch; extract the H&E cell embeddings of each cell in the shared cell set through the H&E branch, map them to a low-dimensional space, obtain the optimized low-dimensional H&E cell embeddings, and generate an H&E aggregation token; calculate the average fluorescence intensity of each cell in the shared cell set in each protein target channel of the mIF image through the mIF branch, form the initial mIF cell embeddings, obtain the optimized mIF cell embeddings, and generate an mIF aggregation token. With the optimization of low-dimensional H&E cell embedding and optimization of mIF cell embedding as alignment objectives, the parameters of a two-branch Transformer architecture are trained so that the embedding of the same cell in the two modalities tends to be consistent in the latent space. S4: After completing the cross-modal alignment training, the phenotypic biomarker discovery stage is entered: Based on the trained H&E branch, the optimized low-dimensional H&E cell embedding of each cell in each H&E patch of the H&E image of the patient to be tested and the H&E aggregation token of each H&E patch are obtained, so as to optimize the K-dimensional space in which the low-dimensional H&E cell embedding is located as the phenotypic embedding space, where K is the number of protein target channels in the mIF image; The optimized low-dimensional H&E cell embeddings are fused with local density information to construct cell-level phenotypic vectors. Using H&Epatch as the unit, the optimized low-dimensional H&E cell embeddings of each cell are density-weighted and averaged according to the normalized density fraction to obtain microenvironment-level phenotypic vectors. The H&E aggregation token is used as the tissue-level phenotypic vector. The tissue-level phenotypic vector corresponds to a tissue region. The optimized low-dimensional H&E cell embeddings of each cell contained in the tissue region are fused with local density information to construct the tissue-cell level phenotypic vectors corresponding to each cell. Cell-level phenotypic vectors, microenvironment-level phenotypic vectors, tissue-level phenotypic vectors, and tissue-cell-level phenotypic vectors are each organized into a set of instances with spatial metadata. S5: Use the phenotypic vector of each instance in the instance set at the specified level as the query center; Radius retrieval is performed in the phenotypic embedding space with the query center as the center. The retrieval radius is gradually increased from the preset initial radius, the scale-dependent neighborhood is dynamically expanded, and the logarithmic multiple change value between different prognostic groups in the scale-dependent neighborhood under each retrieval radius is calculated in real time. The candidate retrieval radius is determined based on the trend of the logarithmic multiple change value. The smaller value between the candidate retrieval radius and the preset maximum retrieval radius is taken as the retrieval radius when the expansion is terminated. The scale-dependent neighborhood under this retrieval radius is taken as the candidate difference region. S6: Filter the candidate differential regions and output the remaining candidate differential regions as phenotypic biomarkers.

[0008] Before the step of extracting multiple H&E patches and mIF patches corresponding to the spatial location, the method further includes: using the DAPI channel of the mIF image as a spatial reference, performing global rigid registration on the H&E image to achieve global spatial coarse alignment; The spatial registration of each patch pair includes: performing local non-rigid registration on each patch pair to optimize the local deformation within the patch pair and achieve finer local spatial alignment.

[0009] The H&E branch includes an H&E-specific cell encoder, a fully connected layer, and an H&E Transformer module. The H&E instance segmentation mask of each cell in the shared cell set is multiplied element-wise with the spatially registered H&E patch to obtain the corresponding cell image region. The cell image region of each cell in the shared cell set is input into the H&E-specific cell encoder to obtain the corresponding initial H&E cell embedding. The initial H&E cell embedding of each cell in the shared cell set is projected through the fully connected layer to obtain the corresponding initial low-dimensional H&E cell embedding. The initial low-dimensional H&E cell embedding of each cell in the shared cell set is input into the H&E Transformer module to obtain the optimized low-dimensional H&E cell embedding for each cell in the shared cell set, as well as the patch-level H&E aggregation token. The mIF branch includes a non-parametric embedding calculation module and an mIF Transformer module. The non-parametric embedding calculation module calculates the average fluorescence intensity of each cell in the shared cell set in each protein target channel of the mIF image to form the corresponding initial mIF cell embedding. The initial mIF cell embedding of each cell in the shared cell set is input into the mIFTransformer module to obtain the optimized mIF cell embedding of each cell in the shared cell set, as well as the patch-level mIF aggregation token.

[0010] The training employs a combination of patch-level contrastive loss and patch-level mean squared error loss. The acquisition of the patch-level contrast loss includes: For the spatially registered H&E patch and mIF patch, construct a positive sample pair by combining the H&E aggregation token corresponding to the current H&E patch and the mIF aggregation token corresponding to the current mIF patch; The H&E aggregate token corresponding to the current H&E patch and the mIF aggregate token corresponding to the mIF patch that does not correspond to its spatial location are used to form a negative sample pair; Calculate the patch-level contrast loss based on the cosine similarity between positive and negative sample pairs; The patch-level mean squared error loss is used to constrain the numerical similarity between the optimized low-dimensional H&E cell embedding and the optimized mIF cell embedding for the same cell.

[0011] The construction of the cell-level phenotypic vector includes: For each cell in the H&E patch, the sum of the attention weights of the optimized low-dimensional H&E cell embedding of that cell to the optimized low-dimensional H&E cell embeddings of other cells in the H&E patch is calculated as the local density information of that cell. The optimized low-dimensional H&E cell embedding of the cell is concatenated with the local density information to obtain the cell-level phenotypic vector of the cell. The construction of the microenvironment-level phenotypic vector includes: Using H&E patches as units, the corresponding normalized density fraction is calculated based on the local density information of each cell; The optimized low-dimensional H&E cell embeddings of each cell were density-weighted and averaged according to the normalized density fraction to obtain the microenvironment-level phenotypic vector.

[0012] Before the query center is determined, the following are included: Balanced sampling is performed on a set of instances at a specified level, wherein the balanced sampling includes at least one of the following methods: setting a sampling upper limit quota for the set of instances, grid-based sampling, and pre-group balancing; All instances in the sampled subset are organized into a KD-Tree in the phenotypic embedding space, and radius retrieval is performed through the KD-Tree.

[0013] The calculation of the logarithmic multiple change value includes: For a set of instances at the cellular level, both the logarithmic fold change in absolute abundance and the logarithmic fold change in relative density are calculated: , Where q represents the query center and r represents the search radius. This represents the logarithmic change in absolute abundance. This represents the logarithmic change in relative density. This represents the set of all cell-level phenotypic vectors with a poor prognostic label within the scale-dependent neighborhood centered at the query center q. ε represents the set of all cell-level phenotypic vectors with a good prognostic label within the scale-dependent neighborhood centered at query center q, where ε is a small constant to prevent division by zero, and µ1(q) represents... The average local density information corresponding to all cell-level phenotypic vectors in the data, µ0(q) represents... The average value of the local density information corresponding to all cell-level phenotypic vectors; then the logarithmic fold change value of absolute abundance and the logarithmic fold change value of relative density are fused into the logarithmic fold change value at the cell level; For a set of instances at the microenvironment level, the logarithmic multiple change value is calculated through kernel similarity aggregation: LogFC env (q,r) represents the logarithmic multiple change value of the microenvironment level, S1 represents the kernel similarity weighted sum of the poor prognosis group, and S0 represents the kernel similarity weighted sum of the good prognosis group. , This represents the set of all microenvironment-level phenotypic vectors with a poor prognosis label within the scale-dependent neighborhood centered at the query center q. z represents the set of all micro-environmental phenotypic vectors with a good prognostic label within the scale-dependent neighborhood centered at the query center q. The microenvironment-level phenotypic vector in the vector, κ(·,·) is the Gaussian kernel function; For a set of instances at the organizational level, the logarithmic multiple change is calculated based on the ratio of the number of organizational-level phenotypic vectors in different prognostic populations: LogFC tis (q,r) represents the logarithmic multiple of the organizational hierarchy. This represents the set of all organizational-level phenotypic vectors with a poor prognostic label within the scale-dependent neighborhood centered at the query center q. It represents the set of all organizational-level phenotypic vectors with good prognostic labels within the scale-dependent neighborhood centered on the query center q; For a set of instances at the tissue-cell level, the distribution differences of cells within their respective tissue regions are considered, and both the absolute abundance logarithmic fold change and the relative density logarithmic fold change are calculated. These two values ​​are then fused into a single logarithmic fold change value (LogFC) for the tissue-cell level. tis-cell (q,r).

[0014] The process of determining the candidate retrieval radius based on the trend of logarithmic multiple changes, using the smaller of the candidate retrieval radius and the preset maximum retrieval radius as the retrieval radius at which expansion is terminated, and using the scale-dependent neighborhood under this retrieval radius as the candidate difference region includes: Monitor the second derivative of the logarithmic multiple change value with respect to the search radius. When the absolute value of the second derivative reaches the extreme point or the search radius reaches the preset maximum search radius, terminate the expansion of the scale-dependent neighborhood and determine the search radius at the time of termination as the adaptive optimal radius. The scale-dependent neighborhood of the query center under the adaptive optimal radius is used as the candidate difference region. The determination of the adaptive optimal radius includes: Calculate the second derivative ρ(q,r) of the logarithmic change with respect to the search radius. Where LogFC(q,r) represents the logarithmic multiple change between different prognostic groups within the neighborhood that depends on the scale of the retrieval radius r; During the scale-dependent neighborhood dynamic expansion process, the search radius corresponding to the first time the second derivative approaches zero from a negative value is determined as the candidate search radius; The smaller value between the candidate retrieval radius and the preset maximum retrieval radius is taken as the adaptive optimal radius.

[0015] The process of filtering candidate differential regions and outputting the retained candidate differential regions as phenotypic biomarkers includes: Calculate the perceived support score for each candidate region of difference at a specified level; All candidate difference regions are deredundant based on the support-perceived score to remove overlapping and redundant candidate difference regions. The retained candidate differential regions are output as phenotypic biomarkers, which include hierarchical type, differential direction, differential intensity, and spatial location information.

[0016] The support-perceived score calculation formula is: Score(q) = |LogFC(q,r)| best )|·F mono (|N ei (q,r best )|), where Score(q) represents the value of the query center q within the adaptive optimal radius r. best Candidate difference region N ei (q,r best Support-perceived score, LogFC(q,r) best ) represents N ei (q,r best The corresponding logarithmic multiple change value, |N ei (q,r best )| indicates finding N ei (q,r best Number of instances within F mono (·) represents a monotonically saturated function; The redundancy removal process is non-maximum suppression, including: All candidate regions of difference are sorted from highest to lowest according to the perceived support score; Select the candidate difference region with the highest support-perceived score from the currently unprocessed candidate difference regions as the retained candidate difference regions; Calculate the neighborhood overlap between the retained candidate difference regions and the remaining unprocessed candidate difference regions. When the neighborhood overlap exceeds a preset overlap threshold, suppress the candidate difference regions with lower support perception scores. The neighborhood overlap is the number of instances shared by two candidate difference regions. Repeat the selection, calculation, and suppression steps described above until all candidate difference regions have been processed, resulting in all retained candidate difference regions.

[0017] Compared with the prior art, the advantages of the present invention are as follows: This invention aligns optimized low-dimensional H&E cell embeddings and optimized mIF cell embeddings into a unified latent space using a dual-branch Transformer architecture. This enables the trained H&E branch to extract optimized low-dimensional H&E cell embeddings and H&E aggregation tokens with protein semantics from conventional H&E images. As a result, molecular-level phenotypic information can be obtained in the phenotypic biomarker discovery stage without wet experiments such as mIF staining or spatial transcriptome sequencing. This eliminates the dependence on wet experimental verification in the traditional "cause-to-effect" paradigm, significantly shortens the phenotypic biomarker discovery cycle, and reduces resource consumption.

[0018] This invention constructs phenotypic vectors at four levels—cell level, microenvironment level, tissue level, and tissue-cell level—and organizes them into instance sets with spatial metadata. This enables multi-scale hierarchical joint modeling from single cells to tissue space, comprehensively covering multi-scale phenotypic differences in the tumor microenvironment. It avoids the one-sidedness caused by single-scale analysis and can more comprehensively reveal the spatial logic of tumor progression.

[0019] This invention uses the phenotypic vector of each instance as the query center, dynamically expands the neighborhood and adaptively determines the optimal radius based on the changing trend of the logarithmic multiple change value, directly identifying the local distribution differences between different prognostic groups in the original K-dimensional phenotypic embedding space. It does not require prior dimensionality reduction, overcomes the defect that fixed-scale neighborhood cannot adapt to tissue heterogeneity, avoids information loss caused by dimensionality reduction, and can discover localized fine phenotypic patterns that are masked by clustering methods.

[0020] This invention automatically filters candidate regions of difference through support-aware scoring and non-maximum suppression, and outputs hierarchical phenotypic biomarkers. The entire process, from image input to phenotypic biomarker output, is fully automated, requiring no manual threshold setting, no manual selection of regions of interest, and no manual posterior interpretation, thus ensuring the objectivity, repeatability, and large-scale scalability of the discovery.

[0021] This invention provides a highly efficient, objective, and scalable tool for the automated discovery of prognostic phenotypic biomarkers for precision oncology. Attached Figure Description

[0022] Figure 1 This is an overall flowchart of the method of the present invention. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0024] This invention proposes an automated method for discovering phenotypic biomarkers in cross-modal pathological images, the overall flowchart of which is shown below. Figure 1 As shown, it includes the following steps: S1: Obtain paired H&E and mIF images of the same tissue slice, extract multiple H&E patches and mIF patches corresponding to the spatial locations, and perform spatial registration on each patch pair.

[0025] The mIF image contains K+1 channels, namely the DAPI channel and K protein target channels.

[0026] Although H&E and mIF images originate from the same tissue slice and have a macroscopic correspondence at the anatomical structure level, differences in the slide preparation processes (dehydration, antigen retrieval, antibody incubation, etc.) of the two staining techniques, as well as differences in imaging parameters of different scanning devices, can lead to non-negligible spatial displacement and local deformation at the pixel level. Without registration, this will cause semantic mismatch in cross-modal training data. Therefore, this invention preferably includes, before the step of extracting multiple H&E patches and mIF patches corresponding to spatial locations, the following steps: using the DAPI (nuclear staining) channel of the mIF image as a spatial reference, performing global rigid registration on the H&E image to achieve global spatial coarse alignment; after extracting patch pairs according to their corresponding spatial locations, performing spatial registration on each patch pair, including: performing local non-rigid registration on each patch pair to optimize the local deformation within the patch pair and achieve finer local spatial alignment to ensure accurate spatial correspondence of each cell in the two modalities. In some implementations, if the deformation between the H&E image and the mIF image is minimal and does not affect the accuracy of subsequent cross-modal characterization alignment, the above-mentioned global spatial coarse alignment and / or local spatial fine alignment steps can be omitted, or a simplified / lightweight alignment method can be adopted.

[0027] In a specific example, global rigid registration is implemented using Mattes mutual information. The number of histogram bins for Mattes mutual information is set to 50, the metric sampling ratio is set to 1.0, the optimizer uses conventional step-size gradient descent, the initial learning rate is set to 1.5, and the minimum step size is set to 10. -6 The maximum number of iterations was set to 500. After coarse global spatial alignment of the DAPI channels of the H&E image and the mIF image, spatial alignment with other protein target channels was also achieved.

[0028] In a specific example, the globally spatially coarsely aligned H&E and mIF images are cropped into multiple non-overlapping H&E patches and mIF patches, thus forming multiple pairs of patches that are initially corresponding in spatial location.

[0029] In a specific example, local non-rigid registration is achieved using the Demons algorithm (80 iterations, deformation field smoothness of 1.0). Fine local spatial alignment further improves spatial consistency. After this fine local spatial alignment step, the resulting patch pairs are highly consistent in spatial anatomical location, with the same cell in both modalities precisely corresponding in spatial location, which can be used as input for subsequent processing.

[0030] S2: Perform cell instance segmentation on the spatially registered H&E patch and mIF patch respectively to obtain the H&E instance segmentation mask of each cell in the H&E patch and the mIF instance segmentation mask of each cell in the mIF patch, and take the intersection of the two mask sets as the shared cell set.

[0031] In a specific example, a pre-trained cell instance segmentation model is used to segment cells on the spatially registered H&Epatch and mIF patch, respectively.

[0032] Preferably, the cell instance segmentation model is the Cellpose model, which is based on the U-Net architecture and flow alignment mechanism, enabling accurate cell instance segmentation without relying on a large amount of labeled data. The spatially registered H&E patch is input into the Cellpose model, which outputs the H&E instance segmentation mask for each cell in the H&E patch. Each cell corresponds to an independent binary mask region, and the H&E instance segmentation masks of all cells in the H&E patch constitute the H&E mask set M. HE Similarly, the spatially registered mIF patch is input into the same Cellpose model, and the output is the mIF instance segmentation mask for each cell in the mIF patch. The mIF instance segmentation masks for all cells in the mIF patch constitute the mIF mask set M. mIF .

[0033] Since the H&E patch and mIF patch have already been spatially registered in this step, the cells in the two modalities have a precise spatial correspondence. Therefore, the intersection of the two mask sets can be directly taken to obtain the shared cell set M. Specifically, this intersection operation involves: for each H&E instance segmentation mask in the H&E mask set, determining whether it coincides spatially with a certain mIF instance segmentation mask in the mIF mask set. If they coincide, the instance segmentation mask is retained.

[0034] S3: To achieve cross-modal alignment of the embeddings of the same cell in the latent space under two modalities, a two-branch Transformer architecture is constructed, including an H&E branch and a mIF branch. The H&E branch extracts the initial H&E cell embeddings of each cell in the shared cell set, maps them to a low-dimensional space, and then obtains optimized low-dimensional H&E cell embeddings, generating H&E aggregation tokens. The mIF branch calculates the average fluorescence intensity of each cell in the shared cell set in each protein target channel of the mIF image, forming the initial mIF cell embeddings, and then obtains optimized mIF cell embeddings, generating mIF aggregation tokens. During the training phase, the parameters of the two-branch Transformer architecture are trained with optimized low-dimensional H&E cell embeddings and optimized mIF cell embeddings as alignment targets, so that the embeddings of the same cell in the two modalities tend to be consistent in the latent space.

[0035] Further defined, the H&E branch includes an H&E-specific cell encoder, a fully connected layer, and an H&E Transformer module. The H&E instance segmentation mask of each cell in the shared cell set is multiplied element-wise with the spatially registered H&E patch to obtain the corresponding cell image region. The cell image region of each cell in the shared cell set is input into the H&E-specific cell encoder to obtain the corresponding initial H&E cell embedding. The initial H&E cell embedding of each cell in the shared cell set is projected through the fully connected layer to obtain the corresponding initial low-dimensional H&E cell embedding. The initial low-dimensional H&E cell embedding of each cell in the shared cell set is input into the H&E Transformer module to obtain the optimized low-dimensional H&E cell embedding for each cell in the shared cell set, as well as the patch-level H&E aggregation token. The mIF branch includes a non-parametric embedding calculation module and an mIFTransformer module. The non-parametric embedding calculation module calculates the average fluorescence intensity of each cell in the shared cell set in each protein target channel of the mIF image to form the corresponding initial mIF cell embedding. The initial mIF cell embedding of each cell in the shared cell set is input into the mIF Transformer module to obtain the optimized mIF cell embedding of each cell in the shared cell set, as well as the patch-level mIF aggregation token.

[0036] Here, the H&E branch is processed on a unit of H&E patch, and similarly, the mIF branch is processed on a unit of mIF patch. That is, for each pair of spatially corresponding and spatially registered H&E patch and mIF patch, the H&E branch and mIF branch are executed respectively.

[0037] In a specific example, the initial H&E cell embedding of the nth cell in the shared cell set is represented as e. HE,n , , of which F enc (·) represents an H&E-specific cell encoder, preferably a pre-trained CTransPath cell feature encoder, which is a hybrid pathological image model based on CNN and Swin Transformer. This model, after self-supervised pre-training on a large number of unlabeled H&E images, can extract highly discriminative cell morphological features, i.e., H&E cell embeddings. The CTransPath cell feature encoder is only an exemplary implementation; other pre-trained pathological image feature extraction models with similar functions can also be used instead. n Let represent the instance segmentation mask of the nth cell in the shared cell set M. Since the shared cell set M is the intersection of the H&E mask set and the mIF mask set, this instance segmentation mask corresponds to the same cell in both the spatially registered H&E patch and mIF patch. The symbol for element-wise multiplication is x. HE This represents the H&E patch after spatial registration. The result represents the cell image region of the nth cell in the shared cell set M. HE,n The dimension is D, and a typical reference value for D is 768 (for the pre-trained CTransPath cellular feature encoder) or 1024.

[0038] In a specific example, both the initial low-dimensional H&E cell embedding and the optimized low-dimensional H&E cell embedding have the dimension of K.

[0039] In a specific example, for each cell in the shared cell set, in each protein target channel of the mIF image, the average fluorescence intensity of all pixels within the area covered by the mIF instance segmentation mask of that cell is taken (pixels below the signal threshold are not counted), and this average is used as the average fluorescence intensity of that cell in that protein target channel. The average fluorescence intensities of all K protein target channels are combined into a K-dimensional vector, which is used as the initial mIF cell embedding for that cell. Specifically, the initial mIF cell embedding of the nth cell in the shared cell set is represented as e. mIF,n e mIF,n Let e ​​be a K-dimensional vector. mIF,n =[v 1,n,v 2,n ,……,v K,n ], where v k,n The value represents the average fluorescence intensity of the nth cell in the shared cell set in the kth protein target channel of the mIF image, where k = 1, 2, ..., K. k,n =Avg(c k ,m n ,r cell,n ,v threshold ), Avg(·) represents the average, c k This represents the fluorescence grayscale image of the spatially registered mIF patch in the k-th protein target channel, r. cell,n v represents the radius range of the nth cell in the shared cell set, used to define the computational region. threshold This represents the signal threshold; fluorescence intensity below this value is considered background noise. mIF,n The dimension is K, where K is much smaller than D, e mIF,n The protein expression profile of each cell was captured in extremely low dimensions.

[0040] Both the H&E Transformer and mIF Transformer modules include self-attention and cross-attention processing.

[0041] Specifically, training employs a patch-level contrastive loss and a patch-level mean squared error loss for joint training. For spatially registered H&E patches and mIF patches, positive sample pairs are formed by combining the H&E aggregate tokens corresponding to the current H&E patch and the mIF aggregate tokens corresponding to the current mIF patch. Negative sample pairs are formed by combining the H&E aggregate tokens corresponding to the current H&E patch and the mIF aggregate tokens corresponding to the mIF patch whose spatial location does not correspond to the current H&E patch. The patch-level contrastive loss is calculated based on the cosine similarity of the positive and negative sample pairs. The patch-level mean squared error loss is used to constrain the numerical similarity between the optimized low-dimensional H&E cell embedding and the optimized mIF cell embedding for the same cell.

[0042] Specifically, during training, the parameters of the H&E specific cell encoder, fully connected layer, H&E Transformer module and mIF Transformer module are updated. Among them, the H&E specific cell encoder is a pre-trained model, and its parameters are fine-tuned during training.

[0043] Specifically, the contrastive loss is calculated for each pair of spatially corresponding and spatially registered H&E patch and mIF patch. When forming a negative sample pair, a negative sample pair is formed by the H&E aggregate token corresponding to the H&E patch in that patch pair and the mIF aggregate token corresponding to an mIF patch at a different spatial location from the H&E patch. T = 2048 mIF patches at different spatial locations are selected randomly or with a preset step size. The contrastive loss L... patch =CTR(E HE E mIF ,{E neg,t |t=1,2,……,T}), where CTR(·) represents the contrastive loss function, E HE This represents the H&E aggregate token corresponding to the H&Epatch in the current patch pair, E mIF E represents the mIF aggregate token corresponding to the mIF patch in the current patch pair. neg,t This represents the mIF aggregate token corresponding to the t-th mIF patch at different spatial locations. The loss alignment H&E aggregate token is compared with the mIF aggregate token.

[0044] Specifically, the mean squared error loss, for each spatially corresponding and spatially registered H&E patch and mIF patch, calculates the mean squared error between the optimized low-dimensional H&E cell embedding and the optimized mIF cell embedding for all cells in the shared cell set of that patch pair. The mean squared error loss aligns the optimized mIF cell embedding with the optimized low-dimensional H&E cell embedding.

[0045] Specifically, the joint loss L total =L patch +λL cell Where λ is a weighting coefficient used to balance the contrast loss and the mean squared error loss, and its value is determined based on L during training. patch With L cell The numerical scale is determined, typically ranging from 0.01 to 10. In a specific example, λ is set to 1, and L... cell This represents the mean square error loss.

[0046] S4: After completing the cross-modal alignment training described above, proceed to the phenotypic biomarker discovery phase. Unlike the training phase, this phase only requires H&E images of the patients to be tested; mIF images and the construction of shared cell sets are not required. The H&E images of the patients to be tested are cropped into multiple H&E patches. For each H&E patch, a pre-trained cell instance segmentation model is used to segment cells, obtaining the H&E cell instance mask for each cell in the H&E patch. The H&E cell instance mask of each cell is multiplied element-wise with the H&E patch to obtain the corresponding cell image region for each cell. The corresponding cell image region of each cell is input into the trained H&E-specific cell encoder to obtain the initial H&E cell embedding for each cell. The initial H&E cell embedding of each cell in the same H&E patch is input into the trained fully connected layer for projection to obtain the initial low-dimensional H&E cell embedding of each cell. This initial low-dimensional H&E cell embedding is then input into the trained H&E Transformer module to obtain the K-dimensional optimized low-dimensional H&E cell embedding for each cell and the K-dimensional patch-level H&E aggregation token for the H&E patch, where K is the number of protein target channels in the mIF image used for cross-modal alignment training.

[0047] It should be noted that although the optimized low-dimensional H&E cell embeddings obtained after training the H&E branch have been aligned to the protein semantic space, they only describe the phenotypic features of a single cell. However, prognostic differences in the tumor microenvironment often manifest at multiple biological scales: including absolute changes in the number of specific cell types, whether cells aggregate in local areas to form specific spatial patterns, and compositional differences in different tissue regions (such as the tumor core, invasion front, and stroma). If the analysis is limited to the cellular level, higher-level phenotypic information such as the microenvironment, tissue, and spatial context will be lost. Therefore, this invention translates the optimized low-dimensional H&E cell embeddings stepwise into phenotypic vectors at four biological levels: cellular, microenvironment, tissue, and tissue-cell, thus providing multi-granularity analytical objects for subsequent differential retrieval. Then, the following multi-level phenotypic spatial transformation operation is performed.

[0048] S41: Based on the trained H&E branch, obtain the optimized low-dimensional H&E cell embedding of each cell in each H&E patch of multiple patients and the H&E aggregation token of each H&E patch, so as to optimize the K-dimensional space in which the low-dimensional H&E cell embedding is located as the phenotypic embedding space, where K is the number of protein target channels in the mIF image.

[0049] In a specific example, each dimension of the phenotypic embedding space corresponds to a protein target channel in the mIF image. Within this space, the optimized low-dimensional H&E cell embedding for each cell is mapped to a feature point, and the distribution differences of cells with different prognostic labels within this space are reflected as differences in local point density. By comparing the differences in point distribution density between different prognostic populations within the phenotypic embedding space, prognostic-related phenotypic biomarkers can be identified.

[0050] S42: The optimized low-dimensional H&E cell embeddings are fused with local density information to construct a cell-level phenotypic vector; using H&Epatch as the unit, the optimized low-dimensional H&E cell embeddings of each cell are density-weighted and averaged according to the normalized density fraction to obtain a microenvironment-level phenotypic vector; the H&E aggregation token is used as the tissue-level phenotypic vector; the tissue-level phenotypic vector corresponds to a tissue region, and the optimized low-dimensional H&E cell embeddings of each cell contained in the tissue region are fused with local density information to construct the tissue-cell-level phenotypic vector corresponding to each cell.

[0051] S43: Organize the cell-level phenotype vector, microenvironment-level phenotype vector, tissue-level phenotype vector, and tissue-cell-level phenotype vector into sets of instances with spatial metadata.

[0052] In a specific example, the construction of the cell phenotype vector includes: for each cell in the H&E patch, calculating the sum of the attention weights of the optimized low-dimensional H&E cell embedding of that cell to the optimized low-dimensional H&E cell embeddings of other cells in the H&E patch, which is used as the local density information of that cell; concatenating the optimized low-dimensional H&E cell embedding of that cell with the local density information to obtain the cell phenotype vector of that cell.

[0053] In a specific example, the construction of the microenvironment phenotype vector includes: calculating the corresponding normalized density score based on the local density information of each cell, using H&E patch as the unit; and performing a density-weighted average of the optimized low-dimensional H&E cell embeddings of each cell according to the normalized density score to obtain the microenvironment phenotype vector.

[0054] Specifically, for the j-th cell in the H&E patch, its local density information is represented as d. j , , where N cell This represents the number of cells in the H&E patch, j=1,2,……,N cell e HE,K,jLet A[·,·] represent the optimized low-dimensional H&E cell embedding of the j-th cell in the H&E patch, and let A[·,·] represent the cross-attention mechanism of the H&E Transformer module in the H&E branch. Used to calculate e HE,K,j right The attention weight and local density information are used to reflect the degree of cell-level phenotypic aggregation. In a H&E patch, when the optimized low-dimensional H&E cell embedding of a cell is highly similar to that of other cells, the attention weight is large, and the local density information value of the cell is high, indicating that the cell belongs to a phenotypically dense cell community; conversely, the local density information value of the cell is low, indicating that the cell is in a phenotypic sparse region. The cell-level phenotypic vector of the j-th cell in the H&E patch is defined as z. cell,j , z cell,j =[e HE,K,j , d j ].

[0055] Specifically, for the j-th cell in the H&E patch, its normalized density fraction is represented as w. j , The microenvironment-level phenotypic vector is represented as z. env , This calculation method makes phenotypic dense cell populations contribute more to the microenvironment-level phenotypic vector, while phenotypic sparse or isolated cells contribute less. It can effectively capture the dominant cell phenotypic composition in local regions and suppress the interference of noise or rare cell types on the microenvironment characterization.

[0056] Specifically, the tissue phenotype vector is represented as z. tis , z tis =E HE Organization-level phenotypic vectors are semantic summaries of all cells within the entire H&E patch, capturing the emergent properties of the cell set. HE The optimized low-dimensional H&E cell embedding of all cells in the H&E patch is obtained through the self-attention and cross-attention mechanisms of the H&ETransformer module. This is aggregated into a fixed-dimensional vector that reflects the overall state of the tissue region covered by the H&E patch and can distinguish different types of tissue regions, such as the tumor core region, the invasion front region, and the mesenchymal region.

[0057] Specifically, for z tis The corresponding tissue region, whose tissue-cell phenotypic vector of the j-th cell is represented by z. tis-cell,j , z tis-cell,j =[e HE,K,j ,dj ,z tis ], e HE,K,j and d j They represent z respectively tis The optimized low-dimensional H&E cell embedding and local density information of the j-th cell within the corresponding tissue region. Tissue-cell level phenotypic vectors can reveal prognostic differences in the same optimized low-dimensional H&E cell embedding at different tissue locations—for example, a certain optimized low-dimensional H&E cell embedding in the tumor core may be associated with poor prognosis, while in the invasion front area it may be associated with good prognosis. By examining location-specific phenotypic variations at different tissue locations, spatially context-dependent prognostic biomarkers can be discovered.

[0058] In a specific example, spatial metadata includes cell-level spatial metadata and patch-level spatial metadata. Cell-level spatial metadata includes the identifier of the H&E image to which the cell belongs, the position of the H&E patch to which the cell belongs in the H&E image, and the position of the cell in the H&E patch. Patch-level spatial metadata includes the identifier of the H&E image to which the H&E patch belongs and the position of the H&E patch in the H&E image. Cell-level phenotypic vectors and tissue-cell-level phenotypic vectors include cell-level spatial metadata when organizing instance sets, while microenvironment-level phenotypic vectors and tissue-level phenotypic vectors include patch-level spatial metadata when organizing instance sets.

[0059] The purpose of spatial metadata is to allow for the mapping back to the original H&E image when subsequent differential retrieval reveals a significant difference in a particular instance region between prognostic groups. This spatial metadata enables visualization verification and clinical interpretation. In a specific example, phenotypic vectors at four granularities are organized into instance sets with spatial metadata. Specifically, the instance set at the cell level is represented as S. cell S cell Each instance in contains z cell,j ,Y,δ j Where Y represents the prognostic label of the H&E image, Y∈{0,1}, Y=0 indicates a good prognosis, Y=1 indicates a poor prognosis, and δ j The spatial metadata of the j-th cell is represented by S; the set of instances of the microenvironment hierarchy is represented by S. env S env Each instance in contains z env Y, Φ, where Φ represents the spatial metadata of the H&E patch; the set of instances at the organizational level is represented by S. tis S tis Each instance in contains z tis The set of instances at the tissue-cell level is represented as S, where Y and Φ are the values ​​of Y and Φ respectively. tis-cell S tis-cellEach instance in contains z tis-cell,j ,Y,δ j .

[0060] In a specific example, each instance in the instance set contains the phenotypic vector of that instance, the corresponding prognostic label, and spatial metadata.

[0061] S5: In the phenotypic embedding space, the differences between different prognostic groups are often local and scale-variable. Some phenotypic differences only appear in specific embedding space regions, and the optimal radius of different regions is different. Traditional methods use fixed-radius neighborhood or clustering strategies for difference analysis. Their fixed neighborhood scale cannot adapt to the differences in data distribution density in different regions, leading to the mixing of too many different types of instances in dense regions and the omission of meaningful difference signals in sparse regions. To address these problems, this invention designs a dynamic retrieval algorithm with the phenotypic vector of each instance as the query center. By adaptively determining the optimal radius of each query center, it automatically discovers candidate difference regions with statistical significance. Specifically: S51: To ensure computational efficiency and unbiased estimation, a balanced sampling is performed on the instance set at a specified level. The balanced sampling includes at least one of the following methods: setting a sampling upper limit quota for the instance set, grid-based sampling, and prognostic population balancing; all instances in the sampled instance subset are organized into a KD-Tree in the phenotypic embedding space, and the radius retrieval is performed through the KD-Tree.

[0062] Sampling Upper Limit Quota: Set an upper limit on the number of sampled instances in the instance set to prevent an excessive number of instances from a single H&E image from dominating subsequent differential retrieval results. Grid-based Sampling: Divide the H&E image space corresponding to the instance set into a uniform grid, and sample a fixed number of instances within each grid to ensure that the sampled instances are evenly distributed in space and avoid sampling concentration in certain specific areas. Prognostic Group Balancing: Perform balanced sampling on instances with different prognostic labels to keep the number of instances in the good-prognostic and poor-prognostic groups roughly consistent, avoiding the bias of imbalanced data on subsequent differential retrieval results. After sampling, the sampled instance subset is organized into a KD-Tree index data structure in the phenotypic embedding space to support efficient radius retrieval. KD-Tree is a binary tree data structure suitable for nearest neighbor queries in high-dimensional spaces, achieving fast retrieval by recursively partitioning the feature space. Radius retrieval is performed using KD-Tree, with a time complexity of O(log (Num)+m) for each query, where Num represents the number of instances in the instance subset and m represents the average number of instances in the scale-dependent neighborhood returned, ensuring computational efficiency on large-scale pathological image data.

[0063] S52: Use the phenotypic vector of each instance in the instance set at the specified level (actually a subset of instances obtained after sampling) as the query center. Here, the specified level is the cell level, microenvironment level, tissue level or tissue-cell level, and the phenotypic vector is the cell-level phenotypic vector, microenvironment-level phenotypic vector, tissue-level phenotypic vector or tissue-cell-level phenotypic vector.

[0064] S53: Perform radius retrieval in the phenotypic embedding space with the query center as the center, gradually increase the retrieval radius from the preset initial radius, dynamically expand the scale-dependent neighborhood, and calculate the logarithmic multiple change value between different prognostic groups in the scale-dependent neighborhood under each retrieval radius in real time.

[0065] In a specific example, multiple instances are randomly selected from the instance set at a specified level (actually a subset of instances obtained after sampling) as sampling instances; the average nearest neighbor distance from the phenotypic vector of each sampling instance to the phenotypic vectors of its nearest neighbors is calculated; a preset percentile of the average nearest neighbor distance of each sampling instance is taken as the local distance scale; the local distance scale of each sampling instance is compared with the value obtained by multiplying the global feature standard deviation by a first preset scale coefficient, and the larger value is taken as the minimum retrieval radius, i.e., the preset initial radius. Specifically, , where r min P represents the minimum search radius, i.e., the preset initial radius. η This represents the ηth percentile, i.e., the preset percentile. The average nearest neighbor distance of the i-th sampled instance is represented by the formula: N near This represents the number of nearest neighbor instances for the i-th sampled instance. The nearest neighbor instances for the i-th sampled instance can be determined as follows: Calculate the Euclidean distance from the phenotypic vector of the i-th sampled instance to the phenotypic vectors of all other instances in the instance set at the specified level. Sort all the obtained Euclidean distances in ascending order and select the top N. near x instances are considered as nearest neighbor instances. i Let x represent the phenotypic vector of the i-th sampled instance. i,j Indicates with x i The phenotypic vector of the instance whose Euclidean distance is closest to the j-th instance, i.e., the first N... near The phenotypic vector of the j-th instance in N instances, ‖·‖2 is used to calculate the L2 norm, N samp α represents the number of sampled instances. min σ represents the first preset scaling factor. global σ represents the global feature standard deviation of the instance set at a specified level (actually a subset of instances obtained after sampling). The global feature standard deviation is the standard deviation of the phenotypic vectors of all instances in the instance set at the specified level, representing the degree of dispersion of instances in the phenotypic embedding space.global A smaller σ value indicates a more concentrated distribution of instances in the phenotypic embedding space. global A larger value indicates a more uniform distribution of instances in the phenotypic embedding space. In a specific example, η = 10, N near =5, α min =0.05.

[0066] The basic idea of ​​dynamically expanding scale-dependent neighborhoods is to use the phenotypic vector of each instance in the phenotypic embedding space as the query center. Starting from a preset initial radius, the retrieval radius is gradually expanded. The prognostic groups of neighbors within the retrieval radius are statistically analyzed, and the logarithmic fold change value is calculated. When the retrieval radius expands to a certain critical value, instances of opposing prognostic groups begin to flood in. At this point, the rate of increase in the logarithmic fold change value tends to level off, the second derivative approaches zero, and expansion stops. The retrieval radius at which expansion stops is the adaptive optimal radius of the query center. This process is entirely data-driven and does not require predefined clustering or cell types. During the dynamic expansion process, as the retrieval radius grows, the corresponding logarithmic fold change value forms a continuous trajectory, representing how the differences between different prognostic groups accumulate with scale. The scale-dependent neighborhood terminates expansion at the data-driven boundary.

[0067] In a specific example, the scale-dependent neighborhood is defined as a spherical neighborhood centered at the query center q with a radius r. The distance metric is the Euclidean distance between the phenotypic vectors of the query center q and other instances in the phenotypic embedding space. Further, based on the prognostic label Y, instances within the scale-dependent neighborhood are divided into a good-prognostic group-specific subset N0(q) and a poor-prognostic group-specific subset N1(q) for comparative analysis between different prognostic groups. At different levels, N... g (q) is concretized at the cellular level. Microenvironment level Organizational hierarchy and tissue-cell level , g∈{0,1}.

[0068] Specifically, the calculation of logarithmic multiple changes includes: For a set of instances at the cellular level, both the logarithmic fold change in absolute abundance and the logarithmic fold change in relative density are calculated: , Where q represents the query center and r represents the search radius. This represents the logarithmic fold change in absolute abundance, reflecting changes in cell numbers within a scale-dependent neighborhood. Specifically, it indicates whether the phenotypic vector of a certain cell type is more enriched in a poorly prognostic population than in a well-prognostic population. This represents the logarithmic multiple change in relative density, reflecting changes in the degree of cell aggregation within a scale-dependent neighborhood. Specifically, it indicates whether a particular cell's phenotypic vector tends to form a denser cell community in a population with poor prognosis. This represents the set of all cell-level phenotypic vectors with a poor prognostic label within the scale-dependent neighborhood centered at the query center q. ε represents the set of all cell-level phenotypic vectors with a good prognostic label within the scale-dependent neighborhood centered at query center q, where ε is a small constant to prevent division by zero, such as ε = 1e-6. µ1(q) represents The average local density information corresponding to all cell-level phenotypic vectors in the data, µ0(q) represents... The average value of the local density information corresponding to all cell-level phenotypic vectors. , g∈{0,1}, d h express The local density information corresponding to the h-th cell-level phenotypic vector is used, where |·| represents the quantity; then, the logarithmic fold change value of absolute abundance and the logarithmic fold change value of relative density are fused into the logarithmic fold change value of the cell level, LogFC. cell (q,r), .

[0069] For a set of instances at the microenvironment level, the logarithmic multiple change value is calculated through kernel similarity aggregation: LogFC env (q,r) represents the logarithmic multiple change value at the microenvironment level, reflecting the differences in the local microenvironment composition patterns among different prognostic groups, and can capture functional state changes at the microenvironment level. S1 represents the kernel similarity weighted sum of the poor prognosis group, and S0 represents the kernel similarity weighted sum of the good prognosis group. , This represents the set of all microenvironment-level phenotypic vectors with a poor prognosis label within the scale-dependent neighborhood centered at the query center q. z represents the set of all micro-environmental phenotypic vectors with a good prognostic label within the scale-dependent neighborhood centered at the query center q. In the microenvironment-level phenotypic vector, κ(·,·) is a Gaussian kernel function used to measure the similarity between z and q.

[0070] For a set of instances at the organizational level, the logarithmic multiple change is calculated based on the ratio of the number of organizational-level phenotypic vectors in different prognostic populations: LogFC tis(q,r) represents the logarithmic multiple of the tissue level, reflecting the distributional differences among different prognostic populations in specific tissue states (such as the tumor core, invasion front, and mesenchymal region), and can capture the emergent characteristics at the tissue structure level. This represents the set of all organizational-level phenotypic vectors with a poor prognostic label within the scale-dependent neighborhood centered at the query center q. It represents the set of all organizational-level phenotypic vectors with good prognostic labels within the scale-dependent neighborhood centered on the query center q.

[0071] For a tissue-cell level instance set, considering the distribution differences of cells within their respective tissue regions, both the absolute abundance logarithmic fold change and the relative density logarithmic fold change are calculated. The calculation method is the same as for the cell level instance set, except that for the tissue-cell level instance set, q is the tissue-cell level phenotypic vector of cells within the tissue region. Then, the absolute abundance logarithmic fold change and the relative density logarithmic fold change are fused into a tissue-cell level logarithmic fold change value, LogFC. tis-cell (q,r) Tissue-cell hierarchy can reveal the prognostic association differences of the phenotypic vector of the same cell in different tissue locations. For example, the phenotypic vector of a certain cell may be associated with a poor prognosis group in the tumor core area, while it may be associated with a good prognosis group in the invasion front area, thus discovering prognostic biomarkers with spatial context dependence.

[0072] S54: Determine the candidate retrieval radius based on the changing trend of the logarithmic multiple value, and take the smaller value between the candidate retrieval radius and the preset maximum retrieval radius as the retrieval radius when terminating the expansion. The scale-dependent neighborhood under this retrieval radius is taken as the candidate difference region, specifically including: S541: Monitor the second derivative of the logarithmic multiple change value with respect to the search radius. Terminate the expansion of the scale-dependent neighborhood when the absolute value of the second derivative reaches an extreme point or the search radius reaches a preset maximum search radius. Determine the search radius at which the expansion terminates as the adaptive optimal radius. The determination of the adaptive optimal radius includes: calculating the second derivative ρ(q,r) of the logarithmic multiple change value with respect to the search radius, i.e., the curvature. Where LogFC(q,r) represents the logarithmic fold change between different prognostic populations within the scale-dependent neighborhood at a search radius r. For example, when the specified level is the cell level, LogFC(q,r) refers to LogFC. cell (q,r); During the scale-dependent neighborhood dynamic expansion process, the retrieval radius corresponding to the first time the second derivative approaches zero from a negative value is determined as the candidate retrieval radius r. be , ,in, Indicates taking such that The minimum value of r, where |·| is the absolute value; the smaller of the candidate search radius and the preset maximum search radius is used as the adaptive optimal radius. The preset maximum search radius is r. max r max =α max σ global α max This represents the second preset scaling factor, α in a specific example. max =0.5.

[0073] S542: Select the scale-dependent neighborhood of the query center under the adaptive optimal radius as the candidate difference region.

[0074] In a specific example, to ensure stable interpretation during dynamic expansion, the growth of logarithmic fold change values ​​is directionally processed. For each query center, the initial direction of difference is determined by the dominant prognostic population within the scale-dependent neighborhood, and logarithmic fold change values ​​are interpreted along this direction, while transient fluctuations in the opposite direction are ignored. This directionality criterion is entirely data-driven and independent of any predefined cell type or cluster label.

[0075] In a specific example, for each level (cell, microenvironment, tissue, tissue-cell), the scale-dependent neighborhoods of all query centers at their respective adaptive optimal radii constitute a set of candidate differential regions. The candidate differential regions in each level's set overlap, providing local evidence of prognostic differences. When subsequent screening reveals a candidate differential region exhibiting significant differences across different prognostic populations, it can be mapped back to the original H&E image using the spatial metadata associated with that candidate differential region for visual verification and clinical interpretation.

[0076] S6: Each query center forms a candidate difference region under its adaptive optimal radius. For H&E image datasets containing a large number of instances, the number of candidate difference regions may reach tens of thousands or even millions. Due to the continuity of adjacent instances in the phenotypic embedding space, a large number of candidate difference regions have high overlap and essentially point to the same biological difference event. Therefore, the candidate difference regions need to be further processed: the candidate difference regions are filtered, and the remaining candidate difference regions are output as phenotypic biomarkers. Specifically, the support-aware score of each candidate difference region at a specified level is calculated; redundancy is removed from all candidate difference regions based on the support-aware score to remove overlapping and redundant candidate difference regions; the remaining candidate difference regions are output as phenotypic biomarkers, which include hierarchical type, difference direction, difference intensity, and spatial location information.

[0077] In a specific example, the absolute value of the logarithmic multiple change of the support-perceived score and the candidate difference region is positively correlated with the size of the candidate difference region. The calculation formula is: Score(q) = |LogFC(q,r)| best )|·F mono (|N ei (q,r best )|), where Score(q) represents the value of the query center q within the adaptive optimal radius r. best Candidate difference region N ei (q,r best Support-perceived score, |LogFC(q,r) best F reflects the intensity of the difference. mono (|N ei (q,r best )|) reflects the robustness of differences, LogFC(q,r best ) represents N ei (q,r best The corresponding logarithmic multiple change value, |N ei (q,r best )| represents N ei (q,r best The size of N is to find the value of N. ei (q,r best Number of instances within F mono (·) denotes a monotonically saturated function, preferably a logarithmic function, i.e., F mono (|N ei (q,r best )|)=log(1+|N ei (q,r best )|).

[0078] In a specific example, nonmaximum suppression includes: All candidate regions of difference are sorted from highest to lowest based on their perceived support scores.

[0079] Select the candidate difference region with the highest perceived support score from the currently unprocessed candidate difference regions as the retained candidate difference regions.

[0080] Calculate the neighborhood overlap between the retained candidate difference regions and the remaining unprocessed candidate difference regions. When the neighborhood overlap exceeds a preset overlap threshold, suppress the candidate difference regions with lower support perception scores. The neighborhood overlap is the number of instances shared by two candidate difference regions, and the preset overlap threshold is 50% to 80% of the number of instances in the smaller region among the two candidate difference regions with neighborhood overlap.

[0081] Repeat the selection, calculation, and suppression steps described above until all candidate difference regions have been processed, resulting in all retained candidate difference regions.

[0082] In a specific example, phenotypic biomarkers contain the following information: Hierarchical types: cellular level, microenvironment level, tissue level, or tissue-cell level.

[0083] Differential direction: enrichment for poor prognosis (>0) or enrichment for good prognosis (<0).

[0084] Difference strength: Logarithmic multiple change value and its corresponding statistical significance. The statistical significance is obtained by permutation test. Specifically, the prognostic labels are randomly shuffled, the logarithmic multiple change value is repeatedly calculated, a null distribution is constructed, and the p-value is determined by the position of the actual logarithmic multiple change value in the null distribution. Preferably, the p-value of multiple candidate difference regions is corrected by FDR, and the q-value is used as the measure of statistical significance.

[0085] Spatial location: Spatial metadata corresponding to each instance.

[0086] Size: The number of instances included.

[0087] Cellular-level phenotypic biomarkers reflect intrinsic cellular phenotypic differences; microenvironment-level phenotypic biomarkers reflect differences in local cell interaction patterns; tissue-level phenotypic biomarkers reflect tissue structural differences; and tissue-cell-level phenotypic biomarkers reflect location-specific phenotypic variations. Each phenotypic biomarker corresponds to a coherent cellular / microenvironment / tissue region with consistent distributional differences in the phenotypic embedding space, which can be directly mapped to the spatial coordinates of the original H&E image for visualization and biological interpretation.

[0088] This invention employs a three-stage progressive optimization strategy for training and application.

[0089] Phase 1: Pre-training Phase. Using only a large number of unlabeled H&E images, a self-supervised contrastive learning pre-training (e.g., SimCLR) is performed on the backbone network of the H&E-specific cell encoder (preferably CTransPath) to enable the model to learn basic tissue morphological features. mIF data is not used in this phase.

[0090] Phase Two: Cross-Modal Alignment Phase. Using paired H&E patches and mIF patches, according to the joint loss L... totalThe dual-branch Transformer architecture is trained. During this stage, the parameters of the H&E branch (containing the H&E-specific cell encoder, fully connected layers, and the H&E Transformer module) are updated, while the parameters of the mIF Transformer module in the mIF branch are also updated. This aligns the H&E aggregation tokens with the mIF aggregation tokens, optimizes the alignment of the low-dimensional H&E cell embeddings with the optimized mIF cell embeddings, and ensures that the output of the H&E branch has protein semantic discrimination capabilities.

[0091] Phase 3: Phenotypic Biomarker Discovery. For the target patient cohort (multiple patients to be tested), only their H&E images are needed, not mIF images. The optimized low-dimensional H&E cell embeddings and H&E aggregation tokens for each H&E patch are extracted using a trained H&E branch. Then, the multi-level phenotypic spatial transformation operation (step S4), density-aware differentiable retrieval (step S5), and automated phenotypic biomarker screening (step S6) are performed sequentially. This phase is fully automated, requiring no manual annotation or intervention.

[0092] In addition, after obtaining the selected phenotypic biomarkers, their spatial metadata can be used to trace back to the original H&E image for statistical validation such as Kaplan-Meier survival analysis and multivariate Cox regression to further evaluate the prognostic value of the phenotypic biomarkers.

[0093] The following experimental verifications are all based on the method flow described in the above specific embodiments of the present invention, including two-branch Transformer architecture training, multi-level phenotypic space transformation, density-aware differentiable retrieval, and automatic phenotypic biomarker screening.

[0094] I. Experimental Dataset This experiment was validated on four independent cancer cohorts, including the Orion-CRC colorectal cancer private dataset, the TCGA-COAD colorectal cancer public dataset, the TCGA-READ rectal cancer public dataset, and the Private cholangiocarcinoma private dataset. All experiments evaluated the phenotypic biomarker discovery performance of the method described above on these four independent cancer cohorts. Each cancer cohort included patients' hematocrit (H&E) images and corresponding survival follow-up data.

[0095] Two- and four-layer phenotypic biomarkers comprehensively cover multi-scale phenotypic differences. Experimental results on four independent cancer cohorts demonstrate that the method of this invention can automatically identify robust phenotypic biomarkers at four levels (cellular, microenvironmental, tissue, and tissue-cell levels), with each biomarker significantly associated with survival outcomes (p-values ​​less than 0.05). These results demonstrate the strong clinical translatability and cross-cancer scalability of the method.

[0096] On the Orion-CRC colorectal cancer private dataset, the logarithmic fold change of the cell-level top markers was -0.69, and the p-value was 3.64 × 10⁻⁶. -5 The logarithmic fold change of tissue-cell level top markers was as high as +1.40, with a p-value of 5.94 × 10⁻⁶. -12 The above results indicate that the differences captured at different levels are complementary: the cellular level captures intrinsic phenotypic differences, while the tissue-cell level captures location-specific phenotypic variations, and together they provide multi-scale prognostic information.

[0097] III. Ability to detect localized and refined differences The density-aware differentiable retrieval method used in this invention is compared with the clustering-based diffcyt method. Experimental results show that the method of this invention can reveal localized fine phenotypic patterns masked by clustering methods, detect finer-grained, localized biomarkers, and expose protein pattern differences hidden by the clustering smoothing process of diffcyt.

[0098] Regarding the hierarchical coverage of differential testing, existing methods such as diffcyt only support cellular-level differential testing, scCODA supports cellular and microenvironment-level differential testing, and Squidpy supports cellular, microenvironment-level, and tissue-level differential testing. However, the method of this invention is the only one that can simultaneously support differential testing at four levels: cellular, microenvironment, tissue, and tissue-cellular. Existing methods cannot achieve full coverage of all four levels.

[0099] IV. Cross-Queue Independent Discovery Consistency Verification Validation on three independent datasets—Orion-CRC, TCGA-COAD, and TCGA-READ—shows that, despite independent mining on each dataset, the method of this invention identified biomarkers with consistent risk and protection patterns across different datasets. Consistent population segregation patterns were observed at the cellular, microenvironmental, and tissue levels; specifically, negative logarithmic fold changes corresponded to risk factors enriched in the poorly prognostic population, while positive logarithmic fold changes corresponded to protective factors enriched in the well-prognostic population.

[0100] The results demonstrate that the logarithmic multiple change value-driven discovery framework of the present invention is robust and reliable under dataset shifts, and that the discovered phenotypic biomarkers are not random statistical fluctuations, but rather genuine phenotypic differences with biological basis and cross-cohort reproducibility.

[0101] V. Computational Efficiency and Scalability The method of this invention employs a KD-Tree for radius retrieval, exhibiting excellent time and space complexity. Regarding time complexity, the pairwise comparison method has a time complexity of O(Num²×K), the full scan method has a time complexity of O(Q×Num×K), while the single-query time complexity of this invention is only O(log(Num)+m), where Q is the number of query centers. In terms of space complexity, the primary cost is only storing a Num×K embedding matrix, and all auxiliary structures scale linearly.

[0102] Ablation experiments show that there is a clear trade-off between the preset overlap threshold in non-maximum suppression and the retrieval radius increment step size in the adaptive optimal radius determination process regarding the sensitivity and computational efficiency of candidate difference region screening. Increasing the preset overlap threshold (i.e., a more stringent region screening criterion) results in a reduced number of retained candidate difference regions but higher region consistency, while decreasing the retrieval radius increment step size improves the retrieval precision but increases computation time. Under the optimal parameter settings, i.e., a retrieval radius increment of 50 steps and a preset overlap threshold of 50% of the smaller region instances in the candidate difference regions, the method of this invention retains approximately 23% of the candidate difference regions and achieves a 3.21-fold enrichment of prognostic label consistency within the candidate difference regions, while requiring only about 5 minutes of runtime.

[0103] VI. Fully automated to eliminate reliance on manual interpretation This invention achieves a fully automated process from image input to phenotypic biomarker output, completely eliminating the reliance on expert interpretation found in traditional methods. Specifically, it eliminates the need for manually setting thresholds, manually selecting regions of interest, and manually interpreting posterior data. By directly driving the discovery process through prognostic outcomes, combined with automated support-aware scoring, non-maximum suppression for redundancy removal, and spatial mapping refinement steps, the entire process is fully automated, ensuring the objectivity, repeatability, and large-scale scalability of the discovery.

Claims

1. A method for automatic discovery of phenotypic biomarkers in cross-modal pathological images, characterized in that, Includes the following steps: S1: Obtain paired H&E and mIF images of the same tissue slice, extract multiple H&E patches and mIF patches corresponding to the spatial location, and perform spatial registration on each patch pair; S2: Perform cell instance segmentation on the spatially registered H&E patch and mIF patch respectively to obtain the H&E instance segmentation mask of each cell in the H&E patch and the mIF instance segmentation mask of each cell in the mIF patch, and take the intersection of the two mask sets as the shared cell set. S3: Construct a dual-branch Transformer architecture, including an H&E branch and a mIF branch; extract the H&E cell embeddings of each cell in the shared cell set through the H&E branch, map them to a low-dimensional space, obtain the optimized low-dimensional H&E cell embeddings, and generate an H&E aggregation token; calculate the average fluorescence intensity of each cell in the shared cell set in each protein target channel of the mIF image through the mIF branch, form the initial mIF cell embeddings, obtain the optimized mIF cell embeddings, and generate an mIF aggregation token. With the optimization of low-dimensional H&E cell embedding and optimization of mIF cell embedding as alignment objectives, the parameters of a two-branch Transformer architecture are trained so that the embedding of the same cell in the two modalities tends to be consistent in the latent space. S4: After completing the cross-modal alignment training, the phenotypic biomarker discovery stage is entered: Based on the trained H&E branch, the optimized low-dimensional H&E cell embedding of each cell in each H&E patch of the H&E image of the patient to be tested and the H&E aggregation token of each H&E patch are obtained, so as to optimize the K-dimensional space in which the low-dimensional H&E cell embedding is located as the phenotypic embedding space, where K is the number of protein target channels in the mIF image; The optimized low-dimensional H&E cell embeddings are fused with local density information to construct cell-level phenotypic vectors. Using H&E patches as units, the optimized low-dimensional H&E cell embeddings of each cell are density-weighted and averaged according to normalized density fractions to obtain microenvironment-level phenotypic vectors. H&E aggregation tokens are used as tissue-level phenotypic vectors. Each tissue-level phenotypic vector corresponds to a tissue region. The optimized low-dimensional H&E cell embeddings of each cell within this tissue region are fused with local density information to construct tissue-cell-level phenotypic vectors for each cell. Cell-level phenotype vectors, microenvironment-level phenotype vectors, tissue-level phenotype vectors, and tissue-cell-level phenotype vectors are each organized into a set of instances with spatial metadata. S5: Use the phenotypic vector of each instance in the instance set at the specified level as the query center; Radius retrieval is performed in the phenotypic embedding space with the query center as the center. The retrieval radius is gradually increased from the preset initial radius, the scale-dependent neighborhood is dynamically expanded, and the logarithmic multiple change value between different prognostic groups in the scale-dependent neighborhood under each retrieval radius is calculated in real time. The candidate retrieval radius is determined based on the trend of the logarithmic multiple change value. The smaller value between the candidate retrieval radius and the preset maximum retrieval radius is taken as the retrieval radius when the expansion is terminated. The scale-dependent neighborhood under this retrieval radius is taken as the candidate difference region. S6: Filter the candidate differential regions and output the remaining candidate differential regions as phenotypic biomarkers.

2. The method for automatic discovery of cross-modal pathological image phenotypic biomarkers according to claim 1, characterized in that, Before the step of extracting multiple H&E patches and mIF patches corresponding to the spatial location, the method further includes: using the DAPI channel of the mIF image as a spatial reference, performing global rigid registration on the H&E image to achieve global spatial coarse alignment; The spatial registration of each patch pair includes: performing local non-rigid registration on each patch pair to optimize the local deformation within the patch pair and achieve finer local spatial alignment.

3. The method for automatic discovery of cross-modal pathological image phenotypic biomarkers according to claim 1, characterized in that, The H&E branch includes an H&E-specific cell encoder, a fully connected layer, and an H&E Transformer module. The H&E instance segmentation mask of each cell in the shared cell set is multiplied element-wise with the spatially registered H&E patch to obtain the corresponding cell image region. The cell image region of each cell in the shared cell set is input into the H&E-specific cell encoder to obtain the corresponding initial H&E cell embedding. The initial H&E cell embedding of each cell in the shared cell set is projected through the fully connected layer to obtain the corresponding initial low-dimensional H&E cell embedding. The initial low-dimensional H&E cell embedding of each cell in the shared cell set is input into the H&E Transformer module to obtain the optimized low-dimensional H&E cell embedding for each cell in the shared cell set, as well as the patch-level H&E aggregation token. The mIF branch includes a non-parametric embedding calculation module and an mIF Transformer module. The non-parametric embedding calculation module calculates the average fluorescence intensity of each cell in the shared cell set in each protein target channel of the mIF image to form the corresponding initial mIF cell embedding. The initial mIF cell embedding of each cell in the shared cell set is input into the mIFTransformer module to obtain the optimized mIF cell embedding of each cell in the shared cell set, as well as the patch-level mIF aggregation token.

4. The method for automatic discovery of cross-modal pathological image phenotypic biomarkers according to claim 3, characterized in that, The training employs a combination of patch-level contrastive loss and patch-level mean squared error loss. The acquisition of the patch-level contrast loss includes: For the spatially registered H&E patch and mIF patch, construct a positive sample pair by combining the H&E aggregation token corresponding to the current H&E patch and the mIF aggregation token corresponding to the current mIF patch; The H&E aggregate token corresponding to the current H&E patch and the mIF aggregate token corresponding to the mIF patch that does not correspond to its spatial location are used to form a negative sample pair; Calculate the patch-level contrast loss based on the cosine similarity between positive and negative sample pairs; The patch-level mean squared error loss is used to constrain the numerical similarity between the optimized low-dimensional H&E cell embedding and the optimized mIF cell embedding for the same cell.

5. The method for automatic discovery of cross-modal pathological image phenotypic biomarkers according to claim 1, characterized in that, The construction of the cell-level phenotypic vector includes: For each cell in the H&E patch, the sum of the attention weights of the optimized low-dimensional H&E cell embedding of that cell to the optimized low-dimensional H&E cell embeddings of other cells in the H&E patch is calculated as the local density information of that cell. The optimized low-dimensional H&E cell embedding of the cell is concatenated with the local density information to obtain the cell-level phenotypic vector of the cell. The construction of the microenvironment-level phenotypic vector includes: Using H&E patches as units, the corresponding normalized density fraction is calculated based on the local density information of each cell; The optimized low-dimensional H&E cell embeddings of each cell were density-weighted and averaged according to the normalized density fraction to obtain the microenvironment-level phenotypic vector.

6. The method for automatic discovery of cross-modal pathological image phenotypic biomarkers according to claim 1, characterized in that, Before the query center is determined, the following are included: Balanced sampling is performed on a set of instances at a specified level, wherein the balanced sampling includes at least one of the following methods: setting a sampling upper limit quota for the set of instances, grid-based sampling, and pre-group balancing; All instances in the sampled subset are organized into a KD-Tree in the phenotypic embedding space, and radius retrieval is performed through the KD-Tree.

7. The method for automatic discovery of cross-modal pathological image phenotypic biomarkers according to claim 1, characterized in that, The calculation of the logarithmic multiple change value includes: For a set of instances at the cellular level, both the logarithmic fold change in absolute abundance and the logarithmic fold change in relative density are calculated: , Where q represents the query center and r represents the search radius. This represents the logarithmic change in absolute abundance. This represents the logarithmic change in relative density. This represents the set of all cell-level phenotypic vectors with a poor prognostic label within the scale-dependent neighborhood centered at the query center q. ε represents the set of all cell-level phenotypic vectors with a good prognostic label within the scale-dependent neighborhood centered at query center q, where ε is a small constant to prevent division by zero, and µ1(q) represents... The average value of the local density information corresponding to all cell-level phenotypic vectors, µ0(q) represents The average value of the local density information corresponding to all cell-level phenotypic vectors; then the logarithmic fold change value of absolute abundance and the logarithmic fold change value of relative density are fused into the logarithmic fold change value at the cell level; For a set of instances at the microenvironment level, the logarithmic multiple change value is calculated through kernel similarity aggregation: LogFC env (q,r) represents the logarithmic multiple change value of the microenvironment level, S1 represents the kernel similarity weighted sum of the poor prognosis group, and S0 represents the kernel similarity weighted sum of the good prognosis group. , This represents the set of all microenvironment-level phenotypic vectors with a poor prognosis label within the scale-dependent neighborhood centered at the query center q. z represents the set of all micro-environmental phenotypic vectors with a good prognostic label within the scale-dependent neighborhood centered at the query center q. The microenvironment-level phenotypic vector in the vector, κ(·,·) is the Gaussian kernel function; For a set of instances at the organizational level, the logarithmic multiple change is calculated based on the ratio of the number of organizational-level phenotypic vectors in different prognostic populations: LogFC tis (q,r) represents the logarithmic multiple of the organizational hierarchy. This represents the set of all organizational-level phenotypic vectors with a poor prognostic label within the scale-dependent neighborhood centered at the query center q. It represents the set of all organizational-level phenotypic vectors with good prognostic labels within the scale-dependent neighborhood centered on the query center q; For a set of instances at the tissue-cell level, the distribution differences of cells within their respective tissue regions are considered, and both the absolute abundance logarithmic fold change and the relative density logarithmic fold change are calculated. These two values ​​are then fused into a single logarithmic fold change value (LogFC) for the tissue-cell level. tis-cell (q,r).

8. The method for automatic discovery of cross-modal pathological image phenotypic biomarkers according to claim 7, characterized in that, The process of determining the candidate retrieval radius based on the trend of logarithmic multiple changes, using the smaller of the candidate retrieval radius and the preset maximum retrieval radius as the retrieval radius at which expansion is terminated, and using the scale-dependent neighborhood under this retrieval radius as the candidate difference region includes: Monitor the second derivative of the logarithmic multiple change value with respect to the search radius. When the absolute value of the second derivative reaches the extreme point or the search radius reaches the preset maximum search radius, terminate the expansion of the scale-dependent neighborhood and determine the search radius at the time of termination as the adaptive optimal radius. The scale-dependent neighborhood of the query center under the adaptive optimal radius is used as the candidate difference region; The determination of the adaptive optimal radius includes: Calculate the second derivative ρ(q,r) of the logarithmic change with respect to the search radius. Where LogFC(q,r) represents the logarithmic multiple change between different prognostic groups within the neighborhood that depends on the scale of the retrieval radius r; During the scale-dependent neighborhood dynamic expansion process, the search radius corresponding to the first time the second derivative approaches zero from a negative value is determined as the candidate search radius; The smaller value between the candidate retrieval radius and the preset maximum retrieval radius is taken as the adaptive optimal radius.

9. The method for automatic discovery of cross-modal pathological image phenotypic biomarkers according to claim 8, characterized in that, The process of filtering candidate differential regions and outputting the retained candidate differential regions as phenotypic biomarkers includes: Calculate the perceived support score for each candidate region of difference at a specified level; All candidate difference regions are deredundant based on the support-perceived score to remove overlapping and redundant candidate difference regions. The retained candidate differential regions are output as phenotypic biomarkers, which include hierarchical type, differential direction, differential intensity, and spatial location information.

10. The method for automatic discovery of cross-modal pathological image phenotypic biomarkers according to claim 9, characterized in that, The support-perceived score calculation formula is: Score(q) = |LogFC(q,r)| best )|·F mono (|N ei (q,r best )|), where Score(q) represents the value of the query center q within the adaptive optimal radius r. best Candidate difference region N ei (q,r best Support-perceived score, LogFC(q,r) best ) represents N ei (q,r best The corresponding logarithmic multiple change value, |N ei (q,r best )| indicates finding N ei (q,r best Number of instances within F mono (·) represents a monotonically saturated function; The redundancy removal process is non-maximum suppression, including: All candidate regions of difference are sorted from highest to lowest according to the perceived support score; Select the candidate difference region with the highest support-perceived score from the currently unprocessed candidate difference regions as the retained candidate difference regions; Calculate the neighborhood overlap between the retained candidate difference regions and the remaining unprocessed candidate difference regions. When the neighborhood overlap exceeds a preset overlap threshold, suppress the candidate difference regions with lower support perception scores. The neighborhood overlap is the number of instances shared by two candidate difference regions. Repeat the selection, calculation, and suppression steps described above until all candidate difference regions have been processed, resulting in all retained candidate difference regions.