Pathological image cross-modality cell alignment method under non-alignable scenario

Abstract

This invention discloses a cross-modal cell alignment method for pathological images in non-registerable scenarios. The method acquires H&E and mIF stained images and their data-augmented views corresponding to spatial anatomical locations; constructs H&E modality processing branches, mIF modality processing branches, and a reverse self-attention module; and sequentially performs three-stage progressive training: intra-modal contrastive pre-training based on the data-augmented view; using mIF modality-level semantic aggregation tokens from the same spatial anatomical location as positive samples and mIF modality-level semantic aggregation tokens from different spatial anatomical locations as negative samples, achieving H&E to mIF modality-level semantic alignment through contrastive learning; inputting the aligned H&E modality-level semantic aggregation tokens and the projected H&E cell feature embedding set into the reverse self-attention module, and achieving unsupervised cell-level semantic alignment by minimizing the Hungarian matching loss; finally, the output H&E cell features semantically aligned with mIF are embedded into a refined set, which can be directly used for downstream pathological analysis tasks such as high-precision cell clustering.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing, computer vision, and artificial intelligence, specifically to a deep learning-based method for pathological image analysis. More specifically, this invention relates to a cross-modal cell alignment method for pathological images in non-registerable scenarios. It achieves cell-level semantic alignment between hematoxylin-eosin (H&E) stained images and multiplex immunofluorescence (mIF) stained images without manual annotation through progressive modal bridging and cell matching. The resulting aligned cell features can be directly used for high-precision unsupervised cell clustering, providing key technical support for downstream pathological tasks such as cell type identification and tissue microenvironment analysis. Background Technology

[0002] In clinical pathological diagnosis and tumor microenvironment studies, it is usually necessary to perform hematoxylin-eosin (H&E) staining and multiplex immunofluorescence (mIF) staining on adjacent physical sections (thickness 3~10μm) of the same biological tissue sample, and then digitally scan them to obtain H&E staining images and mIF staining images.

[0003] However, the two types of images differ fundamentally in their information modality: the H&E staining image is a color image based on color synthesis to reflect tissue morphological characteristics; while the mIF staining image is a multi-channel image, with each channel independently and quantitatively corresponding to a specific protein marker or the fluorescent expression signal of the cell nucleus. To integrate the advantages of the two modalities, it is urgent to transfer the semantic information (i.e., cell type) directly defined by protein markers in the mIF staining image to the corresponding H&E staining image. This process is called cross-modal cell alignment.

[0004] Currently, the core challenge in achieving cross-modal cell alignment lies in non-registration scenarios: due to the inherent physical differences and tissue deformations between adjacent physical slices, there is no precise pixel-level spatial correspondence between H&E and mIF stained images. Under this constraint, existing technical solutions all have fundamental flaws: 1. Image registration-based methods: The core principle of this type of method lies in achieving pixel-level image alignment through spatial transformation algorithms. However, this principle fundamentally contradicts the physical reality that adjacent physical slices cannot be registered, making it difficult to stably achieve effective matching at the cell level, and any information transfer based on this is unreliable.

[0005] 2. Cell-based identification methods: These methods largely rely on large-scale, costly manual cell annotation of H&E-stained images to train supervised learning models. This leads to inherent limitations such as high annotation costs, significant subjective bias, and weak model generalization ability. Some unsupervised cell identification methods reduce the dependence on annotation, but still lack the utilization of the biological semantic information of protein marker definitions in mIF-stained images. Technically, they fail to achieve effective association with mIF and cannot achieve true cross-modal knowledge transfer.

[0006] In summary, under real-world conditions where registration is impossible and there is no spatial correspondence between cells, current technologies cannot effectively achieve accurate and automated cross-modal cell semantic alignment without annotation. This has become a key technological bottleneck hindering the effective integration of H&E morphological information and mIF molecular information, and thus restricting the large-scale clinical application of precision pathological analysis. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a cross-modal cell alignment method for pathological images in non-registration scenarios. This method can achieve unsupervised knowledge transfer from mIF to H&E by utilizing the protein semantic information of mIF stained images and progressively aligning with cells without pixel-level registration and cell annotation.

[0008] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a method for cross-modal cell alignment of pathological images in non-registration scenarios, comprising the following steps: Obtain H&E staining and mIF staining images of the same spatial anatomical location from adjacent physical slices; A H&E modality processing branch, a mIF modality processing branch, and a reverse self-attention module are constructed. The H&E modality processing branch is used to generate an H&E cell feature embedding set and an H&E modality-level semantic aggregation token. The mIF modality processing branch is used to generate an mIF cell feature embedding set and an mIF modality-level semantic aggregation token. The mIF modality processing branch includes a predefined non-parametric computation process to obtain the mIF cell feature embedding set. Based on the data augmented view of the H&E staining image and the data augmented view of the mIF staining image, self-supervised pre-training is performed on the H&E modality processing branch and the mIF modality processing branch respectively; After pre-training, the H&E staining image and the mIF staining image are input into the H&E modality processing branch and the mIF modality processing branch, respectively, and the H&E cell feature embedding set and the H&E modality-level semantic aggregation token, as well as the mIF cell feature embedding set and the mIF modality-level semantic aggregation token, are output, respectively. Using the mIF modality-level semantic aggregation token as the alignment target, the parameters of the H&E modality processing branch are updated through contrastive learning, while keeping the parameters of the mIF modality processing branch unchanged, so that the semantic space of the H&E modality-level semantic aggregation token is aligned with that of the mIF modality-level semantic aggregation token. The aligned H&E modality-level semantic aggregation token and the projected H&E cell feature embedding set are input into the inverse self-attention module to obtain the H&E cell feature embedding refinement set. Based on the H&E cell feature embedding refinement set and the mIF cell feature embedding set, the parameters of the inverse self-attention module and the H&E modality processing branch are updated by minimizing the Hungarian matching loss to achieve cell-level semantic alignment. Using the trained H&E modality processing branch and the reverse self-attention module, the H&E staining image to be inferred is processed to obtain a refined set of H&E cell feature embeddings aligned with the mIF modality semantics, for use in downstream pathological analysis tasks.

[0009] The H&E staining images and the mIF staining images are image patches corresponding to spatial anatomical locations. By explicitly targeting image patches corresponding to spatial anatomical locations as the processing objects, the consistency of the input data at the macroscopic anatomical level is ensured. This provides a reliable foundation for constructing positive sample pairs for subsequent modality-level contrastive learning, thereby improving the spatial rationality and repeatability of the entire method from the data source.

[0010] The H&E modality processing branch includes an H&E cell embedding extractor and a first attention aggregation module, while the mIF modality processing branch includes a feature computation unit for performing the predefined non-parametric computation process and a second attention aggregation module.

[0011] The H&E cell embedding extractor generates the H&E cell feature embedding set in the following manner: Cell instance segmentation is performed on the first processed image to obtain multiple cell regions; each cell region is sequentially input into a Transformer encoder to extract the corresponding cell feature embedding; the H&E cell feature embedding set is composed of the cell feature embeddings of all cells in the first processed image; wherein, the first processed image is the H&E staining image or a data augmented view of the H&E staining image.

[0012] The feature calculation unit obtains the mIF cell feature embedding set in the following manner: Cell nucleus instance segmentation is performed on the DAPI channel of the second processed image to obtain multiple cell nucleus regions; for each of the cell nucleus regions, the following operations are performed: a) Calculate the average signal intensity of the cell nucleus region on the DAPI channel; b) Based on the location of the cell nucleus region, the corresponding cell regions are determined in the K protein marker channels of the second processed image, and the average signal intensity of each cell region in its respective protein marker channel is calculated. The mIF cell feature embedding set is constructed from the K+1 dimensional signal intensity vectors obtained by steps a) and b) corresponding to all cell nucleus regions in the DAPI channel of the second processed image; wherein, the second processed image is the mIF stained image or a data augmented view of the mIF stained image.

[0013] Both the first attention aggregation module and the second attention aggregation module are Transformer encoders. The Transformer encoders generate corresponding modality-level semantic aggregation tokens through the following attention pooling method: A learnable aggregation token is randomly initialized, and the aggregation token is interacted with the input cell feature embedding set through self-attention and cross-attention. The aggregate token after the interaction is output as the corresponding modality-level semantic aggregation token.

[0014] The H&E modality processing branch further includes a fully connected layer located between the H&E cell embedding extractor and the first attention aggregation module. This layer projects the H&E cell feature embedding set onto a dimensional space that matches the mIF cell feature embedding set, thereby obtaining the projected H&E cell feature embedding set. This achieves an explicit and learnable dimensional mapping from the H&E feature space to the mIF feature space. This design significantly reduces the optimization difficulty of subsequent contrastive learning and feature matching, improves training stability and final alignment accuracy, and is key to enhancing the overall system performance.

[0015] The self-supervised pre-training refers to performing intra-modal contrastive learning on the H&E modality processing branch and the mIF modality processing branch respectively: For each modality processing branch, the corresponding modality-level semantic aggregation tokens generated by two different data augmented views from the same stained image after passing through the corresponding modality processing branch constitute a positive sample pair; any modality-level semantic aggregation token in the positive sample pair is combined with the corresponding modality-level semantic aggregation token generated by a data augmented view from a stained image from a different spatial anatomical location within the same modality after passing through the corresponding modality processing branch to form a negative sample pair; optimization is performed by minimizing the contrastive loss to maximize the consistency between the two modality-level semantic aggregation tokens in the positive sample pair, while minimizing the consistency between the two modality-level semantic aggregation tokens in the negative sample pair.

[0016] The step of updating the parameters of the H&E modality processing branch through contrastive learning while keeping the parameters of the mIF modality processing branch unchanged, thereby aligning the semantic space of the H&E modality-level semantic aggregation token with that of the mIF modality-level semantic aggregation token, is specifically achieved through the following cross-modal contrastive learning method: Positive sample pairs are formed by generating H&E modality-level semantic aggregation tokens and mIF modality-level semantic aggregation tokens from H&E staining images and mIF staining images originating from the same spatial anatomical location, respectively, after passing through the H&E modality processing branch and the mIF modality processing branch. The H&E modal semantic aggregation tokens in the positive sample pairs, together with the mIF modal semantic aggregation tokens generated after the mIF staining images from different spatial anatomical locations are processed by the mIF modal processing branch, constitute a negative sample pair set. Optimization is achieved by minimizing the contrast loss to bring the two modality-level semantic aggregate tokens in the positive sample pair closer in semantic space, and to increase the distance between the H&E modality-level semantic aggregate token and each mIF modality-level semantic aggregate token in the negative sample pair set in semantic space. During this stage, all parameters of the mIF modality processing branch are fixed, and only the parameters of the H&E modality processing branch are updated, thereby aligning the semantic space of the H&E modality-level semantic aggregate token to the semantic space of the mIF modality-level semantic aggregate token to obtain the aligned H&E modality-level semantic aggregate token.

[0017] The step of updating the parameters of the inverse self-attention module and the H&E modality processing branch by minimizing the Hungarian matching loss is specifically achieved through the following unsupervised matching method: Under cell-free conditions, the H&E cell feature embedding refinement set and the mIF cell feature embedding set are regarded as two sets to be matched; The Hungarian algorithm is used to automatically find an optimal bipartite graph matching between two sets to be matched, so as to minimize the overall matching cost. Based on the optimal matching result obtained by the Hungarian algorithm, the Hungarian matching loss is calculated, and optimization is performed by minimizing the loss, which forces the matched H&E cell feature embeddings and mIF cell feature embeddings to be close to each other in the feature space, thereby achieving cell-level semantic alignment.

[0018] Compared with the prior art, the advantages of the present invention are as follows: First, by using data augmented views of H&E-stained images and mIF-stained images, self-supervised pre-training is performed on the H&E modality processing branch and the mIF modality processing branch, respectively. This enables the model to provide robust initialization for cell feature embedding extraction of both modalities using the image's own information without any manual annotation. This fundamentally solves the inherent limitations of existing cell recognition-based methods, such as high annotation costs and large subjective biases, and lays a solid intra-modal feature foundation for subsequent cross-modal unsupervised alignment.

[0019] Second, by using the mIF modality-level semantic aggregation token as the alignment target, the parameters of the H&E modality processing branch are updated through contrastive learning while keeping the parameters of the mIF modality processing branch unchanged. This aligns the semantic spaces of the H&E modality-level semantic aggregation token with those of the mIF modality-level semantic aggregation token, enabling the model to effectively converge the global semantic representation of H&E staining images to the mIF semantic space explicitly defined by protein markers without relying on any pixel-level spatial correspondences. This innovative design avoids the failure of existing image registration methods due to physical infeasibility, achieves cross-modal semantic bridging, and realizes the goal of knowledge transfer using mIF staining image information.

[0020] Third, by inputting the aligned H&E modality-level semantic aggregation token and the projected H&E cell feature embedding set into the inverse self-attention module, and updating the parameters of the inverse self-attention module and the H&E modality processing branch by minimizing the Hungarian matching loss based on the refined H&E cell feature embedding set and the mIF cell feature embedding set, cell-level semantic alignment is achieved. This allows the method of the present invention to further utilize the optimal matching algorithm on the basis of global semantic alignment to achieve fine-grained cell-level feature alignment under the condition of no cell correspondence annotation. This completely solves the core challenge of achieving automated alignment under real-world conditions of non-registration and no cell spatial correspondence, and refines cross-modal knowledge transfer from the overall image level to the level of individual cell instances, thus fully realizing progressive alignment.

[0021] Fourth, the H&E cell features output by the method of the present invention, which are aligned with the mIF modality semantics, are embedded into a refined set, so that each cell in the H&E staining image obtains a feature representation aligned with the mIF protein semantic information. This feature can be directly used for high-precision downstream tasks (such as cell clustering and type recognition). Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the overall implementation of the method of the present invention. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0024] This invention aims to solve the problem of cross-modal cell alignment in pathological images under non-registration scenarios. The core idea of ​​this method is to use a two-stage framework of "modal bridging-cell alignment" to transfer the precise cellular semantic information defined by protein markers in multiplex immunofluorescence (mIF) stained images to the cellular feature embedding in hematoxylin and eosin (H&E) stained images without any need for manual cell annotation or pixel-level image registration.

[0025] The core of this invention lies in proposing a novel cross-modal cell alignment paradigm applicable to "registration-free scenarios." In adjacent physical slices, although the tissue structure is similar, due to differences in staining and cell distribution in three-dimensional space, precise instance-level registration between H&E and mIF stained images cannot be achieved at the single-cell level. Therefore, this invention completely abandons attempts at pixel-level or region-level image registration, instead pursuing a higher level of semantic consistency based on cell type distribution. The entire method does not rely on any form of pixel-level annotation or paired cell annotation, requiring only adjacent H&E and mIF slice pairs prepared conventionally in histology. This brings significant efficiency and practical advantages: it completely avoids the time-consuming, laborious, and highly subjective whole-slice cell-level annotation work, and eliminates the need for complex, potentially error-inducing, multi-stage image registration processes.

[0026] This invention proposes a method for cross-modal cell alignment of pathological images in non-registration scenarios, such as... Figure 1 As shown, it includes the following steps: S1. Data preparation and preprocessing to obtain H&E staining images and mIF staining images from the same spatial anatomical location derived from adjacent physical slices. Here, the H&E staining images and mIF staining images are image patches corresponding to the spatial anatomical location.

[0027] In a specific embodiment, the data preparation and preprocessing process is as follows: S11. Data preparation: Obtain the original H&E staining images and original mIF staining images from adjacent physical sections of the same biological tissue sample, convert them to grayscale images and perform smoothing.

[0028] As a specific example, the smoothness level for the smoothing process after converting to grayscale is set to 1.5, and the smoothing method is selected as Gaussian filtering.

[0029] S12. Preprocessing: To overcome the inherent physical misalignment between the original H&E staining image and the original mIF staining image, a two-stage progressive registration preprocessing is used to obtain image patch pairs corresponding to the anatomical locations. The specific process is as follows: Global rigid registration (coarse alignment): Using the DAPI (nuclear staining) channel of the smoothed mIF staining image as a spatial reference, rigid registration is performed on the smoothed H&E staining image to achieve global spatial coarse alignment. In a specific example, rigid registration is implemented using Mattes mutual information, which is used as the similarity measure (with the histogram bin number set to 50 and 1% of pixels randomly sampled). Optimization is achieved using a conventional step-size gradient descent method (learning rate 1.5, 150 iterations). Transformation initialization uses anisotropic scaling (horizontal scaling by a factor of 1.05) and center translation to estimate the optimal translation and rotation parameters, achieving global spatial coarse alignment. After coarse alignment of the DAPI channel of the smoothed H&E staining image with that of the smoothed mIF staining image, spatial alignment with other protein marker channels is also achieved.

[0030] Image patch cropping: The two colored images after global spatial coarse alignment are cropped into multiple non-overlapping image patches, thus forming multiple pairs of H&E-mIF image patch pairs that are initially spatially corresponding.

[0031] Local non-rigid registration (fine alignment): Non-rigid registration is performed on each pair of H&E-mIF image patches to optimize local deformation within the H&E-mIF image patch pair, achieving finer local spatial alignment. In a specific example, non-rigid registration is implemented using the Demons algorithm (80 iterations, deformation field smoothness of 1.0). Local spatial fine alignment further improves spatial consistency, ensuring that H&E-mIF image patch pairs originate from the same anatomical region (e.g., the same tumor microenvironment region). After this fine alignment step, each pair of H&E-mIF image patches obtained after local spatial fine alignment is highly consistent in spatial anatomical location and can be used as input for subsequent processing, i.e., "H&E stained images and mIF stained images at the same spatial anatomical location".

[0032] S2. Model Architecture: The core model includes the H&E modality processing branch, the mIF modality processing branch, and the inverse self-attention module.

[0033] This invention creatively distinguishes between two levels of tasks: "inter-modal semantic alignment" and "inter-cell instance alignment," and adopts a divide-and-conquer strategy. First, the model architecture design does not attempt to directly align individual cells that cannot correspond, but instead utilizes the biological prior that adjacent physical slices have similar cell types to design an efficient semantic aggregation and alignment method.

[0034] The H&E modality processing branch is responsible for generating an H&E cell feature embedding set based on the H&E stained image or its data augmented view, and generating an H&E modality-level semantic aggregation token based on the H&E cell feature embedding set. The mIF modality processing branch is responsible for generating an mIF cell feature embedding set based on the mIF stained image or its data augmented view, and generating an mIF modality-level semantic aggregation token based on the mIF cell feature embedding set.

[0035] The H&E modal processing branch includes, in sequence: An H&E cell embedding extractor is used to segment and encode each cell in an H&E-stained image or its data-augmented view into a feature vector. In one embodiment, the H&E cell embedding extractor is a Transformer encoder. In some embodiments, the H&E cell embedding extractor generates a set of H&E cell feature embeddings by: segmenting the H&E-stained image or its data-augmented view into cell instances to obtain multiple cell regions; sequentially inputting each cell region into a Transformer encoder to extract the corresponding cell feature embeddings; and constructing the H&E cell feature embedding set from the cell feature embeddings of all cells in the H&E-stained image or its data-augmented view.

[0036] The fully connected layer, located after the H&E cell embedding extractor, is used to project the H&E cell feature embedding set onto a dimensional space that matches the mIF cell feature embedding set, so as to obtain the projected H&E cell feature embedding set.

[0037] The first attention aggregation module, located after the fully connected layer, inputs the projected H&E cell feature embedding set into the first attention aggregation module. It is used to aggregate the projected H&E cell feature embedding set into a global semantic representation representing the entire image patch, namely the H&E modality-level semantic aggregation token.

[0038] The mIF modal processing branches include, in sequence: The feature calculation unit is a unit for performing a predefined non-parametric calculation process and does not contain trainable parameters. It directly calculates the biomarker expression vector for each cell based on the fluorescence channel intensity of the mIF staining image or its data-enhanced view, obtaining an mIF cell feature embedding set. In some embodiments, the feature calculation unit obtains the mIF cell feature embedding set by: segmenting the DAPI channels of the mIF staining image or its data-enhanced view into cell nucleus instances to obtain multiple cell nucleus regions; for each cell nucleus region, performing the following operations: a) calculating the average signal intensity of the cell nucleus region on the DAPI channels; b) based on the location of the cell nucleus region, determining the corresponding cell region on each of the K protein marker channels of the mIF staining image or its data-enhanced view, and calculating the average signal intensity of each cell region on its respective protein marker channel; the K+1 dimensional signal intensity vectors corresponding to all cell nucleus regions in the DAPI channels of the mIF staining image or its data-enhanced view, obtained from steps a) and b), constitute the mIF cell feature embedding set.

[0039] In step b) above, based on the location of the nuclear region, the protein marker channels are expanded to the corresponding cellular regions (e.g., through fixed-radius expansion or morphological manipulation), and then the average signal intensity of each protein marker channel within the cellular region is calculated to better capture protein expression levels. This expansion operation aims to locate the entire cellular region, including the cytoplasm, based on the nuclear region, ensuring that the calculated average signal intensity more accurately reflects the cell's expression level of the corresponding protein marker.

[0040] The second attention aggregation module, located after the feature calculation unit, inputs the mIF cell feature embedding set into the second attention aggregation module to aggregate the mIF cell feature embedding set into a global semantic representation representing the entire image patch, namely the mIF modal-level semantic aggregation token.

[0041] In some embodiments, both the first attention aggregation module and the second attention aggregation module are Transformer encoders. The Transformer encoder generates the corresponding modal-level semantic aggregation token through the following attention pooling method: randomly initialize a learnable aggregation token, and perform self-attention and cross-attention interaction between the aggregation token and the input cell feature embedding set, and output the aggregate token after interaction as the corresponding modal-level semantic aggregation token; wherein, the input cell feature embedding set is the projected H&E cell feature embedding set or the mIF cell feature embedding set, and the corresponding modal-level semantic aggregation token is the H&E modal-level semantic aggregation token or the mIF modal-level semantic aggregation token.

[0042] The reverse self-attention module is a separate module used to inject global semantic information back into each H&E cell feature embedding in a subsequent stage for refinement.

[0043] In specific implementations, cell instance segmentation of H&E staining images and their data-augmented views, as well as nuclear instance segmentation of the DAPI channel of mIF staining images and their data-augmented views, can both be achieved using modality-specific segmentation tools, such as the CellPose tool. Regarding the dimensions of cell feature embeddings: the length (i.e., feature dimension) of a single cell feature embedding extracted from an H&E staining image or its data-augmented view is determined by the structure of the Transformer encoder; the single cell feature embedding extracted from an mIF staining image or its data-augmented view is a calculated signal intensity vector with a fixed length of (K+1) dimensions, where 1 dimension corresponds to the average signal intensity on the DAPI (nuclear staining) channel, and K dimensions correspond to the average signal intensity on the K protein marker channels. Regarding the generation principle of single-cell feature embedding in mIF staining images: mIF staining images contain one DAPI channel and K protein marker channels. These channels are spatially aligned and correspond to different staining markers of the same batch of cells. Specifically, the DAPI channel stains the nuclei of all cells. Therefore, in this embodiment, the nucleus region obtained by segmenting the nucleus instances of the DAPI channel is used as the localization reference for each cell in the entire mIF staining image. Based on this reference, the corresponding cell regions are determined on each protein marker channel and the average signal intensity is calculated, thereby generating a unified (K+1)-dimensional feature representation for each cell. This process is entirely based on biophysical signals and requires no learning.

[0044] The training process is divided into three phases, performed sequentially, to achieve stable and efficient alignment. Although the process is divided into two phases (modal bridging and cell alignment), the entire model achieves end-to-end trainability through this progressive optimization strategy, ensuring the stability of training and the quality of the final cell feature embedding.

[0045] S3. The first stage, self-supervised pre-training, refers to performing intra-modal contrastive learning on the H&E modality processing branch and the mIF modality processing branch, respectively. This stage involves independently pre-training the H&E and mIF modality processing branches to learn robust feature representations within their respective modalities. Two different data augmentation views (e.g., through cropping, color dithering, etc.) are generated for the H&E and mIF stained images, respectively.

[0046] For the H&E modality processing branch, positive sample pairs are formed by generating H&E modality-level semantic aggregation tokens from two different data augmented views of the current H&E staining image after processing them through the H&E modality processing branch. A negative sample pair is formed by generating H&E modality-level semantic aggregation tokens from any positive sample pair and a data augmented view from an H&E staining image originating from a different spatial anatomical location (i.e., a different spatial anatomical location than the current H&E staining image) after processing it through the H&E modality processing branch. Optimization is performed by minimizing the contrast loss to maximize the consistency between the two H&E modality-level semantic aggregation tokens in the positive sample pair while minimizing the consistency between the two H&E modality-level semantic aggregation tokens in the negative sample pair.

[0047] Similarly, for the mIF modality processing branch, the mIF modality-level semantic aggregation tokens generated after processing the two different data augmentation views of the current mIF stained image through the mIF modality processing branch constitute a positive sample pair; any mIF modality-level semantic aggregation token in the positive sample pair is combined with the mIF modality-level semantic aggregation token generated after processing the data augmentation view of a data augmentation view from a different spatial anatomical location (i.e., a different spatial anatomical location than the current mIF stained image) through the mIF modality processing branch to form a negative sample pair; optimization is performed by minimizing the contrast loss to maximize the consistency between the two mIF modality-level semantic aggregation tokens in the positive sample pair, while minimizing the consistency between the two mIF modality-level semantic aggregation tokens in the negative sample pair.

[0048] S4. The second stage, modal bridging, uses contrastive learning-driven modal-level representation alignment. The goal of this stage is to achieve global semantic alignment across modalities, addressing the cross-modal semantic gap problem.

[0049] After pre-training, H&E staining images and mIF staining images are input into the H&E modality processing branch and the mIF modality processing branch, respectively, and output H&E cell feature embedding set and H&E modality-level semantic aggregation token, as well as mIF cell feature embedding set and mIF modality-level semantic aggregation token. Using the mIF modality-level semantic aggregation token as the alignment target, the parameters of the H&E modality processing branch are updated through contrastive learning, while keeping the parameters of the mIF modality processing branch (including non-parametric feature calculation units and the trained second attention aggregation module) unchanged, so that the semantic space of the H&E modality-level semantic aggregation token is aligned with that of the mIF modality-level semantic aggregation token, so that the aligned H&E modality-level semantic aggregation token has similar sparsity and discriminability to the mIF modality-level semantic aggregation token.

[0050] In some embodiments, the parameters of the H&E modality processing branch are updated through contrastive learning while keeping the parameters of the mIF modality processing branch unchanged, thereby aligning the semantic spaces of the H&E modality-level semantic aggregation token and the mIF modality-level semantic aggregation token. This is specifically achieved through the following cross-modality contrastive learning method: 1. Positive sample pair construction: A positive sample pair is formed by generating H&E modality-level semantic aggregation tokens and mIF modality-level semantic aggregation tokens from H&E stained images and mIF stained images (i.e., a pair of H&E-mIF image blocks from the same spatial anatomical location) after passing through the H&E modality processing branch and the mIF modality processing branch, respectively.

[0051] 2. Negative Sample Pair Construction: From the mIF modality-level semantic aggregation tokens generated after mIF modality processing branches of mIF stained images (i.e., different image patches) at other different spatial anatomical locations, T tokens (e.g., T=2048) are sampled as negative samples. These negative samples, together with the current H&E modality-level semantic aggregation tokens (i.e., the H&E modality-level semantic aggregation tokens in the current positive sample pair), constitute T negative sample pairs, forming a set of negative sample pairs. It should be noted that the negative sample pairs here do not include those using the mIF modality-level semantic aggregation tokens from the current positive sample pair as negative samples.

[0052] 3. Optimization Objective: Optimization is achieved by minimizing the contrastive loss (such as InfoNCE loss). This loss function brings the two modal-level semantic aggregate tokens in the positive sample pair closer in the semantic space, and widens the distance between the H&E modal-level semantic aggregate tokens and each mIF modal-level semantic aggregate token in the negative sample pair set. In this stage, all parameters of the mIF modal processing branch are fixed, and only the parameters of the H&E modal processing branch are updated, thereby aligning the semantic space of the H&E modal-level semantic aggregate tokens with the semantic space of the mIF modal-level semantic aggregate tokens to obtain the aligned H&E modal-level semantic aggregate tokens.

[0053] Experiments show that this invention achieves significant improvements in cell type alignment and recognition accuracy through this stage of alignment. On multiple benchmark datasets, including HCC5mIF, HCC7mIF, and Kidney7mIF, its core metrics, such as Dice coefficient, mean pixel accuracy (mPA), and mean intersection-over-union ratio (mIoU), are comprehensively superior. Compared to the strongest unsupervised baseline, the relative improvement in mIoU exceeds 5% on HCC5mIF, and the relative improvement in mPA exceeds 20% on HCC7mIF. More importantly, the unsupervised method of this invention even surpasses fully supervised baseline models that require extensive manual annotation, demonstrating that protein biomarker information transferred from mIF is more discriminative and generalizable than manual morphological annotation. Furthermore, this invention exhibits excellent cross-tissue and cross-tumor type generalization capabilities. On a large external test set containing 3,915 image patches of 19 different tissue types, it achieved an excellent average Dice coefficient of 80.34% and mPA of 94.94% without any task-specific fine-tuning, demonstrating that it captures the essential relationship between cell type and morphology and protein expression.

[0054] S5. The third stage, cell alignment, is fine-grained matching driven by Hungarian matching loss. This stage achieves fine-grained matching at the cell level based on global alignment.

[0055] Following the second-stage modal bridging, the semantic space of the H&E modal-level semantic aggregation tokens has been aligned with that of the mIF modal-level semantic aggregation tokens. Building upon this, this stage aims to further refine this global, image patch-level semantic alignment information and inject it into the feature representation of each H&E cell. This aligns the refined set of H&E cell feature embeddings with the set of mIF cell feature embeddings in the feature space, thus laying the foundation for cell-level unsupervised matching.

[0056] The aligned H&E modality-level semantic aggregation token and the projected H&E cell feature embedding set are input into the inverse self-attention module to obtain the H&E cell feature embedding refinement set. Based on the H&E cell feature embedding refinement set and the mIF cell feature embedding set, the parameters of the inverse self-attention module and the H&E modality processing branch are updated by minimizing the Hungarian matching loss, while keeping the parameters of the mIF modality processing branch unchanged, thus achieving cell-level semantic alignment.

[0057] In some embodiments, the parameters of the reverse self-attention module are updated by minimizing the Hungarian matching loss, specifically through the following unsupervised matching method: Under the condition of no cell annotation, the refined set of H&E cell feature embeddings and the set of mIF cell feature embeddings are regarded as two sets to be matched; the Hungarian algorithm is used to automatically find an optimal bipartite graph match between the two sets to be matched, so as to minimize the overall matching cost; based on the optimal matching result obtained by the Hungarian algorithm, the Hungarian matching loss is calculated, and optimization is performed by minimizing the loss, forcing the matched H&E cell feature embeddings and mIF cell feature embeddings to be close to each other in the feature space, thereby achieving cell-level semantic alignment.

[0058] In a specific implementation, the aligned H&E modality-level semantic aggregation token is used as the Query, and the projected H&E cell feature embedding set is used as the Key and Value. The global context information is injected back into each H&E cell feature embedding through the reverse self-attention module to obtain the refined set of H&E cell feature embeddings.

[0059] In a specific implementation, the matching cost of optimal bipartite graph matching is defined by the cosine distance (or Euclidean distance, negative dot product, etc.) between each cell feature embedding in the H&E cell feature embedding refinement set and each cell feature embedding in the mIF cell feature embedding set. The Hungarian algorithm seeks a one-to-one match that minimizes the sum of the distances of all matching pairs. The Hungarian matching loss can be the sum (or mean) of the distances of all matching pairs. By minimizing this loss, the model is driven to optimize the parameters of the inverse self-attention module and the H&E modality processing branch.

[0060] S6. Inference and Downstream Applications: After training, the trained and usable H&E modality processing branch and inverse self-attention module are obtained. For a new original H&E staining image, the original H&E staining image is converted into a grayscale image and smoothed. The smoothed H&E staining image is cropped into multiple non-overlapping image patches, and each image patch is used as the H&E staining image to be inferred. The H&E staining image to be inferred is input into the trained H&E modality processing branch to obtain the corresponding H&E cell feature embedding set, the projected H&E cell feature embedding set, and the aligned H&E modality-level semantic aggregation token. The aligned H&E modality-level semantic aggregation token and the projected H&E cell feature embedding set are input into the trained inverse self-attention module to obtain the refined H&E cell feature embedding set aligned with the mIF modality semantics.

[0061] During the inference phase, all parameters (such as smoothness, filter type, and image patch size) of the grayscale conversion, smoothing, and image patch cropping operations must be consistent with those used in the S1 data preparation and preprocessing phases to ensure consistency of the model input.

[0062] The refined set of H&E cell feature embeddings aligned with mIF modality semantics obtained above can enable high-quality downstream pathological analysis tasks: Tissue region segmentation: By clustering H&E cell feature embeddings aligned with mIF modality semantics to refine the set, this invention can be directly applied to unsupervised tissue region segmentation. On the CRCD dataset, its segmentation performance significantly outperforms other unsupervised methods by 20%-47%, and even approaches the level of fully supervised methods.

[0063] Prognostic and predictive association: Downstream experiments on an independent Liver-162 patient cohort showed that clustering results were significantly associated with patient survival prognosis (p<0.05). Machine learning models trained using these H&E cell features semantically aligned with mIF protein expression can effectively predict histological grades (e.g., tumor differentiation), indirectly demonstrating the important biological and clinical discriminative value of semantic information transferred from mIF to H&E.

[0064] Discovering potential subtypes: When the number of clusters is set to be greater than the number of cell types that can be identified by the naked eye, the cell population can be further subdivided to reveal potential subgroups with different morphological or microenvironmental characteristics, providing a new tool for the fine study of the tumor microenvironment.

[0065] This invention transfers the protein expression semantics of mIF to the morphological features of H&E, enabling the inference of protein expression patterns from conventional H&E sections without immunohistochemistry, and is particularly suitable for cell phenotype analysis in the tumor microenvironment.

[0066] This invention, through the specific embodiments described above, fully realizes an end-to-end solution. It creatively distinguishes between modality-level and cell-level alignment tasks, and solves them respectively using contrastive learning and the Hungarian algorithm, thereby completely avoiding the two major practical constraints of non-registration and lack of annotation. The entire solution has a clear flow and strong modularity. The role of key modules (such as the inverse self-attention module) has been verified through ablation experiments, proving its indispensable role in performance improvement, and providing reliable, efficient, and scalable technical support for large-scale pathological image analysis.

[0067] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a method for cross-modal cell alignment of pathological images in non-registration scenarios.

[0068] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for cross-modal cell alignment of pathological images in non-registration scenarios.

[0069] The above description is merely a preferred embodiment of the present invention and does not limit the scope of the patent. Any equivalent modifications made based on the concept of the present invention and the content of this specification, or direct / indirect applications in other related technical fields, are included within the scope of patent protection of the present invention.

Claims

1. A method for pathological image cross-modality cell alignment under non- registrable scenarios, characterized in that, Includes the following steps: Obtain H&E staining and mIF staining images of the same spatial anatomical location from adjacent physical slices; A H&E modality processing branch, a mIF modality processing branch, and a reverse self-attention module are constructed. The H&E modality processing branch is used to generate an H&E cell feature embedding set and an H&E modality-level semantic aggregation token. The mIF modality processing branch is used to generate an mIF cell feature embedding set and an mIF modality-level semantic aggregation token. The mIF modality processing branch includes a predefined non-parametric computation process to obtain the mIF cell feature embedding set. Based on the data augmented view of the H&E staining image and the data augmented view of the mIF staining image, self-supervised pre-training is performed on the H&E modality processing branch and the mIF modality processing branch respectively; After pre-training, the H&E staining image and the mIF staining image are input into the H&E modality processing branch and the mIF modality processing branch, respectively, and the H&E cell feature embedding set and the H&E modality-level semantic aggregation token, as well as the mIF cell feature embedding set and the mIF modality-level semantic aggregation token, are output, respectively. Using the mIF modality-level semantic aggregation token as the alignment target, the parameters of the H&E modality processing branch are updated through contrastive learning, while keeping the parameters of the mIF modality processing branch unchanged, so that the semantic space of the H&E modality-level semantic aggregation token is aligned with that of the mIF modality-level semantic aggregation token. The aligned H&E modality-level semantic aggregation token and the projected H&E cell feature embedding set are input into the inverse self-attention module to obtain the H&E cell feature embedding refinement set. Based on the H&E cell feature embedding refinement set and the mIF cell feature embedding set, the parameters of the inverse self-attention module and the H&E modality processing branch are updated by minimizing the Hungarian matching loss to achieve cell-level semantic alignment. Using the trained H&E modality processing branch and the reverse self-attention module, the H&E staining image to be inferred is processed to obtain a refined set of H&E cell feature embeddings aligned with the mIF modality semantics, for use in downstream pathological analysis tasks.

2. The method for cross-modal cell alignment of pathological images in non-registration scenarios according to claim 1, characterized in that, The H&E staining image and the mIF staining image are image blocks corresponding to spatial anatomical locations.

3. The method for cross-modal cell alignment of pathological images in non-registration scenarios according to claim 1 or 2, characterized in that, The H&E modality processing branch includes an H&E cell embedding extractor and a first attention aggregation module, while the mIF modality processing branch includes a feature computation unit for performing the predefined non-parametric computation process and a second attention aggregation module.

4. The method for cross-modal cell alignment of pathological images in non-registration scenarios according to claim 3, characterized in that, The H&E cell embedding extractor generates the H&E cell feature embedding set in the following manner: Cell instance segmentation is performed on the first processed image to obtain multiple cell regions; each cell region is sequentially input into a Transformer encoder to extract the corresponding cell feature embedding; the H&E cell feature embedding set is composed of the cell feature embeddings of all cells in the first processed image; wherein, the first processed image is the H&E staining image or a data augmented view of the H&E staining image.

5. The method for cross-modal cell alignment of pathological images in non-registration scenarios according to claim 3, characterized in that, The feature calculation unit obtains the mIF cell feature embedding set in the following manner: Cell nucleus instance segmentation is performed on the DAPI channel of the second processed image to obtain multiple cell nucleus regions; for each of the cell nucleus regions, the following operations are performed: a) Calculate the average signal intensity of the cell nucleus region on the DAPI channel; b) Based on the location of the cell nucleus region, the corresponding cell regions are determined in the K protein marker channels of the second processed image, and the average signal intensity of each cell region in its respective protein marker channel is calculated. The mIF cell feature embedding set is constructed from the K+1 dimensional signal intensity vectors obtained by steps a) and b) corresponding to all cell nucleus regions in the DAPI channel of the second processed image; wherein, the second processed image is the mIF stained image or a data augmented view of the mIF stained image.

6. The method for cross-modal cell alignment of pathological images in non-registration scenarios according to claim 3, characterized in that, Both the first attention aggregation module and the second attention aggregation module are Transformer encoders. The Transformer encoders generate corresponding modality-level semantic aggregation tokens through the following attention pooling method: A learnable aggregation token is randomly initialized, and the aggregation token is interacted with the input cell feature embedding set through self-attention and cross-attention. The aggregate token after the interaction is output as the corresponding modality-level semantic aggregation token.

7. The method for cross-modal cell alignment of pathological images in non-registration scenarios according to claim 3, characterized in that, The H&E modality processing branch further includes a fully connected layer located between the H&E cell embedding extractor and the first attention aggregation module, used to project the H&E cell feature embedding set to a dimensional space that matches the mIF cell feature embedding set, so as to obtain the projected H&E cell feature embedding set.

8. The method for cross-modal cell alignment of pathological images in non-registration scenarios according to claim 1, characterized in that, The self-supervised pre-training refers to performing intra-modal contrastive learning on the H&E modality processing branch and the mIF modality processing branch respectively: for each modality processing branch, the corresponding modality-level semantic aggregation tokens generated by two different data augmentation views from the same stained image after passing through the corresponding modality processing branch constitute a positive sample pair; Any modality-level semantic aggregation token in the positive sample pair is combined with the corresponding modality-level semantic aggregation token generated by a data augmentation view of a stained image from a different spatial anatomical location within the same modality after processing through the corresponding modality branch to form a negative sample pair. Optimization is performed by minimizing the contrast loss to maximize the consistency between the two modality-level semantic aggregation tokens in the positive sample pair, while minimizing the consistency between the two modality-level semantic aggregation tokens in the negative sample pair.

9. The method for cross-modal cell alignment of pathological images in non-registration scenarios according to claim 1, characterized in that, The step of updating the parameters of the H&E modality processing branch through contrastive learning while keeping the parameters of the mIF modality processing branch unchanged, thereby aligning the semantic space of the H&E modality-level semantic aggregation token with that of the mIF modality-level semantic aggregation token, is specifically achieved through the following cross-modal contrastive learning method: Positive sample pairs are formed by generating H&E modality-level semantic aggregation tokens and mIF modality-level semantic aggregation tokens from H&E staining images and mIF staining images originating from the same spatial anatomical location, respectively, after passing through the H&E modality processing branch and the mIF modality processing branch. The H&E modal semantic aggregation tokens in the positive sample pairs, together with the mIF modal semantic aggregation tokens generated after the mIF staining images from different spatial anatomical locations are processed by the mIF modal processing branch, constitute a negative sample pair set. Optimization is achieved by minimizing the contrast loss to bring the two modality-level semantic aggregate tokens in the positive sample pair closer in semantic space, and to increase the distance between the H&E modality-level semantic aggregate token and each mIF modality-level semantic aggregate token in the negative sample pair set in semantic space. During this stage, all parameters of the mIF modality processing branch are fixed, and only the parameters of the H&E modality processing branch are updated, thereby aligning the semantic space of the H&E modality-level semantic aggregate token to the semantic space of the mIF modality-level semantic aggregate token to obtain the aligned H&E modality-level semantic aggregate token.

10. The method for cross-modal cell alignment of pathological images in non-registration scenarios according to claim 1, characterized in that, The step of updating the parameters of the inverse self-attention module and the H&E modality processing branch by minimizing the Hungarian matching loss is specifically achieved through the following unsupervised matching method: Under cell-free conditions, the H&E cell feature embedding refinement set and the mIF cell feature embedding set are regarded as two sets to be matched; The Hungarian algorithm is used to automatically find an optimal bipartite graph matching between two sets to be matched, so as to minimize the overall matching cost. Based on the optimal matching result obtained by the Hungarian algorithm, the Hungarian matching loss is calculated, and optimization is performed by minimizing the loss, which forces the matched H&E cell feature embeddings and mIF cell feature embeddings to be close to each other in the feature space, thereby achieving cell-level semantic alignment.

Images