Pathological image processing method and system based on progressive cross-modal semantic interaction

Abstract

The invention relates to the technical field of medical image processing, and discloses a pathological image processing method and system based on progressive cross-modal semantic interaction, and the method comprises the following steps: carrying out the preprocessing and deep registration of HE and mIF images, and constructing a high-quality pairing data set through mutual information screening; thirdly, constructing a progressive cross-modal semantic interaction module, fusing HE morphology and mIF molecular features through a two-stage cross-attention mechanism, and generating semantic enhancement features; pre-training adopts a double-branch framework, and mask self-encoding is combined to reconstruct a local structure and global contrast learning alignment modal semantics. In addition, perception and fine-grained auxiliary loss are introduced, and learning is performed from multi-scale perception quality and local region correspondence constraint features. According to the method, the cross-modal alignment problem is effectively solved, the generated features have both anatomical structures and molecular semantics, and the accuracy and generalization ability of pathological image analysis are improved.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a pathological image processing method and system based on progressive cross-modal semantic interaction. Background Technology

[0002] Currently, pathological image analysis plays a decisive role in the field of tumor microenvironment analysis and clinical prognostic assessment. Hematoxylin-eosin (HE) staining, as the gold standard for morphological diagnosis, provides rich tissue structure information and is the main method for routine clinical testing. Meanwhile, multiplex immunofluorescence (mIF) staining technology can simultaneously label multiple biomarkers, revealing complex molecular expression and functional states between cells. Although HE images are inexpensive to acquire and widely available, they only present morphological features and cannot directly reflect deep molecular biological information; while mIF images, although providing the gold standard at the molecular level, are difficult to widely apply in large-scale clinical screening due to their complex preparation process and high cost.

[0003] For applications involving the aforementioned multimodal data, existing technical solutions primarily focus on spatial correlation mining between HE (healing emulsion) and mIF (middle infiltrate) images. Conventional processing typically employs image registration algorithms to align slices with different staining methods on anatomical structures. In deep learning applications, mainstream methods often utilize convolutional neural networks or Transformer architectures to independently extract features from HE images, or attempt to construct a mapping from HE to mIF using generative adversarial networks. Some studies employ contrastive learning strategies, attempting to bridge the gap between the two modalities in the latent feature space, aiming to leverage paired data to assist the model in learning common representations between images.

[0004] However, existing technologies still have limitations when processing cross-modal pathological images. First, traditional registration methods often struggle to accurately handle non-rigid deformations that occur during the preparation process and lack effective quantitative screening of registration quality, resulting in a large amount of noise and mismatches in the training data. Second, current cross-modal fusion strategies mostly remain at a shallow level of feature stitching or simple global alignment, lacking a progressive interaction mechanism and failing to deeply explore the layer-by-layer guidance of molecular semantics on morphological features, leading to insufficient feature fusion. Furthermore, a single pre-training task cannot simultaneously consider global semantics and local details. Existing loss functions often ignore multi-scale perceptual quality constraints and fine-grained region correspondences, making the features extracted by the model insufficient in biological interpretability and generalization ability for downstream tasks.

[0005] Therefore, the present invention provides a pathological image processing method and system based on progressive cross-modal semantic interaction to address the shortcomings of the prior art. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a pathological image processing method and system based on progressive cross-modal semantic interaction, which solves the problems of lack of molecular semantic information in HE images, difficulty in cross-modal data spatial alignment, and weak generalization ability due to insufficient feature fusion depth in existing pathological image analysis.

[0007] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a pathological image processing method based on progressive cross-modal semantic interaction, employing the following technical solution: A pathological image processing method based on progressive cross-modal semantic interaction includes the following steps: Step S1: Perform channel rearrangement, pseudo-color rendering and correction on the multiple immunofluorescence (mIF) image to generate a standardized mIF image. Use a deep learning framework to spatially register the hematoxylin-eosin (HE) image with the standardized mIF image. Step S2: Perform gridded traversal on the registered full slice to extract paired image blocks in batches, calculate the mutual information value between the image blocks, and retain the paired data that meet the threshold. Step S3: Extract the original features of HE and mIF image patches using the encoder, and generate semantically enhanced features through a two-stage progressive cross-attention mechanism that first establishes modality dependency and then constructs joint context; Step S4: Use a dual-branch framework to pre-train the semantic enhancement features, including a mask autoencoder branch for mask reconstruction of HE features, and a global contrastive learning branch for aligning global semantics between modalities. Step S5: Extract features, calculate perceptual auxiliary loss to constrain the high-level semantic structure, and simultaneously extract local image patch features to calculate fine-grained contrastive loss to learn local cross-modal associations.

[0008] By adopting the above technical solutions, this invention first solves the data format differences and spatial misalignment problems between HE images and mIF images through standardized preprocessing and deep learning registration, laying the foundation for subsequent analysis. Secondly, a high-quality pairing dataset is constructed using a mutual information filtering mechanism, effectively eliminating noisy data with large registration errors. The core progressive cross-modal semantic interaction mechanism, through two-stage attention interaction, not only establishes preliminary dependencies between modalities but also extracts deep semantic associations through joint context queries, injecting this cross-modal information back into the original features, significantly enhancing the expressive power of the features. Furthermore, the dual-branch pre-training framework combines mask reconstruction and global contrastive learning, enabling the model to capture both the local texture structure of HE images and achieve cross-modal global semantic alignment. Finally, perceptual auxiliary loss and fine-grained contrastive loss are introduced to further constrain the model at the level of high-level semantic features and local image patches, improving the accuracy and robustness of feature representation.

[0009] Preferably, in step S1, the preprocessing of the mIF image includes: establishing a predefined list of standardized channel names and their order; parsing all channels from the original mIF image data; traversing the predefined target channel sequence and searching for matching channels in the original image data; filling unmatched channels with all zeros to generate a standardized channel sequence; assigning a pseudo-color composed of red, green, and blue primary color components to each standardized channel; performing intensity normalization on the image data and performing pixel-by-pixel multiplication with the pseudo-color components; and applying a linear multiplication factor and a power-law function for brightness and gamma correction. The depth registration includes: resampling and color normalizing the HE image and the preprocessed mIF image; calculating the approximate spatial correspondence between the source image and the target image for initial alignment; calculating a non-rigid transform field for deformable registration; and applying the deformable field at the original resolution to complete the final deformation.

[0010] By adopting the above technical solution, the complex original mIF data can be converted into a standardized, visualized image format. The non-rigid registration technique effectively addresses the deformation that occurs during tissue slicing, ensuring the accurate correspondence between the two modal images in anatomical structures.

[0011] Preferably, in step S2, the calculation process of the mutual information value includes: normalizing and quantizing the image pixel values ​​to a preset level to construct a discrete probability distribution; calculating the joint histogram of the two images and the edge histogram of each image; calculating the entropy of each image based on the edge histogram; calculating the joint entropy of the two images based on the joint histogram; adding the entropy of the two images and subtracting the joint entropy to obtain the mutual information value.

[0012] By adopting the above technical solution and using mutual information in information theory as a quantitative indicator, the registration quality between image patches can be automatically and objectively evaluated. High-confidence training samples can be selected without manual intervention, ensuring the reliability of model input.

[0013] Preferably, in step S3, both the HE encoder and the mIF encoder adopt a visual Transformer architecture that includes a self-attention layer and a self-attention head, and load pre-trained initial weights; the original feature sequence includes class tokens, register tokens, and all image patch tokens; before extracting the original feature sequence, the input image patches are subjected to enhancement and standardization operations including: performing horizontal flipping, random rotation within a preset angle range, introducing random brightness and contrast perturbations, adjusting to a uniform size and mapping pixel values ​​to between 0 and 1, and standardizing using the mean and standard deviation of a preset dataset.

[0014] By adopting the above technical solutions, the powerful global modeling capabilities of the Vision Transformer architecture are used to extract features, and diverse data augmentation methods are used to simulate variations in real-world scenes, thereby improving the model's generalization ability and robustness to changes in rotation and lighting.

[0015] Preferably, in step S3, the first stage of the two-stage progressive cross-attention mechanism includes: projecting the original feature sequence of the HE image onto the query space, projecting the original feature sequence of the mIF image onto the key space and value space respectively, and updating the HE features by calculating attention weights; simultaneously, projecting the original feature sequence of the mIF image onto the query space, projecting the original feature sequence of the HE image onto the key space and value space respectively, and updating the mIF features by calculating attention weights, thereby establishing preliminary intermodal dependencies; the second stage of the two-stage progressive cross-attention mechanism includes: projecting the original feature sequence of the HE image onto the query space, projecting the original feature sequence of the HE image onto the key space and value space respectively, and updating the mIF features by calculating attention weights, thereby establishing preliminary intermodal dependencies; The original feature sequences of the image and the original feature sequences of the mIF image are concatenated and a joint context sequence is generated through linear layer projection. The joint context sequence is projected onto the query space, and the HE features and mIF features updated in the first stage are projected onto the key space and value space, respectively. Cross-attention calculation is performed to output a cross-modal semantic feature sequence. The semantic injector transforms the semantic information contained in the cross-modal semantic feature sequence through a multilayer perceptron and injects it into the original feature sequences of the HE image and the mIF image, respectively, to generate semantically enhanced hematoxylin-eosin features and multiple immunofluorescence features.

[0016] By adopting the above technical solutions, the first stage realizes mutual query and response between modalities and captures explicit correspondences; the second stage further mines deep, implicit common semantics and complementary information by constructing a joint context containing information from both parties as a query vector; the semantic injection mechanism ensures that the final features retain modality-specific attributes and integrate cross-modal contextual enhancement information, significantly improving the discriminative power of the features.

[0017] Preferably, in step S4, the construction process of the mask autoencoder branch includes: extracting image patch features representing local regions of the image from the semantically enhanced hematoxylin-eosin features; randomly masking the image patch features according to a preset probability; inputting the remaining unmasked image patch features into the decoder, reducing the dimensionality through a linear layer, processing them through a self-attention layer, and then projecting them back to the original image dimension to predict the original content of the masked image patch; and calculating the pixel-level mean square error between the content of the masked image patch predicted by the decoder and the true content of the corresponding image patch in the original HE image as the reconstruction loss.

[0018] By adopting the above technical solution and using the masked image modeling task, the model is forced to infer the missing region using contextual information, thereby learning the fine morphological structure and texture features inside the HE image, and improving the ability to understand the pathological tissue structure.

[0019] Preferably, in step S4, the construction process of the global contrastive learning branch includes: extracting category tokens representing the overall semantics as global features from the semantically enhanced hematoxylin-eosin features and the semantically enhanced multiple immunofluorescence features, respectively; calculating the cosine similarity between the global features of the HE image and the global features of the mIF image to form a similarity matrix; and using a symmetric cross-entropy loss, comparing the global features of the HE image with the global features of all mIF images on the one hand, and comparing the global features of the mIF image with the global features of all HE images on the other hand, to maximize the similarity between paired images and minimize the similarity between unpaired images.

[0020] By adopting the above technical solution, the paired HE and mIF image representations are brought closer together in the feature space, while the unpaired representations are pushed further away, thus achieving global semantic alignment across modalities. This enables the model to understand the high-level semantic consistency of samples of the same tissue under different modalities.

[0021] Preferably, in step S5, the construction process of the perception-assisted loss includes: using a pre-trained and parameter-frozen residual network as a feature extractor to obtain feature maps output by the original HE image and the reconstructed HE image at different intermediate layers; fusing the extracted multi-layer features to obtain the perception features of the original HE image and the perception features of the reconstructed HE image; subtracting the two fused feature tensors element by element and calculating the average of the squared differences to obtain the L2 distance of the feature layer as the perception-assisted loss.

[0022] By adopting the above technical solution, we go beyond simple pixel-level errors and introduce perceptual constraints based on feature space. This guides the model to focus on the high-level semantics and structural consistency of the image, thereby generating reconstruction results with higher visual quality and more accurate semantics.

[0023] Preferably, in step S5, the construction process of the fine-grained contrast loss includes: removing category tokens and register tokens from the semantically enhanced hematoxylin-eosin feature sequence and the semantically enhanced multiple immunofluorescence feature sequence, and retaining image patch tokens; flattening the retained local feature tensors to form a feature vector list, and performing L2 normalization; calculating the inner product between the flattened HE feature matrix and the transposed mIF feature matrix using matrix multiplication to generate a similarity matrix, and multiplying the similarity matrix by a learnable scaling factor; using each HE image patch as an anchor point, treating the corresponding mIF image patch at the same position as a positive sample, and the remaining image patches as negative samples, calculating the difference in both directions using the cross-entropy loss function and taking the average value to obtain the image patch-level fine-grained contrast loss.

[0024] By adopting the above technical solution, the granularity of contrastive learning is refined from the whole image level to the local image patch level, forcing the model to learn the fine correlation between the local regions of the HE image and the corresponding local regions of the mIF image, thereby further improving the accuracy of cross-modal feature alignment and the ability to express local details.

[0025] Secondly, the present invention provides a pathological image processing system based on progressive cross-modal semantic interaction, employing the following technical solution: A pathological image processing system based on progressive cross-modal semantic interaction includes: a preprocessing and deep registration module for performing data normalization and spatial alignment of HE and mIF images; a high-quality paired dataset construction module for constructing an image patch dataset for model training and filtering paired data based on mutual information; a progressive cross-modal semantic interaction module for extracting and fusing cross-modal features of HE and mIF images using a two-stage cross-attention mechanism; a dual-branch contrastive pre-training module for feature learning and optimization of the model through mask autoencoder and global contrastive learning; and an auxiliary loss calculation module for calculating perceptual auxiliary loss and fine-grained contrastive loss to constrain model training.

[0026] By adopting the above technical solutions, the system achieves full-process processing from data preprocessing, dataset construction, feature interaction and fusion to model pre-training and optimization through modular design. The modules work together to efficiently and accurately process multimodal pathological image data, providing powerful tool support for pathological diagnosis and medical research.

[0027] This invention provides a method and system for pathological image processing based on progressive cross-modal semantic interaction. It has the following beneficial effects: 1. This invention effectively solves the problem of spatial non-rigid deformation and alignment caused by the fabrication process in multimodal pathological images by implementing a deep learning-based strategy for depth registration and mutual information quality screening of HE and mIF images. This technical solution ensures accurate pixel-level correspondence between morphological features and molecular expression features, automatically eliminates low-quality or misregistered data pairs, and constructs a highly reliable spatial benchmark for cross-modal representation learning, thereby improving the quality and consistency of input data for pre-trained models.

[0028] 2. This invention employs a progressive cross-modal semantic interaction module, utilizing a two-stage cross-attention mechanism to achieve a progressive fusion between modalities, moving from shallow mutual attention to deep joint context mining. This design enables the model to effectively inject the rich molecular biological information contained in mIF images into the feature representation of HE images, breaking through the limitations of traditional single-modal visual features. This allows the generated pathological features to possess both macroscopic anatomical structures and microscopic molecular semantics, enhancing the biological interpretability of the features.

[0029] 3. This invention constructs a dual-branch pre-training framework combining mask autoencoder and global contrastive learning, and introduces perceptual and fine-grained auxiliary losses to achieve multi-dimensional feature constraints from global semantic alignment to local detail reconstruction. The perceptual loss ensures the integrity of the high-level semantic structure of the features, while the fine-grained contrastive loss enhances the cross-modal mapping capability of local regions. This multi-task collaborative optimization pre-training paradigm enhances the model's feature extraction capability and generalization performance, providing a more robust and accurate feature foundation for subsequent downstream tasks such as pathological diagnosis, prognostic assessment, and molecular subtype classification. Attached Figure Description

[0030] Figure 1 This is an overall flowchart of the pathological image processing method based on progressive cross-modal semantic interaction according to an embodiment of the present invention. Figure 2 This is a flowchart of the preprocessing and depth registration process according to an embodiment of the present invention; Figure 3 This is an architecture diagram of the progressive cross-modal semantic interaction module according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the construction of a dual-branch contrastive pre-training framework according to an embodiment of the present invention. Figure 5 This is an architecture diagram of the perception-assisted loss module according to an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] See attached document Figure 1 This invention provides a pathological image processing method based on progressive cross-modal semantic interaction, the method specifically including the following steps: First, step S1 is performed to preprocess and depth register the HE and mIF images. This step aims to address the data format differences and spatial misalignment issues between multimodal pathological images. Specifically, the mIF image is processed first, with channel rearrangement, pseudo-color rendering, and brightness and gamma correction performed to convert the raw data into a TIFF format image with standardized channel order, custom pseudo-color, and adjustable brightness, thus providing a unified data foundation for subsequent processing. Then, the DeeperHistReg framework, based on deep learning, is used to process the HE and preprocessed mIF images. This framework uses a series of cascaded operations, including resampling, color normalization, initial alignment, deformable registration, and final deformation, to accurately register the HE and mIF images spatially, ensuring consistency in anatomical structure between the two modalities.

[0033] Next, step S2 is executed to construct a high-quality paired dataset of HE and mIF images. Since the full-slice images have high resolution, they need to be segmented into image patches suitable for model training. From the preprocessed and depth-registered full-slice HE and mIF images, a gridded traversal is performed according to a preset fixed size to extract paired image patches in batches. During the extraction process, to eliminate data pairs with large registration errors, mutual information is used as a quantification metric to rigorously screen the extracted image patches. The mutual information value between paired HE and mIF image patches is calculated, and only image patch pairs whose mutual information values ​​meet a specific threshold are retained, thus constructing a paired dataset consisting of high-quality, spatially aligned HE and mIF image patches.

[0034] Step S3 is then executed to construct a progressive cross-modal semantic interaction module. This module is used to fuse the morphological features of the HE image and the molecular expression features of the mIF image during the feature encoding stage. First, the input image patches undergo data augmentation preprocessing, followed by the extraction of original feature sequences through both the HE encoder and the mIF encoder. This module employs a two-stage progressive cross-attention mechanism: in the first stage, HE features and mIF features interact bidirectionally as queries and keys to establish initial intermodal dependencies; in the second stage, a joint context sequence containing HE and mIF information is constructed as a query, which interacts again with the features from the first stage to extract deep semantic associations. The final generated joint semantic representation is fed back into the original feature paths of the HE and mIF images via a semantic injector, generating semantically enhanced features that incorporate cross-modal context.

[0035] Then, step S4 is executed to construct a dual-branch contrastive pre-training framework. This framework utilizes semantically enhanced features for model training and includes a masked autoencoder branch and a global contrastive learning branch. In the masked autoencoder branch, the semantically enhanced HE feature blocks are randomly masked, and the decoder predicts the pixel information of the masked regions. By minimizing the difference between the reconstructed image and the original image, the model is encouraged to understand the local pixel-level structure of the HE image. In the global contrastive learning branch, global classification labels are extracted from the HE image and the mIF image respectively, and their similarity in the feature space is calculated. By maximizing the similarity between paired images and minimizing the similarity between unpaired images, the global semantic representation of the HE image and the global semantic representation of the mIF image are aligned in the feature space.

[0036] Finally, step S5 is executed to construct the perceptual and fine-grained auxiliary loss module. This module is used to further optimize the model's feature representation capabilities. The perceptual auxiliary loss utilizes a pre-trained residual network to extract multi-scale features from the original HE image and the reconstructed image, respectively, and calculates the distance between them at the feature level, thereby guiding the model to focus on the high-level semantic and structural information of the image, rather than being limited to pixel-level errors. The fine-grained contrastive loss focuses on the correspondence of local regions, extracting, flattening, and normalizing the local image patch features corresponding to the HE image and the mIF image, calculating cross-modal similarity at the image patch level, and performing comparative learning. By combining the perceptual loss and the fine-grained loss, the model can simultaneously learn the high-level semantic features of the HE image and the cross-modal correlation between the HE and mIF images in local details.

[0037] See attached document Figure 1 and attached Figure 2 One embodiment of the present invention discloses specific steps for preprocessing and depth registration of HE and mIF images. First, the CZI format multiplex immunofluorescence image is converted into a TIFF format image with a standardized channel order, custom pseudocolor, and adjustable brightness, providing basic data for subsequent registration with the hematoxylin and eosin stained image. Then, the preprocessed hematoxylin and eosin stained image and the multiplex immunofluorescence image are registered using the DeeperHistReg framework.

[0038] This step first performs channel rearrangement and standardization of the multiplex immunofluorescence image. A predefined, standardized list of channel names and their desired order is established, with channels 0 to 6 corresponding to the KI67, CD23, PANCK, CD20, CD21, CD3, and DAPI markers, respectively. The system parses and identifies all channels, their original names, and their order from the input raw multiplex immunofluorescence image data. Next, it iterates through the predefined target channel sequence, searching the raw image data for a matching channel name. If no matching channel is found, the channel position is filled with all zeros. Finally, all extracted or generated channel image data are arranged according to the predefined target channel sequence to generate a standardized channel sequence.

[0039] After channel normalization, pseudo-color rendering is performed on each channel in the normalized channel sequence. A unique pseudo-color is pre-assigned to each normalized channel, composed of the three primary color components: red, green, and blue. Intensity normalization is then performed on the image data of each normalized channel, scaling the pixel values ​​of each channel to between 0 and 1 based on the maximum pixel value of that channel. The normalization calculation formula is as follows: ; in, This represents the pixel intensity value after normalization. This represents the original pixel intensity value.

[0040] For each normalized channel, its normalized intensity value is multiplied pixel-by-pixel with the preset R, G, and B pseudo-color components. If a channel is designated as red (R = 1 and G and B = 0), then the intensity values ​​of all pixels in that channel only affect the red component of the final image, while the green and blue components remain zero. This operation generates a single-channel color image with the same size as the original channel, where the signal intensity of that channel is mapped to the specified pseudocolor. Subsequently, all the pseudo-color processed single-channel color images are superimposed pixel-by-pixel to generate a composite RGB color image that incorporates all the labeled signals.

[0041] During pseudo-color rendering, luminance and gamma correction are performed. A linear multiplication factor is applied to the intensity of each normalized channel. When the factor is greater than 1, the overall pixel value of the image increases; when the factor is less than 1, the overall pixel value of the image decreases. This adjustment is linear, scaling all intensity values ​​proportionally. Before luminance adjustment, a power-law function is applied to the intensity value of each normalized channel for gamma correction. When the gamma value is less than 1, the values ​​in low-grayscale areas of the image are non-linearly increased, increasing the significance of weak signals with minimal impact on high-grayscale areas, thereby increasing the dynamic range and detail visibility of the image. When the gamma value is greater than 1, the values ​​in low-grayscale areas of the image are decreased, increasing contrast. When the gamma value is 1, the original values ​​remain unchanged.

[0042] After image preprocessing, depth registration is performed using the DeeperHistReg framework. First, the input hematoxylin and eosin stained image and the preprocessed multiplex immunofluorescence image are resampled and color normalized. Resampling adjusts the image resolution to a preset scale, making it suitable for subsequent computation. Color normalization unifies the color distribution characteristics of the images, reducing color differences between different images. After preprocessing, initial alignment is performed to calculate the approximate spatial correspondence between the source and target images, providing basic parameters for subsequent fine alignment. After initial alignment, deformable registration is performed. This step calculates a non-rigid transform field, adjusting local regions of the image to match the local geometric features of the target image. Finally, a final deformation operation is performed. The calculated deformation field is applied at the original resolution to adjust the source image to be spatially aligned with the target image.

[0043] See attached document Figure 1 and attached Figure 2 This invention provides a pathological image processing method based on progressive cross-modal semantic interaction, including the step of constructing a high-quality HE-mIF paired dataset. Since the registered whole slice images (WSI) have high resolution, to facilitate subsequent model training, the registered WSIs need to be segmented into paired, fixed-size image patches and subjected to quality screening.

[0044] This step first performs batch extraction of paired, fixed-size image patches. Specifically, a systematic gridded traversal is performed on the registered HE and mIF images, using a preset fixed image patch size as the step size, to determine the starting position of each potential target image patch in the HE image. For each determined position, combined with the preset spatial offset of the source image (i.e., the mIF image) relative to the target image (i.e., the HE image), the starting coordinates of the region corresponding to the target image patch in the source image are precisely calculated. Subsequently, a strict integrity check is performed on both potential image patches—the image patch in the target image and its corresponding region in the source image. This check includes verifying whether their respective lower right corner coordinates exceed the actual boundaries of their respective images. Only when both checks are successful, ensuring that the target image patch and the corresponding region in the source image can form a complete image patch of the specified size within their respective image ranges, is the position considered a valid extraction point. Based on this, the actual data reading operation is performed to extract and save this pair of complete and precisely aligned image patches.

[0045] After image patch extraction is complete, a registration and filtering step based on mutual information is performed. First, the input color image is converted to grayscale, as mutual information is typically calculated on a single-channel image. To calculate mutual information, the image pixel values ​​need to be quantized and a histogram constructed. Specifically, this involves processing two images (denoted as )... and The pixel values ​​are normalized to the range of 0 to 1, and then quantized to a preset number of bins to create a discrete probability distribution.

[0046] Next, the joint histogram of the two images is calculated. This histogram represents the frequency with which pixel pairs in two images simultaneously exhibit a specific combination of values. The edge histogram for each image is derived from the joint histogram. and These two edge histograms represent the frequency of each pixel value in each image. Based on these histograms, the entropy of each image is calculated using the Shannon entropy formula. , and their joint entropy Entropy is a measure of the uncertainty of a random variable. Finally, mutual information is obtained through the following formula. : ; in, Indicates mutual information, This represents the entropy of the first image. This represents the entropy of the second image. This represents the joint entropy of the two images. The entropy involved in the above formula... The calculation formula is as follows: ; in, Represents random variables Entropy; Representing variables The probability of occurrence. By calculating mutual information, the registration quality between HE image patches and mIF image patches can be quantitatively evaluated, thereby selecting high-quality paired datasets for subsequent model training.

[0047] See attached document Figure 3 This invention provides a pathological image processing method based on progressive cross-modal semantic interaction. The steps of constructing the progressive cross-modal semantic interaction module specifically include the following processes: First, augmentation and normalization operations are performed on the input image data. The images are horizontally flipped to increase data diversity. The images are randomly rotated within a range of ±10 degrees to simulate slight pose changes that may occur in real-world scenes. Random perturbations of no more than 0.2 are introduced into both brightness and contrast. After data augmentation, the images are resized to a uniform size and converted from PIL format to tensors, with pixel values ​​mapped to between 0 and 1. The mean and standard deviation of the ImageNet dataset are used to normalize the three channels of the images individually to ensure a consistent pixel distribution across different images.

[0048] Preprocessed paired hematoxylin-eosin stained image blocks and multiplex immunofluorescence image blocks were input into the corresponding hematoxylin-eosin encoder and multiplex immunofluorescence encoder, respectively. Both the hematoxylin-eosin encoder and the multiplex immunofluorescence encoder adopted the VisionTransformer architecture of ViT. Giant The model comprises 24 self-attention layers and 24 self-attention heads. Initial weights pre-trained on over 350,000 hematoxylin-eosin and immunohistochemical pathological images using DINOv2 were loaded. Hematoxylin-eosin image patches were processed by a hematoxylin-eosin encoder to generate raw hematoxylin-eosin feature sequences containing class tokens, register tokens, and all image patch tokens. These feature sequences contain morphological information about the hematoxylin-eosin images. Corresponding multiplex immunofluorescence images were encoded using an isomorphic multiplex immunofluorescence image encoder, outputting raw multiplex immunofluorescence feature sequences containing class tokens, register tokens, and all image patch tokens. These feature sequences contain molecular information about the multiplex immunofluorescence images. The model was fine-tuned using a low-rank adaptation technique, training only approximately 26% of the parameters.

[0049] The generated raw feature sequences are then fed into the progressive cross-modal semantic interaction module. In the first stage, hematoxylin-eosin features are used as query vectors to focus on multiple immunofluorescence features, while multiple immunofluorescence features are used as query vectors to focus on hematoxylin-eosin features, thereby establishing preliminary intermodal dependencies. The calculation method for the first stage is as follows: ; ; in, This represents the hematoxylin-eosin features updated after the first stage of cross-attention; Represents the original feature sequence of the hematoxylin-eosin image; This indicates the multiple immunofluorescence features updated after the first stage of attention; The original feature sequences representing multiplex immunofluorescence images; This represents a linear mapping matrix that projects hematoxylin-eosin features into the query space to focus on multiple immunofluorescence. This represents the linear mapping matrix projected from multiple immunofluorescence features onto the bond space; This represents the linear mapping matrix projected from multiple immunofluorescence features onto the value space; This represents a linear mapping matrix that projects multiple immunofluorescence features into the query space to focus on hematoxylin-eosin features; This represents the linear mapping matrix of hematoxylin-eosin features projected onto the bond space; This represents the linear mapping matrix projected from the hematoxylin-eosin features onto the value space; Indicates the feature dimension.

[0050] In the second stage, the original hematoxylin-eosin features and the original multiplex immunofluorescence features are concatenated and projected using a linear layer to generate a joint context sequence. This joint context sequence is used as a query vector and cross-attention is calculated with the interaction result from the first stage to extract deep semantic associations between modalities. The output is a cross-modal semantic feature sequence that captures the shared and complementary semantic information of hematoxylin-eosin and multiplex immunofluorescence. The calculation method for the second stage is as follows: ; ; ; in, This represents a linear layer used to project the spliced ​​sequence into a shared context space; Indicates a joint context sequence; This represents the deep cross-modal features of hematoxylin and eosin obtained after the second stage of semantic refinement; This represents the deep cross-modal features obtained after the second stage of semantic refinement; This represents the linear mapping matrix projected onto the query space from the combined context sequence; This represents the linear mapping matrix from the updated features to the key space in the first stage; This represents the linear mapping matrix from the first-stage updated features to the value space.

[0051] The cross-modal semantic injector transforms the semantic information contained in the cross-modal semantic feature sequence through a multilayer perceptron and injects it into the original hematoxylin-eosin feature sequence and the original multiplex immunofluorescence feature sequence, respectively, to generate semantically enhanced hematoxylin-eosin features and multiplex immunofluorescence features. This ensures that the final output hematoxylin-eosin image and multiplex immunofluorescence image representation include their own visual information as well as the incorporated molecular semantic context.

[0052] See attached document Figure 4 The step S4 provided by the present invention specifically includes constructing a dual-branch contrastive pre-training framework, which includes a mask autoencoder branch and a global contrastive learning branch.

[0053] First, a masked autoencoder branch is constructed. This branch aims to enable the model to learn the intrinsic structure, local texture, and integrity of the hematoxylin and eosin stained image. Specifically, patch features representing various local regions of the image are extracted from the semantically enhanced hematoxylin and eosin stained features generated in step S3. These patch features are then randomly masked with a 75% probability, preventing the model from directly acquiring information about these masked regions. The remaining unmasked patch features are input into a separate decoder. The hematoxylin and eosin stained decoder contains 8 self-attention layers and 16 self-attention heads. The decoder first reduces the dimensionality of the semantically enhanced hematoxylin and eosin stained features through a linear layer acting as a projection module, then processes them through the 8 self-attention layers, and finally projects them back to the same dimension as the original image. The decoder's task is to predict the original content of the masked patches based on the visible patch features. The model compares the content of the masked patches predicted by the decoder with the actual content of the corresponding patches in the original hematoxylin and eosin stained image and calculates the reconstruction loss, calculated as follows: ; in, Indicates the losses incurred during reconstruction; Indicates the total number of pixels that are masked; Indicates the first The value of each reconstructed pixel; Indicates the first The model optimizes by minimizing this reconstruction loss, thereby helping the encoder learn to recover complete hematoxylin and eosin staining image information from semantically enhanced hematoxylin and eosin staining features.

[0054] Secondly, a global contrastive learning branch is constructed. This branch aims to enable the model to learn cross-modal semantic alignment between hematoxylin and eosin stained images and multiplex immunofluorescence images, ensuring that their feature representations are semantically correlated. Specifically, the process involves extracting CLSTokens representing the overall semantics of the hematoxylin and eosin stained image from the semantically enhanced hematoxylin and eosin stained features, using them as global features for the hematoxylin and eosin stained image. Simultaneously, CLSTokens representing the overall semantics of the multiplex immunofluorescence image are extracted from the semantically enhanced multiplex immunofluorescence features, also used as global features for the multiplex immunofluorescence image. The similarity between the global features of the hematoxylin and eosin stained images and the global features of the multiplex immunofluorescence image is calculated. For all hematoxylin and eosin stained image and multiplex immunofluorescence image pairs in a batch, a similarity matrix is ​​formed by calculating cosine similarity. Each element in this matrix represents the similarity between a hematoxylin and eosin stained image patch and a multiplex immunofluorescence image patch. The model's goal is to maximize the similarity between paired hematoxylin and eosin stained images and minimize the similarity between unpaired images. This process is achieved using a symmetric cross-entropy loss.

[0055] On one hand, the global features of the hematoxylin and eosin stained image are compared with the global features of all multiplex immunofluorescence images, with the expectation that it is most similar to the correctly paired multiplex immunofluorescence image. The loss calculation formula in this direction is as follows: ; in, This indicates the contrast loss from hematoxylin and eosin staining images to multiplex immunofluorescence images; Indicates the first The first hematoxylin-eosin stained image and the second Similarity scores of multiple immunofluorescence images; This represents the similarity score of paired samples; Indicates the batch size.

[0056] On the other hand, the global features of multiplex immunofluorescence images are compared with the global features of all hematoxylin and eosin stained images, with the expectation that they are most similar to the correctly paired hematoxylin and eosin stained images. The loss calculation formula in this direction is as follows: in, This indicates the contrast loss in the direction from multiplex immunofluorescence images to hematoxylin and eosin stained images; Represents the similarity matrix after transpose. Line number Column elements; This represents the elements of paired samples after the similarity matrix is ​​transposed; This represents the batch size. By minimizing the contrast loss in the two directions mentioned above, the model can learn how to map hematoxylin and eosin stained images and multiplex immunofluorescence images to a shared semantic space, so that their respective features can be aligned in the feature space.

[0057] See attached document Figure 5 This invention provides a pathological image processing method based on progressive cross-modal semantic interaction. The method comprises two main parts: constructing a perceptual auxiliary loss module and constructing a fine-grained contrast loss module. This module aims to optimize the parameter update process of the encoder and decoder through multi-scale feature constraints and comparative calculations of local details.

[0058] For the perceptual auxiliary loss component, the network utilizes a pre-trained ResNet34 model with frozen parameters as a feature extractor. The original HE image and the HE image reconstructed via a masked autoencoder branch decoder are input into this feature extractor. During feature extraction, feature maps from the intermediate layers (layer 1, layer 2, layer 3, and layer 4) of the ResNet34 model are obtained. These feature maps at different layers correspond to semantic information at different scales of the image. The extracted features from the four layers are then fused to obtain the perceptual features of the original HE image and the perceptual features of the reconstructed HE image.

[0059] Subsequently, the L2 distance between the perceptual features of the original HE image and the perceptual features of the reconstructed HE image is calculated. Specifically, the two fused feature tensors are subtracted element-wise, and the difference is squared before averaging. The formula for calculating the perceptual assistance loss is as follows: ; in, This indicates perceptual assistance loss; This indicates the number of feature layers selected, which is 4 here; This indicates the ResNet34 network's... Feature extraction function of the layer; Represents the reconstructed HE image; Represents the original HE image; This represents the square of the L2 norm. This loss function quantifies the difference between the reconstructed image and the original image in the feature space, rather than simply calculating the pixel-level mean square error. The calculated perceptual loss is multiplied by preset hyperparameter weights and added to the model's total loss function. The total loss function includes the masked autoencoder pixel reconstruction loss, the global contrastive learning loss, and the fine-grained loss. During training, the parameters of the decoder and encoder, including the parameters of the LoRA adaptation layer, are updated using the backpropagation algorithm based on this perceptual loss to optimize the perceptual quality of the feature representation.

[0060] For the fine-grained contrast loss, this step aims to align fine-grained features of HE and multiplex immunofluorescence images in local regions. First, semantically enhanced HE feature sequences and semantically enhanced mIF feature sequences are obtained after processing by a progressive cross-modal semantic interaction module. For both feature sequences, classification identifiers (CLStokens) and register identifiers (REGtokens) at the beginning of the sequence are removed, retaining only the remaining 256 image patch identifiers (Patchtokens). These image patch identifiers correspond to a fixed number of non-overlapping local regions into which the input image is segmented. The same extraction operation is performed on both HE and mIF features to ensure a one-to-one spatial alignment between the retained HE image patch features and the mIF image patch features.

[0061] Next, the extracted local feature tensors are flattened. The image patch features of all images in the entire batch are concatenated to form a feature vector list, where each row vector represents an independent image patch feature. The flattened HE feature vectors and mIF feature vectors are then L2 normalized to make the magnitude of each vector equal to 1, ensuring that subsequent similarity calculations are based solely on vector direction (cosine similarity) and are not affected by vector magnitude. The inner product between the flattened HE feature matrix and the transposed mIF feature matrix is ​​calculated using matrix multiplication to generate a similarity matrix. In this similarity matrix, elements... Indicates the first The HE image block and the first Cosine similarity between mIF image patches.

[0062] The generated similarity matrix is ​​then multiplied by a learnable scaling factor, logit. scaleTo adjust the dynamic range of the numerical values, the elements on the diagonal of the similarity matrix correspond to HE (high-resolution) and mIF (multi-resolution) image patches in the same spatial location, representing positive sample pairs. Elements off-diagonally correspond to combinations of image patches in different locations, representing negative sample pairs. A full-batch computation approach is used to cover all possible cross-modal matching. During the loss calculation phase, an InfoNCE-style contrastive loss function is employed. For each HE image patch as an anchor point, its corresponding mIF image patch at the same location is considered a positive sample, while all other image patches are considered negative samples.

[0063] Specifically, for the direction from HE to mIF, each row of the similarity matrix is ​​treated as a logits value, and a softmax function is applied to transform it into a probability distribution. This probability distribution is compared with the target one-hot label, where the label value at the positive sample position is 1, and the label values ​​at other positions are 0. The difference between the two is calculated using the cross-entropy loss function. Similarly, for the direction from mIF to HE, each column of the similarity matrix is ​​treated as a logits value, and the same softmax transformation and cross-entropy loss calculation are performed. The final patch-level fine-grained contrast loss is the average of the loss values ​​from these two directions to ensure optimization symmetry. This loss value is multiplied by the corresponding weight coefficient and added to the total loss. The encoder parameters are then updated through the backpropagation algorithm, thereby refining the feature representation at the patch level.

[0064] This invention also provides a pathological image processing system based on progressive cross-modal semantic interaction. Each module in this system corresponds one-to-one with each step in the above method embodiments. The system specifically includes: Preprocessing and Depth Registration Module: This module performs data normalization and spatial alignment of the HE image and the mIF image, corresponding to step S1 in the above method embodiment. Specifically, this module first performs channel rearrangement, pseudo-color rendering, and brightness and gamma correction on the mIF image to generate a standardized TIFF format image; then, using a built-in deep learning registration framework (such as DeeperHistReg), it resamples, normalizes the color, performs initial rigid alignment, and performs non-rigid deformable registration on the HE image and the preprocessed mIF image, outputting spatially precisely corresponding full-slice image data.

[0065] High-quality paired dataset construction module: This module is connected to the preprocessing and deep registration module and is used to construct an image patch dataset for model training, corresponding to step S2 in the above method embodiment. This module performs gridded traversal on the registered full slices, extracting fixed-size paired image patches in batches; and has a built-in mutual information calculation unit to calculate the mutual information value between the extracted HE image patches and mIF image patches, automatically removing low-quality or misregistered data pairs according to a preset threshold, and retaining high-quality paired data.

[0066] Progressive cross-modal semantic interaction module: This module is used to extract and fuse multimodal features, corresponding to step S3 in the above method embodiment. This module includes a parallel HE encoder and an mIF encoder (both using the Vision Transformer architecture), as well as a two-stage cross-attention submodule. The first-stage submodule is used to calculate the bidirectional attention weights between HE features and mIF features to establish initial dependencies; the second-stage submodule is used to construct a joint context sequence as a query vector to extract deep semantic associations; finally, the fused semantic information is fed back to the original feature path through a semantic injector to generate semantically enhanced HE features and mIF features.

[0067] Dual-branch contrastive pre-training module: This module is used for feature learning and optimization of the model, corresponding to step S4 in the above method embodiment. This module includes a mask autoencoder unit and a global contrastive learning unit. The mask autoencoder unit randomly masks the semantically enhanced HE feature blocks and reconstructs the occluded regions using a decoder, learning the local structure by minimizing the reconstruction pixel error; the global contrastive learning unit extracts global classification tokens (CLS tokens) from the HE and mIF images, achieving global semantic alignment between modalities by maximizing the cosine similarity of paired samples and minimizing the similarity of unpaired samples.

[0068] Auxiliary Loss Calculation Module: This module is used to calculate multi-scale losses to constrain model training, corresponding to step S5 in the above method embodiment. This module includes a perceptual loss calculation unit and a fine-grained contrastive loss calculation unit. The perceptual loss calculation unit uses a pre-trained residual network to extract multi-layer feature maps of the original HE image and the reconstructed image, and calculates the L2 distance at the feature level; the fine-grained contrastive loss calculation unit extracts and flattens local image patch features (Patch Tokens), calculates the similarity matrix between the HE and the local mIF region, and aligns the local fine-grained features based on InfoNCE loss.

Claims

1. A pathological image processing method based on progressive cross-modal semantic interaction, characterized in that, Includes the following steps: S1. Channel rearrangement, pseudo-color rendering and correction are performed on the multiple immunofluorescence (mIF) image to generate a standardized mIF image. The hematoxylin-eosin (HE) image is spatially registered with the standardized mIF image using a deep learning framework. S2. Perform gridded traversal on the registered full slice to extract paired image blocks in batches, calculate the mutual information value between the image blocks and retain the paired data that meet the threshold. S3. The encoder extracts the original features of HE and mIF image patches. Through a two-stage progressive cross-attention mechanism that first establishes modality dependency and then constructs joint context, semantically enhanced features are generated by the semantic injector. S4. A dual-branch framework is used to pre-train semantic enhancement features, including a mask autoencoder branch for mask reconstruction of HE features and a global contrastive learning branch for aligning global semantics between modalities. S5. Extract features, calculate perceptual auxiliary loss to constrain high-level semantic structures, and simultaneously extract local image patch features to calculate fine-grained contrastive loss to learn local cross-modal associations.

2. The pathological image processing method based on progressive cross-modal semantic interaction according to claim 1, characterized in that, In step S1, the preprocessing of the mIF image includes: Establish a predefined list of standardized channel names and their order, parse all channels from the original mIF image data, traverse the predefined target channel sequence and search for matching channels in the original image data, fill the unmatched channels with all zeros, and generate a standardized channel sequence. Each standardized channel is assigned a pseudo-color composed of red, green, and blue primary color components. After intensity normalization of the image data, a pixel-by-pixel multiplication operation is performed with the pseudo-color components, and a linear multiplication factor and a power-law function are applied for brightness and gamma correction. The depth registration includes: resampling and color normalization of the HE image and the preprocessed mIF image, calculating the approximate spatial correspondence between the source image and the target image for initial alignment, calculating a non-rigid transformation field for deformable registration, and applying the deformation field at the original resolution to complete the final deformation.

3. The pathological image processing method based on progressive cross-modal semantic interaction according to claim 1, characterized in that, In step S2, the calculation process of the mutual information value includes: The image pixel values ​​are normalized and quantized to a preset level to construct a discrete probability distribution, and the joint histogram of the two images and the edge histogram of each image are calculated. The entropy of each image is calculated based on the edge histogram, and the joint entropy of the two images is calculated based on the joint histogram. The mutual information value is obtained by adding the entropies of the two images and subtracting the joint entropy.

4. The pathological image processing method based on progressive cross-modal semantic interaction according to claim 1, characterized in that, In step S3, both the HE encoder and the mIF encoder employ a visual Transformer architecture that includes a self-attention layer and a self-attention head, and load pre-trained initial weights; the original feature sequence includes class tokens, register tokens, and all image patch tokens. Before extracting the original feature sequence, the input image patch is enhanced and standardized by performing the following operations: performing horizontal flipping, random rotation within a preset angle range, introducing random brightness and contrast perturbations, adjusting to a uniform size and mapping pixel values ​​to between 0 and 1, and standardizing using the mean and standard deviation of a preset dataset.

5. The pathological image processing method based on progressive cross-modal semantic interaction according to claim 1, characterized in that, In step S3, the first stage of the two-stage progressive cross-attention mechanism includes: projecting the original feature sequence of the HE image onto the query space, projecting the original feature sequence of the mIF image onto the key space and value space respectively, and updating the HE features by calculating attention weights; simultaneously, projecting the original feature sequence of the mIF image onto the query space, projecting the original feature sequence of the HE image onto the key space and value space respectively, and updating the mIF features by calculating attention weights, thereby establishing preliminary intermodal dependencies; The second stage of the two-stage progressive cross-attention mechanism includes: concatenating the original feature sequence of the HE image and the original feature sequence of the mIF image, and generating a joint context sequence through linear layer projection; projecting the joint context sequence onto the query space, projecting the HE features and mIF features updated in the first stage onto the key space and value space respectively, performing cross-attention calculation, and outputting a cross-modal semantic feature sequence. The semantic injector transforms the semantic information contained in the cross-modal semantic feature sequence through a multilayer perceptron and injects it into the original feature sequences of the HE image and mIF image, respectively, to generate semantically enhanced hematoxylin-eosin features and multiple immunofluorescence features.

6. The pathological image processing method based on progressive cross-modal semantic interaction according to claim 1, characterized in that, In step S4, the construction process of the mask autoencoder branch includes: Image patch features representing local regions of the image are extracted from the semantically enhanced hematoxylin-eosin features, and the image patch features are randomly masked according to a preset probability. The features of the remaining undisturbed image patches are input into the decoder. After the dimensionality is reduced by a linear layer, the features are processed by a self-attention layer and then projected back to the original image dimension to predict the original content of the undisturbed image patches. The pixel-level mean square error between the content of the masked image patch predicted by the decoder and the true content of the corresponding image patch in the original HE image is calculated as the reconstruction loss.

7. The pathological image processing method based on progressive cross-modal semantic interaction according to claim 1, characterized in that, In step S4, the construction process of the global contrastive learning branch includes: Category tokens representing the overall semantics were extracted as global features from semantically enhanced hematoxylin-eosin features and semantically enhanced multiple immunofluorescence features, respectively. Calculate the cosine similarity between the global features of the HE image and the global features of the mIF image to form a similarity matrix; A symmetrical cross-entropy loss is employed, which compares the global features of the HE image with the global features of all mIF images, and the global features of the mIF image with the global features of all HE images, thereby maximizing the similarity between paired images and minimizing the similarity between unpaired images.

8. The pathological image processing method based on progressive cross-modal semantic interaction according to claim 1, characterized in that, In step S5, the process of constructing the perception-assisted loss includes: Using a pre-trained residual network with frozen parameters as a feature extractor, feature maps of the original HE image and the reconstructed HE image output at different intermediate layers are obtained respectively. The extracted multi-layer features are fused to obtain the original HE image perception features and the reconstructed HE image perception features, respectively. The L2 distance at the feature level is calculated by subtracting the two fused feature tensors element by element and taking the average of the squared differences, and is used as the perception-assisted loss.

9. A pathological image processing method based on progressive cross-modal semantic interaction according to claim 1, characterized in that, In step S5, the process of constructing the fine-grained contrast loss includes: Remove category tokens and register tokens from semantically enhanced hematoxylin-eosin feature sequences and semantically enhanced multiple immunofluorescence feature sequences, and retain image block tokens; The retained local feature tensors are flattened to form a list of feature vectors, and then L2 normalized. A similarity matrix is ​​generated by calculating the inner product between the flattened HE feature matrix and the transposed mIF feature matrix using matrix multiplication, and then multiplying the similarity matrix by a learnable scaling factor. For each HE image block as an anchor point, the corresponding mIF image block at the same position is regarded as a positive sample, and the remaining image blocks are regarded as negative samples. The difference is calculated using the cross-entropy loss function in both directions and the average value is taken to obtain the fine-grained contrast loss at the image block level.

10. A pathological image processing system based on progressive cross-modal semantic interaction, characterized in that, include: The preprocessing and depth registration module is used to perform data normalization and spatial alignment of HE and mIF images; A high-quality paired dataset building module is used to construct image patch datasets for model training and to filter paired data based on mutual information; A progressive cross-modal semantic interaction module is used to extract and fuse cross-modal features of HE and mIF images using a two-stage cross-attention mechanism; A dual-branch contrastive pre-training module is used to learn and optimize the model through mask autoencoder and global contrastive learning; The auxiliary loss calculation module is used to calculate the perceptual auxiliary loss and fine-grained contrastive loss to constrain model training.

Images