A gastric cancer claudin 18.2 expression multi-dimensional quantitative evaluation method and system based on bidirectional cross-modal virtual staining and double-anchored adaptive calibration
By employing a bidirectional cross-modal virtual staining and dual-anchored adaptive calibration method, the accuracy and stability issues of Claudin18.2 expression assessment in HE images were resolved, enabling multidimensional quantitative assessment of Claudin18.2 in gastric cancer, which is suitable for the precise diagnosis of gastric cancer.
Patent Information
- Application Number
- CN202611319149.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-28
- Publication Date
- 2026-11-10
AI Technical Summary
Existing automated assessment methods based on HE images using Claudin 18.2 cannot accurately quantify staining ratio, staining intensity, and membrane staining location, and lack absolute chromogenic references. This results in poor prediction stability across batches, susceptibility to interference from the heterogeneous microenvironment of gastric cancer, and high false positive/false negative rates, making it difficult to meet clinical diagnostic needs.
A bidirectional cross-modal virtual staining and dual-anchored adaptive calibration method was adopted. The complete cell membrane structure was extracted through the cell center spatial perception resampling mechanism, a Bi-VCAN network was constructed and structural consistency loss was introduced. Combined with the dual-anchored comparison mechanism of IHC positive and negative control and endogenous normal gastric mucosa, cross-modal feature mapping and adaptive calibration were realized to perform end-to-end multi-dimensional quantitative assessment.
It achieves precise quantitative localization of Claudin18.2 membrane expression location, cross-center standardized evaluation, reduces detection costs and cycle time, improves model interpretability and robustness, and is suitable for detection in advanced gastric cancer patients with limited tissue samples.
Smart Images

Figure CN123071304A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of digital pathological image processing and artificial intelligence-assisted analysis technology, specifically to a method and system for multidimensional quantitative assessment of Claudin18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchor adaptive calibration, belonging to the interdisciplinary field of pathomics and artificial intelligence. Background Technology
[0002] Claudin18.2 (Claudin18 isoform 2, CLDN18.2) is an important member of the tight junction protein family. In normal tissues, it is primarily expressed locally in gastric mucosal epithelial cells. However, during the development of gastric cancer and adenocarcinoma of the gastroesophageal junction, it is abnormally exposed and persistently expressed. In recent years, with the rapid development of targeted therapies targeting CLDN18.2, such as monoclonal antibodies (e.g., Zolbetuximab), antibody-drug conjugates (ADCs), and chimeric antigen receptor T-cell (CAR-T) therapy, CLDN18.2 has become an important biomarker for precision treatment of gastric cancer, and its expression level directly affects patient treatment decisions and clinical benefits.
[0003] Currently, immunohistochemistry (IHC) staining remains the gold standard for clinically assessing CLDN18.2 expression status. Pathologists typically interpret the results based on a comprehensive assessment of indicators such as the positive percentage (0–100%) of tumor cell membrane staining, staining intensity (0, 1+, 2+, 3+), and membrane staining pattern (lateral membrane, basolateral membrane, or intact ring membrane), and generate corresponding expression scores. However, IHC testing has several limitations: First, IHC testing requires additional tissue section preparation, the use of specific antibody reagents, and reliance on automated staining equipment and manual slide reading, which not only increases testing costs but also prolongs the pathological diagnosis cycle, making it difficult to routinely perform in some primary healthcare institutions. Furthermore, for puncture or endoscopic biopsy specimens from patients with advanced gastric cancer, the limited tissue volume makes performing multiple diagnostic and treatment-related immunohistochemical tests in large batches prone to tissue depletion, affecting subsequent molecular testing and treatment decisions.
[0004] Second, the interpretation of CLDN18.2 immunohistochemical results is highly subjective. In particular, the distinction between weak positive (1+) and moderate positive (2+), the identification of incomplete membrane staining patterns, and the assessment of areas with high spatial heterogeneity in tumors can lead to significant differences in consistency among different pathologists, which can easily cause fluctuations in scoring results and affect the stability and reproducibility of clinical decisions.
[0005] With the development of digital pathology and artificial intelligence technologies, methods for predicting immunohistochemical results based on H&E staining images have gradually attracted attention. Currently, related technologies mainly include weakly supervised classification methods based on convolutional neural networks (CNNs) or multiple instance learning (MIL), and image-translated virtual staining methods based on models such as Pix2Pix and CycleGAN. However, existing technologies still have the following shortcomings: First, because cell membrane boundaries and membrane structure features are not obvious in H&E images, the model lacks clear membrane structure information to guide it, which often only achieves coarse-grained positive and negative discrimination, and is difficult to output fine-grained quantitative indicators such as staining ratio, staining intensity and membrane localization required in clinical practice, resulting in poor interpretability.
[0006] Second, while image translation-based virtual staining methods can convert H&E images to IHC images, most methods employ a global mapping strategy, which easily leads to artifacts or false-positive staining in stromal areas, inflammatory infiltration areas, and non-tumor regions. Furthermore, existing methods typically fail to effectively decouple histological and staining features, making it difficult to accurately separate key features associated with CLDN18.2 expression. This is particularly problematic in mucinous adenocarcinoma, inflammatory cell infiltration, or areas with high spatial heterogeneity, where misdiagnosis is likely, thus failing to meet the needs of clinical diagnostic applications.
[0007] Third, in actual clinical immunohistochemical staining, to ensure the accuracy of staining results, pathologists usually place standard positive and negative control tissues (such as amnion rolls) next to the tissue sections to calibrate the differences in staining intensity between different batches, instruments, and staining conditions. However, existing artificial intelligence models for predicting IHC results based on H&E images lack a similar standard reference mechanism. When faced with image color differences from different medical institutions, different scanning equipment, and different H&E staining conditions, the model's prediction results are prone to quantitative drift, making it difficult to achieve cross-center standardization.
[0008] Fourth, existing models lack the ability to dynamically correct background noise and non-specific regions, and cannot perform real-time calibration of staining depth based on control tissues like professional pathologists. Therefore, they are prone to producing false positive or false negative prediction results in highly heterogeneous regions.
[0009] In summary, there is an urgent need for a method that can achieve multi-dimensional, accurate, and stable assessment of the expression status of Claudin18.2 in gastric cancer based on conventional H&E stained sections, in order to overcome the shortcomings of the existing technologies. Summary of the Invention
[0010] This invention designs a multidimensional quantitative assessment method and system for Claudin18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchored adaptive calibration. The technical problem it solves is that existing automated assessment methods for Claudin18.2 based on HE images cannot accurately quantify the three-dimensional clinical indicators of "staining ratio, staining intensity, and membrane staining location" due to the lack of dominant cell membrane color contrast in HE sections. At the same time, existing AI models lack absolute colorimetric references, have poor cross-batch prediction stability, and are easily affected by the heterogeneous microenvironment of gastric cancer, resulting in a high false positive / false negative rate, which is difficult to meet the needs of clinical diagnosis.
[0011] To solve the aforementioned technical problems, the present invention adopts the following solution: A multidimensional quantitative assessment method for Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchor adaptive calibration includes the following steps: Step 1: Obtain H&E stained full-scan slice images of gastric cancer patients, perform color normalization processing, and use a lightweight semantic segmentation network to extract the tumor parenchymal epithelial region and the patient's own normal gastric mucosal gland region from the image, and remove acellular regions; introduce a cell center spatial perception resampling mechanism, use cell nucleus detection algorithm to lock the centroid of each cell nucleus as a physical anchor point, and adaptively expand outward to extract local image patches containing complete cytoplasm and cell membrane boundaries; Step 2: Construct a bidirectional cross-modal virtual staining adversarial network Bi-VCAN as a feature extractor, which includes a generator G that transforms HE modality features into virtual Claudin18.2 IHC modality features. H →I, and the generator G that inversely restores the virtual IHC mode to the HE mode. I →H; Introduce structural consistency loss, use edge detection operators to extract cell boundary structure information in HE images, and constrain the high-response regions in the virtual IHC feature map to maintain spatial consistency with the cell membrane boundaries in the HE image; Step 3: Construct an expression intensity calibration space based on the dual-anchoring comparison mechanism of IHC positive and negative control and endogenous normal gastric mucosa; During the training phase, the amniotic roll control region on the IHC slice is simultaneously extracted as the exogenous highest intensity anchor point, and the patient's own normal gastric mucosa region is used as the endogenous normal expression intensity anchor point; Map the amniotic roll IHC features to a global strong positive benchmark feature vector v max The IHC features of normal gastric mucosa were constructed into a normal mucosal physiological baseline vector group v. n Using paired HE-IHC image patches of normal gastric mucosa from patients as cross-modal bridging samples, a cross-modal mapping relationship is established, enabling the model to learn and solidify the absolute colorimetric standard jointly defined by the amnion roll and normal gastric mucosa. Step 4: Input the high-order cross-modal transformation features extracted in Step 2 into the decoupled multi-task head, and map them to the staining intensity subspace S respectively. i , membrane staining location subspace S l and the coloring ratio subspace S r This method enables the joint prediction of Claudin18.2 expression intensity, membrane localization pattern, and the proportion of positive cells; orthogonal constraint loss is introduced to reduce information coupling between the three tasks. Step 5: Use a geometrically adaptive joint loss supervision mechanism to perform end-to-end joint training on Bi-VCAN, the dual-anchor calibration module, and the multi-task head; Step 6: In actual clinical inference, the HE-enhanced whole-scan slices of gastric cancer from the test patient without exogenous amniotic membrane rolls are input into the trained model. The model automatically locates the endogenous normal gastric mucosal gland regions in the slices and extracts their HE features. Based on the cross-modal mapping relationship established in Step 3, the individualized intensity reference scale corresponding to the current slice is dynamically restored, realizing online adaptive calibration across batches and devices. A sliding window strategy is used for intensive inference of the entire WSI tumor parenchyma region. The prediction results of all image blocks are aggregated at the whole-slice level through a self-calibrated spatial topology integrator, outputting the final Claudin 18.2 staining intensity level, positive cell ratio, and main membrane expression pattern of the entire WSI, and generating a standardized quantitative scoring report.
[0012] Preferably, the cell center spatial awareness resampling mechanism in step 1 is as follows: the HoVer-Net cell nucleus detection algorithm is used to identify all cell nuclei in the tissue region and extract their centroid coordinates; the cell nucleus centroid is used as a physical anchor point to adaptively expand outward to form a local sampling region, and the image block size is dynamically adjusted according to the local cell density. The dense cell region uses a 256×256 pixel image block, and the sparse cell region uses a 512×512 pixel image block. The color normalization process described in step 1 specifically involves: converting the original RGB image to optical density space, extracting the principal component staining matrix through singular value decomposition, and mapping the staining features of the original image to a preset standard reference gastric cancer image.
[0013] Preferably, the structural consistency loss in step 2 includes edge consistency loss L. SC And skeleton consistency loss L SK : ; ; ; Where N is the number of image patch samples in a training batch. This represents the edge map extracted from the i-th HE image patch. This represents the edge map extracted from the i-th virtual IHC image patch. This represents the skeleton map extracted from the i-th HE image patch. This represents the skeleton map extracted from the i-th virtual IHC image patch. ‖·‖1 is the L1 norm, which is the sum of the absolute values of all elements; ‖·‖2² is the square of the L2 norm, which is the sum of the squares of all elements.
[0014] Preferably, the dual-anchoring comparison mechanism in step 3 specifically involves: dividing the amniotic roll image block X... A Deep features extracted using a shared encoder: By anchoring losses Map it to the global strong positive prototype vector v max X-ray a normal gastric mucosa image block N Extracting deep features By anchoring losses ;
[0015] Map it to the normal mucosal physiological reference vector group v n The combined loss of dual anchoring is: .in, f A It is a depth feature representation of the amniotic roll image patch; E shared It is an encoder network that shares modes for HE and IHC; Proj It is a reference object aligned with the projection layer; v max It is a globally strong positive prototype vector; f N (i) is the depth feature of the i-th normal gastric mucosal image patch; ||·|2² is the square of the L2 norm, i.e., the square of the Euclidean distance.
[0016] Preferably, the cross-modal mapping relationship in step 3 is achieved through anchoring constraint loss based on margin sorting: ; Where m is the sorting margin parameter, d(·,·) is the Euclidean distance metric, and v t Let v be the feature vector of the gastric cancer region to be predicted. n v is the normal physiological reference prototype vector of gastric mucosa. max Σ is the global strong positive prototype vector. t This sums all gastric cancer samples in the current training batch. max(0, ·) is the cutoff function, taking the larger value between 0 and the value in parentheses.
[0017] Preferably, the decoupled multi-task header in step 4 uses three task-specific projection networks to share the feature F. shared Mapped to three subspaces: f I = P I(F shared ) ∈S I ; f L = P L (F shared ) ∈S L ; f R = P R (F shared ) ∈S R ; Among them, P I P L P R These represent mappings to the staining intensity subspace S, respectively. I , membrane expression location subspace S L and positive proportion subspace S R Projection network; f I f L f R These are task-specific feature vectors mapped to each subspace; The orthogonal constraint loss is : Where the superscript T denotes vector transpose, ||·|| F 2 This represents the square of the Frobenius norm, which is the sum of the squares of the elements in the matrix.
[0018] Preferably, the staining intensity prediction in step 4 uses a Softmax classifier: pi = softmax(Wi⋅fi + bi); The intensity classification loss is the cross-entropy loss: ; The membrane expression pattern prediction uses a multi-class classifier: p l = softmax(W l · f l + b l ); The membrane pattern classification loss is: ; The positive cell proportion prediction was performed using the Smooth L1 loss function: L R = SmoothL1(r pred -r gt ); SmoothL1(x) = { 0.5·x², if |x| < 1; |x| - 0.5, otherwise}; The combined loss from multiple tasks is: W ib is the weight matrix of the intensity classifier; i It is the bias vector of the intensity classifier; p i W is the output intensity category probability vector; l This is the weight matrix of the membrane pattern classifier; b l It is the bias vector of the membrane pattern classifier; p l It is the output membrane pattern category probability vector; r pred It is the positive rate predicted by the model; r gt It represents the true positive rate; SmoothL1 is the smoothed L1 loss function; λ I , λ L , λ R , λ O These are the weighting coefficients for the losses of each subtask.
[0019] Preferably, the overall loss function of the geometrically adaptive joint loss supervision mechanism in step 5 is: ;
[0020] Among them, L GAN To generate adversarial loss across modalities, L Cycle For the cycle consistency loss, L DACIC For the dual-anchored expression intensity calibration loss, L Task For multi-task joint loss, L Heter This is a heterogeneous adaptive loss; The heterogeneity adaptive loss dynamically adjusts the training weights based on the cell density, tissue complexity, and model prediction uncertainty of the image patch: ; ; ; Among them, h i D is the comprehensive heterogeneity score for the i-th image patch. i As a cell density indicator, C i U is an indicator of organizational structure complexity. i To represent the model's prediction uncertainty, α, β, and γ are weighting coefficients that satisfy α + β + γ = 1, w i Let be the training weights for the i-th image patch; L i It represents the basic task loss for the i-th image patch.
[0021] Preferably, the self-calibrating spatial topology integrator in step 6 performs whole-patch aggregation of image patch prediction results in the following manner: A topology graph structure is constructed based on the spatial adjacency relationships between image patches, and spatial smoothing and topology fusion are performed on the prediction results of adjacent image patches. ; The percentage of positive cells in the entire WSI was: ;
[0022] The final staining intensity level is determined using a confidence-weighted topology mapping network: ; Where, p i Let w be the original prediction vector of the i-th image patch, and N(i) be the set of image patches adjacent to the i-th image patch. ij Here, adjacency weights are based on spatial distance, η is a smoothing coefficient, and conf i For the model prediction confidence of the i-th image patch, area i N represents the proportion of the area of the i-th image patch to the total area of the tumor region. i positive N is the number of tumor cells with positive membrane expression in the i-th image patch; i total It is the total number of tumor cells in the i-th image patch; R WSI It represents the percentage of positive cells across the entire WSI layer.
[0023] A multidimensional quantitative assessment system for Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchor adaptive calibration includes: The image preprocessing module is used to normalize the color of the acquired H&E stained full scan slice images of gastric cancer patients, extract the tumor parenchymal epithelial region and the patient's own normal gastric mucosal gland region, and extract local image patches containing intact cytoplasm and cell membrane boundaries through the cell center spatial perception resampling mechanism. A bidirectional cross-modal virtual staining feature extraction module is used to construct a Bi-VCAN network, which transforms the HE modality into higher-order features of the virtual Claudin18.2 IHC modality, and constrains the virtual staining signal to expand along the real physical boundary of the cell membrane through structural consistency loss. The dual-anchored expression intensity calibration module is used to introduce the amniotic roll control region as the exogenous highest intensity anchor point and the patient's own normal gastric mucosa as the endogenous normal expression intensity anchor point during the training phase, establish a cross-modal mapping relationship, and dynamically restore the individualized intensity reference scale of the current slice during the inference phase. A multi-dimensional decoupled multi-task decision module is used to map cross-modal transformation features to the staining intensity subspace, membrane staining location subspace, and staining ratio subspace, respectively, to achieve joint prediction of Claudin18.2 expression intensity, membrane localization pattern, and positive cell ratio; The geometrically adaptive joint loss supervised training module is used for end-to-end joint training of the entire network; The self-calibrated spatial topology integration module is used to perform sliding window dense inference on the entire WSI in actual clinical reasoning, perform whole-area topological aggregation and confidence-weighted fusion of prediction results for all image patches, and output a standardized quantitative scoring report.
[0024] This method and system for multidimensional quantitative assessment of Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchor adaptive calibration has the following beneficial effects: (1) This invention addresses the technical challenge of lacking dominant cell membrane color contrast in H&E stained sections. It adaptively extracts the complete cell membrane structure by using a cell center spatial perception resampling mechanism with the cell nucleus centroid as the anchor point. Combined with the structural consistency loss, it forcibly constrains the virtual IHC colorimetric signal to expand along the real physical boundary of the cell membrane. This effectively solves the problem that existing models can only achieve coarse-grained positive and negative discrimination due to the lack of membrane structure information guidance. For the first time, it has achieved precise quantitative localization of Claudin18.2 membrane expression location, providing clinical cell-level fine assessment capabilities.
[0025] (2) This invention constructs a dual-anchoring comparison mechanism based on IHC positive and negative control (amniotic membrane roll) and endogenous normal gastric mucosa. During the training phase, the IHC features of the amniotic membrane roll are solidified into the network parameters. During the inference phase, the individualized intensity reference scale is dynamically restored using the patient's own normal gastric mucosa. This invention creatively simulates the interpretation logic of pathologists based on the control tissue to calibrate the staining depth. This ensures that the prediction results of the model no longer undergo quantitative drift under different medical institutions, different scanning equipment and different staining conditions. This invention is the first to realize cross-center standardized assessment based on H&E images.
[0026] (3) This invention decouples the multi-task head in multiple dimensions and separates the cross-modal conversion features into the staining intensity subspace, membrane staining location subspace and staining ratio subspace through the orthogonal constraint mechanism. This enables the joint and accurate prediction of three-dimensional clinical indicators of Claudin18.2 staining intensity level, membrane positioning pattern and positive cell ratio. Each subspace does not interfere with each other and focuses on the feature patterns related to its own task, which significantly improves the interpretability of the model.
[0027] (4) The present invention uses a geometrically adaptive joint loss supervision mechanism to dynamically adjust the training weights based on the cell density, tissue structure complexity and model prediction uncertainty of the image patch, so that the model focuses on the tumor parenchyma region with dense cells and complex structure, automatically reduces the interference of mucus lake, inflammatory infiltration and stroma region, and significantly improves the robustness and generalization ability of the model in the highly heterogeneous microenvironment of gastric cancer.
[0028] (5) The present invention is fully automated end-to-end processing from H&E image input to standardized report output in the entire processing flow. The inference stage only requires conventional H&E slices, without the need for additional IHC slice preparation or the use of specific antibody reagents. The detection cost is greatly reduced and the detection cycle is shortened from the traditional 3 to 5 days to several hours. It is especially suitable for puncture or endoscopic biopsy specimens of advanced gastric cancer patients with limited tissue samples, avoiding the problem of tissue exhaustion caused by multiple immunohistochemical tests being carried out in parallel. Attached Figure Description
[0029] Figure 1 This is a schematic diagram of the cell center spatial perception resampling and dual-region segmentation process in step 1 of the present invention; Figure 2 This is a schematic diagram of the bidirectional cross-modal virtual staining adversarial network (Bi-VCAN) structure in step 2 of the present invention; Figure 3 This is a schematic diagram of the calibration space in step 3 of the present invention, based on the dual-anchoring comparison mechanism of IHC positive and negative control and endogenous normal gastric mucosa. Figure 4 This is a flowchart of the overall process of the method of the present invention.
[0030] HE—Hematoxylin and Eosin (HE); IHC—Immunohistochemistry (IHC); WSI—Whole Slide Image (WSI); H&E—Hematoxylin and Eosin. Detailed Implementation
[0031] The following is combined Figures 1 to 4 The present invention will be further described as follows: like Figures 1 to 4 As shown, this embodiment provides a multi-dimensional quantitative scoring method and system for gastric cancer Claudin18.2 H&E images based on bidirectional cross-modal virtual staining and dual-anchor adaptive calibration. The overall process includes: cell-center spatially perceived resampling preprocessing and dual-region segmentation, as follows... Figure 1 As shown; bidirectional cross-modal virtual staining feature extraction, such as Figure 2 As shown; expression intensity calibration based on IHC anion / positive control and normal gastric mucosa dual anchoring, such as Figure 3 As shown; the steps include multi-dimensional decoupling of multi-task decision-making, geometrically adaptive joint loss supervised training, and output of a complete WSI-level score report, such as... Figure 4 As shown.
[0032] This embodiment uses the gastric cancer Claudin18.2 expression prediction task as an example for illustration.
[0033] Step 1: Cell center spatial perception resampling preprocessing and adjacent / gastric cancer cell dual-region segmentation.
[0034] like Figure 1 As shown, firstly, whole-slide images (WSI) of H&E stained sections from gastric cancer patients were acquired, and preprocessing with spatially sensed resampling at the cell center and dual-region segmentation of adjacent / gastric cancer was performed. H&E whole-slide images and corresponding Claudin 18.2 immunohistochemical sections from gastric cancer patients confirmed by pathologists were collected.
[0035] First, the original H&E images are color-normalized by converting the RGB images to the Optical Density (OD) space. Principal staining vectors are extracted through singular value decomposition to establish a staining matrix. The staining distribution of the original images is then mapped to a pre-defined standard reference gastric cancer image, thereby eliminating color deviations caused by different batches, scanners, and staining conditions. The specific calculation process for color normalization is as follows: Let the RGB pixel value matrix of the original H&E image be I∈R (H×W×3) Convert it to OD space: OD = -log 10 (I / I0), where I0 is the incident light intensity, H is the image height (number of pixels), W is the image width (number of pixels), and R... (H×W×3) It represents an H×W×3-dimensional real space matrix.
[0036] Singular value decomposition of the OD space matrix: OD = U·Σ·V T ; Where U is the left singular matrix, Σ is the singular value diagonal matrix, and V T This is the transpose of a right singular matrix, with the superscript T indicating the matrix transpose operation.
[0037] Extract the first two principal components as coloring vectors, and construct the coloring matrix M∈R. (2×3) Map the original image to a predefined standard reference coloring matrix M. ref :I norm =I o ·exp(-OD·(M T ·M ref )); Among them, I norm For the color-normalized image, exp represents the natural exponential function, and M... T M is the transpose of the coloring matrix M. ref The preset standard reference coloring matrix has a dimension of 2×3.
[0038] Background removal and tissue region detection are performed on the color-normalized image. Specifically, grayscale conversion combined with adaptive threshold segmentation is used to remove the slide background area, and the Otsu algorithm is used to automatically separate the tissue foreground and background. At the same time, morphological closing operations are used to remove small noise areas, retaining only the real tissue areas.
[0039] This embodiment employs a lightweight semantic segmentation network to segment the tissue region, separately identifying the tumor parenchymal epithelial region and the patient's own normal gastric mucosal glandular region, and automatically removing large areas of acellular mucus lakes, stroma, adipose tissue, necrotic debris, and slide background areas. The lightweight semantic segmentation network uses an encoder-decoder structure, with the encoder using MobileNetV2 as the backbone network, and the decoder employing bilinear interpolation upsampling and skip connections to recover the full-resolution segmentation result.
[0040] Considering that traditional fixed-size image slicing can easily lead to incomplete cell structures, this invention further proposes a cell center spatially aware resampling mechanism. Specifically, firstly, a HoVer-Net-based cell nucleus detection algorithm is used to identify all cell nuclei in the tissue region and extract their centroid coordinates; then, using the cell nucleus centroid as a physical anchor point, an adaptive outward expansion is used to form a local sampling region, ensuring that each image patch contains complete cell nuclei, cytoplasm, and cell membrane structures. Let the cell nucleus centroid coordinates be (c x , c y The adaptive expansion radius R is determined by the local cell density ρ. ; Where R0 is the base radius, in pixels, preferably set to 64 in this embodiment; λ is an adjustment coefficient, dimensionless, preferably set to 32 in this embodiment; ρ0 is the reference density unit: pixels / square millimeter, preferably set to 100 in this embodiment; ρ is the density unit in units of (c x ,c y The density of cell nuclei in the central local neighborhood is expressed as cells per square millimeter. The image patch size is dynamically adjusted based on the local cell density: a 256×256 pixel image patch is used for densely celled regions, where ρ > ρ0. high , where ρ high For high-density thresholding, this embodiment preferably sets it to 150 cells / mm²; for sparsely populated areas, a 512×512 pixel image block is used, where ρ < ρ low , where ρ low As a low density threshold, it is preferably set to 50 per square millimeter in this embodiment, so as to take into account both the details of the microstructure and the macroscopic organizational context information.
[0041] Finally, the obtained image patches are randomly divided into training, validation, and test sets. The training set is used for model parameter optimization, the validation set is used for model hyperparameter tuning, and the test set is used to evaluate model performance. In this embodiment, the ratio of the training, validation, and test sets is 7:1.5:1.5.
[0042] Step 2: Construct a feature extractor based on a bidirectional cross-modal virtual staining adversarial network (Bi-VCAN): like Figure 2 As shown, this embodiment constructs a bidirectional cross-modal virtual staining adversarial network Bi-VCAN, whose network structure includes a HE to IHC direction generator G. H →I, IHC to HE direction generator G I →H and the corresponding discriminator D H and D I Generator G H →I and G I →H all adopt a fully symmetrical encoder-residual learning-decoder structure, D H and D I The discriminator uses the PatchGAN local discriminant network to ensure that the virtual generated image has high realism in both the overall tissue structure and the details of local membrane expression.
[0043] The generator first inputs a 256×256 pixel HE image patch into the encoder. The encoder consists of a convolutional layer with a kernel size of 7×7 and 64 output channels, followed by two downsampled convolutional layers with kernel sizes of 3×3, strides of 2, and 128 and 256 output channels respectively. Each convolutional layer is then followed by an Instance Normalization layer and a LeakyReLU activation function with a negative slope of 0.2 to improve the model's robustness to color differences between different staining batches.
[0044] The encoder produces a high-dimensional latent feature representation of size 64×64×256, which is then input into a deep feature extraction network consisting of nine residual modules. Each residual module includes two 3×3 convolutional layers and uses a residual skip connection method to directly add the input features to the convolution output. The calculation process is as follows: ;
[0045] Where IN represents Instance Normalization, σ represents the LeakyReLU activation function, and the negative slope is set to 0.2; F i F represents the input feature of the i-th residual module, with dimensions of 64×64×256; i+1Let represent the output feature of the i-th residual module (with the same dimension as the input), and Conv1 and Conv2 represent the 3×3 convolution operations of the first and second layers, respectively. Through multi-layer residual learning, the membrane structure texture information related to the Claudin18.2 expression can be fully extracted, while avoiding the gradient vanishing problem during deep network training.
[0046] The derived high-dimensional features are fed into the decoder for cross-modal reconstruction. The decoder consists of two deconvolutional upsampling modules with output channels of 128 and 64 respectively. Finally, a 7×7 convolutional layer restores the image to a three-channel virtual Claudin18.2 immunohistochemical image. The Tanh function is then used to output the generated result. Simultaneously, the inverse generator G... I →H uses the same network structure to achieve reverse reconstruction from virtual IHC images to HE images, ensuring the complete preservation of histological structural information during cross-modal conversion.
[0047] To improve the realism of virtual immunohistochemistry images, this embodiment uses a 70×70 PatchGAN as the discriminator structure. The discriminator consists of a five-layer convolutional network, with the number of output channels in each layer set to 64, 128, 256, 512, and 1, respectively. Each convolutional layer uses the LeakyReLU activation function with a negative slope of 0.2, and all layers except the first layer use InstanceNormalization normalization. The discriminator outputs the probability that the corresponding local image patch belongs to the real immunohistochemistry image, thereby constraining the generator to learn the real Claudin18.2 staining texture distribution and improving the realism of local details in the virtual staining image.
[0048] Considering the lack of clear membrane staining information in HE images, and the fact that Claudin18.2 expression is mainly located in the cell membrane, this embodiment further designs a Structure Consistency Loss to constrain the virtual IHC stained area to maintain consistency with the real cell membrane boundary in the HE image. Specifically, the Sobel operator is used to extract edge structures from both the HE image and the virtual IHC image, obtaining the corresponding edge maps S. H With S I The structural consistency loss is defined as: ;
[0049] Where N is the number of image patch samples in a training batch, a dimensionless integer. In this embodiment, the batch size is set to 16. This represents the edge map extracted from the i-th HE image patch. This represents the edge map extracted from the i-th virtual IHC image patch. This represents the L1 norm, which is the sum of the absolute values of all elements.
[0050] To enhance the continuity of the membrane structure, the skeleton was extracted from the edge map to obtain HE skeleton map B. H and virtual IHC skeleton diagram B I Constructing a skeleton consistency loss: ;
[0051] in, This represents the skeleton map extracted from the i-th HE image patch. This represents the skeleton map extracted from the i-th virtual IHC image patch. This represents the square of the L2 norm (i.e., the sum of the squares of the elements).
[0052] The final structural constraint loss is defined as: ;
[0053] Wherein, 0.5 is the weighting coefficient (dimensionless) for skeleton consistency loss, used to balance the contributions of edge consistency loss and skeleton consistency loss.
[0054] Through the above structural constraints, the model can effectively avoid spatial drift of the virtual immunohistochemical staining region, ensuring that the predicted membrane expression is strictly distributed continuously along the real cell boundary, thereby significantly improving the model's ability to learn Claudin18.2 membrane localization features.
[0055] This embodiment also employs adversarial loss and cycle consistency loss to jointly optimize the cross-modal conversion process, with the adversarial loss used to improve the realism of the generated images: ;
[0056] Among them, G H→I D represents the generator in the HE to IHC direction. I The discriminator representing the IHC mode, x I x represents a real IHC image patch sample. H pdata(x) represents HE image patch samples. I pdata(x) represents the data distribution of a real IHC image patch. H ) represents the data distribution of the HE image patch, E represents the expected value (i.e., the mean), log represents the natural logarithm, and D I (x I ) represents the discriminant's probability output for classifying a real IHC image, with a value ranging from 0 to 1. H→I (x H This indicates the virtual IHC image obtained by the generator from the HE image.
[0057] Cyclic consistency loss is used to ensure that HE images can still recover the original tissue structure after virtual immunohistochemical transformation and inverse reconstruction. ;
[0058] Among them, G I→H G represents the generator in the IHC to HE direction. I→H (G H→I (x H )) indicates that the HE image x H The result of converting to a virtual IHC and then reconstructing it into an HE image, G H→I (G I →H(x I )) represents the actual IHC image x I The result of converting to a virtual HE image and then reconstructing it into an IHC image in the forward direction, ||·||1 is the L1 norm.
[0059] The combined effect of the aforementioned losses enables Bi-VCAN to not only accurately learn the cross-modal mapping relationship between HE and Claudin18.2 immunohistochemistry, but also to ensure the consistency of tissue structure, cell membrane position and chromogenic features, providing stable and reliable high-level semantic features for subsequent dual-anchoring intensity calibration and multi-dimensional quantitative scoring.
[0060] Step 3: Construct an expression intensity calibration module (DACIC) based on a dual-anchoring comparison mechanism between IHC positive / negative control and endogenous normal gastric mucosa; like Figure 3 As shown, this embodiment addresses the problem that existing models based on HE images for predicting immunohistochemical results lack a unified colorimetric reference standard and are difficult to adapt to different staining batches and scanning conditions. It proposes an expression intensity calibration strategy (DACIC) based on a dual-anchoring comparison mechanism between IHC positive / negative control and endogenous normal gastric mucosa. Specifically, during the model training phase, spatially registered HE image blocks, Claudin18.2 IHC image blocks, and standard control tissue regions with accompanying staining from immunohistochemical sections are simultaneously acquired for each patient. The standard control tissue is preferably amniotic roll tissue, which simultaneously contains stable negative and strongly positive regions, serving as an absolute intensity reference for Claudin18.2 immunohistochemical staining.
[0061] First, the amniotic roll region is automatically identified and extracted from a real Claudin18.2 IHC slice, and then input into the Bi-VCAN feature extractor constructed in step 2 to obtain the corresponding depth feature representation. Specifically, the amniotic roll image patch is represented as X. A The deep features extracted by the encoder shared by Bi-VCAN in step 2 are represented as follows: ; Among them, X A This represents an RGB image tensor with dimensions of 256×256×3, representing an amnion roll image patch. E sharedThis represents the encoder network in Bi-VCAN that shares the HE and IHC modes, with an output feature dimension of 64×64×256. f A The depth feature representation of the amniotic roll image patch has dimensions of 64×64×256.
[0062] Subsequently, an independent reference alignment projection layer is set up in the network shared latent space, and the deep features of the amniotic roll IHC are mapped to the global strong positive prototype vector v through the anchoring loss function. max And use it as the upper limit of absolute color intensity for the entire model: ;
[0063] Where Proj represents the reference alignment projection layer, implemented by a single fully connected layer, with an input dimension of 64×64×256 and an output dimension of 256; v max This is a learnable global strong positive prototype vector with a dimension of 256, updated via backpropagation during training. This represents the square of the L2 norm, which is the square of the Euclidean distance.
[0064] Simultaneously, normal gastric mucosal gland regions were extracted from IHC slices of the same patient to obtain the physiological expression characteristics of normal gastric mucosa under the IHC modality, and these characteristics were mapped to a shared latent space to form a normal gastric mucosal physiological reference vector group v. n Specifically, let X represent a normal gastric mucosal image patch. N The feature representation in their shared latent space is defined as: ;
[0065] in, X N This represents a normal gastric mucosa image patch, an RGB image tensor with dimensions of 256×256×3; f N The depth features of a normal gastric mucosa image block are represented, with dimensions of 64×64×256.
[0066] Anchoring loss constraint clusters normal gastric mucosal features into physiologically expressed reference regions: ;
[0067] Where N represents the number of normal gastric mucosa samples in the current training batch (a dimensionless integer). V represents the depth features of the i-th normal gastric mucosa image patch. n It is a learnable normal gastric mucosal physiological reference prototype vector (dimension 256, updated through backpropagation during training).
[0068] By jointly optimizing the above anchoring loss: ;
[0069] A stable dual-anchored expression intensity reference space can be established.
[0070] Since normal gastric mucosa is naturally present in all patient tissues, and Claudin18.2 has stable physiological expression characteristics, it can serve as a cross-modal bridging reference between the training and inference phases. Based on this, this embodiment constructs a same-modal intensity comparison matrix, performing pairwise comparisons between amniotic roll IHC features and normal gastric mucosa IHC features. According to their distance and ranking relationships in the shared latent space, a continuous IHC chromogenic intensity ranking baseline V is automatically established. IHC_S The intensity baseline describes the continuous variation from normal physiological expression to absolutely strong positive expression. The amniotic roll positive prototype vector v is used as an example. max Compared with the reference vector v of normal gastric mucosa n Constructing continuous expression directions: ;
[0071] Where d represents the unit direction vector (dimension 256) pointing from the normal gastric mucosa reference vector to the global strong positive prototype vector, ||v max -v n ||2 represents the L2 norm (i.e., Euclidean distance) of the difference between two vectors.
[0072] For any region of gastric cancer to be predicted, its shared potential spatial features are represented as v t (With a dimension of 256), its projection coefficients along the continuous expression direction are calculated as follows: ;
[0073] Where, α t The scalar projection coefficients (dimensionless real numbers) represent the feature v to be measured. t In the case of v n Pointing to v max Relative position in direction, (v t - v n )·d represents the vector (v) t - v n The dot product (inner product) operation between the vector and the unit direction vector d.
[0074] According to the projection coefficient α t This allows for the establishment of a continuous IHC colorimetric intensity ranking baseline V. IHC_S This enables a unified quantitative representation of expression intensity across different patients.
[0075] To transfer the intensity criteria from the IHC end to the HE feature space, this embodiment further uses paired HE-IHC image patches of normal gastric mucosa from patients as cross-modal bridging samples, and designs a cross-modal asymmetric intensity constraint loss. This loss requires that the gastric cancer feature vector to be predicted maintain the correct relative positional relationship with the dual anchoring benchmarks (i.e., the strong positive prototype and the normal mucosa prototype) in the shared latent space; that is, the gastric cancer feature vector should be closer to the global strong positive prototype vector than the normal mucosa reference vector.
[0076] This embodiment uses anchorage constraint loss based on margin sorting, and its expression is: ; Where, Σ t : Summing all gastric cancer samples in the current training batch. In this embodiment, the batch size is 16; max(0, ·): truncation function, the loss is 0 when the value in parentheses is less than or equal to 0, and takes the positive value when it is greater than 0; m: sorting margin parameter, in this embodiment, m = 0.5 is preferably set to ensure the discriminative power of the feature vectors; d(·, ·) is the distance metric function in the feature space, in this embodiment, Euclidean distance (L2 distance) is preferably used; v t The gastric cancer regions to be predicted share a latent spatial feature vector with a dimension of 256; v n : Normal gastric mucosa physiological reference prototype vector, dimension 256; v max : Global strong positive prototype vector, dimension 256, generated by amniocentesis; The core constraint logic of the above formula is: for any gastric cancer sample feature v t The model needs to be forced to conform to the strongly positive prototype v. max The distance is significantly smaller than its distance to the normal mucosal prototype v n The distance between them must be greater than the margin m. If v t Distance v n Too close and distance v max If the distance is too great, the loss function outputs a positive value and generates a gradient, driving the model to adjust its parameters so that the relative ranking of virtual IHC features is consistent with the chromogenic gradient of real IHC from normal to strongly positive. By minimizing the above loss, the model successfully solidifies the cross-modal absolute chromogenic standard of "amniotic roll - normal gastric mucosa" into the network parameters, achieving a stable migration of the IHC intensity standard to the HE modality.
[0077] Furthermore, this embodiment employs both anchoring loss and ranking consistency loss to optimize the DACIC module, and its joint loss function is defined as: ;
[0078] Wherein, λ1 and λ2 represent the dimensionless weighting coefficients of anchoring loss and ranking consistency loss, respectively, used to balance the contributions of the two losses. In this embodiment, they are preferably set to λ1=1.0 and λ2=2.0.
[0079] After the model is trained, only a standard HE full-scan slide from the patient needs to be input. The model can automatically identify normal gastric mucosal regions in the slide and dynamically restore the individualized intensity reference scale of the current slide based on the cross-modal mapping relationship learned during the training phase. This enables stable prediction of Claudin 18.2 staining intensity under different slide batches, different scanning equipment, and different tissue heterogeneity backgrounds. Through the above-mentioned dual-anchoring calibration mechanism, this invention effectively solves the problems of traditional artificial intelligence models, such as lack of absolute colorimetric standards, insufficient cross-center generalization ability, and easy drift of quantitative results.
[0080] During the inference phase, the model no longer requires amniotic roll images. The absolute strength upper bound of the amniotic roll definition has been fixed in the network parameters v during the training phase. max In the process, the inference only utilizes the patient's own normal gastric mucosal characteristics and combines them with the solidified mapping relationship to restore the individualized intensity scale.
[0081] Step 4: Multi-dimensional decoupling and multi-task header determination After obtaining the high-order cross-modal transition features output in step 2, this embodiment constructs a multi-dimensional decoupled multi-task head to jointly determine the Claudin18.2 expression state. First, the cross-modal transition features are input into the feature decoupling module, and the shared features are further separated into three spaces through an orthogonal constraint mechanism, thereby reducing feature interference between different tasks and improving model interpretability.
[0082] Let F be the shared cross-modal feature output from step 2. shared (Dimensions are 64×64×256), and are mapped to the chromatic intensity subspace S through three task-specific projection networks. I Membrane expression location subspace S L and positive proportion subspace S R Their mathematical representations are as follows: f I = P I (F shared ) ∈S I ; f L = P L (F shared ) ∈S L ; f R = P R (F shared ) ∈S R ; Among them, P I P L P R These represent the feature projection networks for the three tasks: each consists of two fully connected layers, with a hidden layer dimension of 256 and an output dimension of 128. I f L f R represents the task-specific feature vectors mapped to the staining intensity subspace, membrane expression location subspace, and positive proportion subspace, respectively, all with a dimension of 128.
[0083] To reduce information coupling among the three tasks and improve the independence of features learned in each subspace, this embodiment further introduces orthogonal constraint loss, the expression of which is: ; Where the superscript T denotes vector transpose, ||·|| F 2 f represents the square of the Frobenius norm, which is the sum of the squares of the elements in the matrix. I f L and f R These are the feature vectors mapped by the task-specific projection network in step 4, each with a dimension of 128. This loss term minimizes the absolute value of the dot product between feature vectors in different subspaces, making the features of each task as orthogonal as possible. This ensures that the three task branches—stain intensity determination, membrane pattern recognition, and positive ratio prediction—do not interfere with each other, and each focuses on the feature patterns relevant to its own task.
[0084] In the color intensity subspace S i In the process, the system first calculates the feature vector v of the gastric cancer cells to be tested. t With the global strong positive prototype vector v max Normal gastric mucosa reference vector group v n and the IHC intensity ranking baseline V established in step 3 IHC_S The relative distance between them. Subsequently, based on the projection position of the feature to be tested in the intensity baseline, combined with indicators such as local membrane color density, color area ratio, and color continuity, a sample-specific intensity discrimination boundary is dynamically constructed, and Claudin 18.2 staining intensity levels of 0, 1+, 2+, or 3+ are output. The staining intensity prediction uses a Softmax classifier, and its prediction results are expressed as follows: pi = softmax(Wi⋅fi + bi); Among them, W i This represents the weight matrix of the intensity classifier, with dimensions 4×128, where 4 corresponds to the four intensity categories, b i f represents the bias vector, which has a dimension of 4. i For the task-specific features of the input intensity subspace in step 4 (dimension 128), pi The output is a four-dimensional probability vector, with each element ranging from 0 to 1, and the sum is 1, corresponding to the probabilities of four intensity levels: 0, 1+, 2+, and 3+.
[0085] softmax is a normalized exponential function: ;
[0086] Where z is the input vector of the softmax function, k is the class index, representing the k-th output class, and zk is the k-th component of the input vector z, i.e., the k-th class score / logit of the classifier output; e zk It is an exponential function with the natural constant e as the base and zk as the exponent; softmax(z)k is the normalized probability of the kth class.
[0087] The corresponding intensity classification loss is defined as cross-entropy loss: ;
[0088] Among them, y k True category labels are represented by one-hot encoding, where the dimension corresponding to the true category is set to 1, and the rest are set to 0. k p represents the probability that the model predicts the class as class k. I The k-th component, log represents the natural logarithm.
[0089] In the membrane staining location subspace S l In this embodiment, a morphological prior constraint algorithm is introduced. First, the edges of potential chromogenic regions are located in the virtual IHC feature map. Then, the geometric distance, spatial azimuth, and boundary continuity index between the edge of the chromogenic region and the corresponding cell nucleus centroid are calculated. Simultaneously, combined with cell contour segmentation results, the coverage pattern of the chromogenic region along the cell contour is analyzed, thereby automatically identifying the expression of apical membrane types, lateral membrane types, basal membrane types, and complete circumferential membrane types. Membrane expression pattern prediction is also completed using a multi-class classifier: p l = softmax(W l · f l + b l ) ; Among them, W l The weight matrix for the membrane pattern classifier has a dimension of 4×128, where 4 corresponds to the four membrane expression modes; b l f represents the bias vector, which has a dimension of 4. l The input membrane location subspace in step 4 is a task-specific feature with dimension 128 and p. L This is the output four-dimensional probability vector.
[0090] The classification loss corresponding to the membrane expression pattern is defined as the cross-entropy loss: ;
[0091] Among them, z k Represents the true membrane expression pattern label, one-hot encoded; q k This represents the probability that the model predicts the membrane mode to be of type k.
[0092] Regarding the staining ratio and cell counting subspace S R This embodiment employs a cell-level discrimination method based on an instance-level cell aggregation strategy. First, a cell instance segmentation algorithm is used to obtain the outlines of all tumor cells within an image patch. Then, each cell is analyzed to determine the presence of positive membrane expression features, and the number of positive tumor cells is compared to the total number of tumor cells. Finally, the percentage of positive tumor cells within a single image patch is calculated: r = N positive / N total ; Where, N positive N represents the number of tumor cells with positive membrane expression within an image patch, a dimensionless integer; total The value of r represents the total number of tumor cells within the image patch, a dimensionless integer; the value of r represents the proportion of positive tumor cells, a dimensionless real number, ranging from 0 to 1.
[0093] The proportional prediction loss uses the Smooth L1 loss function: L R = SmoothL1(r pred -r gt ); Where, r pred r represents the positive rate predicted by the model, a dimensionless real number ranging from 0 to 1. gt The true positive rate is represented by annotations made by the pathologist on paired IHC slides; it is a dimensionless real number ranging from 0 to 1. SmoothL1 is defined as a piecewise function. SmoothL1(x) = { 0.5·x², if |x| < 1; |x| - 0.5, otherwise} Where x is the difference between the predicted value and the actual value, i.e.: r pred -r gt ), which is a dimensionless real number.
[0094] Finally, each image patch outputs a multidimensional quantitative score that includes staining intensity level, membrane expression pattern, and proportion of positive cells, which serves as the basis input for subsequent WSI-level spatial topological integration.
[0095] To achieve joint optimization of the three tasks, this embodiment further constructs a multi-task joint loss function: ;
[0096] Where, λ I , λL , λ R and λ O λ represents the weighting coefficients (dimensionless) for the staining intensity classification loss, membrane expression pattern classification loss, positive proportion prediction loss, and orthogonal constraint loss, used to balance the contributions of the four sub-tasks. In this embodiment, λ is preferably set to λ. I =1.0、λ L =1.0、λ R =2.0、λ O =0.5. By jointly optimizing the above loss function, information sharing and collaborative learning among the three tasks are achieved, while ensuring that each task has good discriminative ability and generalization performance.
[0097] Step 5: Model training based on the geometrically adaptive joint loss supervision mechanism (GAML) like Figure 4 As shown, in order to achieve collaborative optimization of cross-modal virtual staining, expression intensity calibration and multi-dimensional scoring tasks, this embodiment designs a geometrically adaptive multi-task loss (GAML) mechanism for end-to-end training of the entire network.
[0098] Specifically, the overall joint loss function is defined as: ; Among them, L GAN L represents the cross-modal generative adversarial loss in step 2; Cycle L represents the loss of cycle consistency. DACIC This represents the calibration loss of the dual-anchored expression intensity constructed in step 3; L Task L represents the joint loss of multiple tasks in step 4; Heter λ1 represents the heterogeneity adaptive loss; λ1-λ4 represent the weight coefficients (dimensionless) corresponding to each loss term, used to balance the contributions between different optimization objectives. In this embodiment, λ1=10.0, λ2=2.0, λ3=1.0, and λ4=0.5 are preferably set.
[0099] Considering the significant spatial heterogeneity of gastric cancer tissue, this embodiment further designs a heterogeneity adaptive loss. This loss dynamically adjusts the weights of different task losses based on cell density, tissue complexity, and model prediction uncertainty within the image patch, enabling the model to focus on information-rich regions and reduce the interference of mucus lakes, inflammatory infiltration, and stromal regions on the training process. For the i-th image patch, its comprehensive heterogeneity score is defined as: ; Among them, h i D represents the overall heterogeneity score of the i-th image patch, a dimensionless real number; iCell density index: calculated by cell nucleus counting, with values ranging from 0 to 1, and normalized; C i Organizational complexity index: calculated using image entropy or edge density, with a value ranging from 0 to 1, U i The uncertainty in model prediction is represented by the standard deviation of multiple predictions sampled using MC Dropout, with a value ranging from 0 to 1; α, β, and γ are weighting coefficients (dimensionless) that satisfy α + β + γ = 1. In this embodiment, α = 0.4, β = 0.3, and γ = 0.3 are preferably set.
[0100] Calculate the training weights for each image patch based on the heterogeneity score: ;
[0101] Among them, w i The weight of the i-th image patch is a dimensionless real number, and the sum of the weights of all image patches is 1; exp(h i ) represents a base of the natural constant e, h i is an exponential function of the exponent, where N represents the total number of image patches in the current training batch, a dimensionless integer, which is 16 in this embodiment.
[0102] Heterogeneity adaptive loss is defined as: ;
[0103] Among them, L i This represents the basic task loss for the i-th image patch, i.e., the L value corresponding to that image patch. Task Loss value, dimensionless real number, w i These are the training weights for the image patch.
[0104] During model training, the AdamW optimizer is used for parameter updates, and its parameter update rules are as follows: ;
[0105] Where: θ t θ: The model parameters (tensors) at step t, containing all learnable weights and biases in the network; t+1 Updated model parameters (tensors), dimensions and θ t Same; η t The learning rate (a positive real number) at step t is dynamically determined by the cosine annealing strategy described later. The first moment estimate of the gradient, i.e., the momentum term with bias correction, tensor, dimension, and θ. t The same principle applies; it is used to accelerate convergence and smooth the gradient direction. The second-order moment estimate of the gradient (i.e., the variance term with bias correction, tensor), dimension and θ tThe same applies; ε: numerical stability constant, which is preferably set to ε = 1 × 10 in this embodiment. -8 , dimensionless, used to avoid numerical overflow due to an excessively small denominator; λ: weight decay coefficient, which is preferably set to λ=1×10 in this embodiment. -4 Dimensionless, used for L2 regularization to suppress overfitting.
[0106] The AdamW optimizer described above is used for parameter updates during model training, with an initial learning rate set to 1×10⁻⁶. -4 The weight decay coefficient is set to 1×10. -4 And a cosine annealing strategy is used to dynamically adjust the learning rate: ;
[0107] Where, η t η represents the learning rate in the t-th training epoch; max This represents the initial learning rate, which is preferably set to 1×10 in this embodiment. -4 η min This represents the minimum learning rate (preferably set to 1×10 in this embodiment). -6 T represents the total number of training epochs, which is preferably set to 150 in this embodiment; t represents the current training epoch, starting from 0; cos represents the cosine function, and π is the constant of pi. During model training, a batch size of 16 is used, and the number of training epochs is preferably set to 150. In the first 10 epochs, a warm-up strategy is used to gradually increase the learning rate, linearly increasing it from 0 to η. max This is to improve the convergence stability of the model in the early stages of training.
[0108] To improve the model's generalization ability across different scanning devices and staining batches, this embodiment employs data augmentation strategies during training, including random rotation ±90°, horizontal flipping, vertical flipping, random scaling (0.8-1.2x), color perturbation (brightness, contrast, and saturation adjustments), Gaussian Blur (kernel size 5×5 pixels, standard deviation σ=1.0), and Macenko staining normalization, to augment the training samples online. When the joint loss on the validation set fails to decrease for 20 consecutive epochs, an Early Stopping strategy is used to terminate training early, and the model parameters with the best performance on the validation set are saved as the final model.
[0109] Step 6: Output a full-scale WSI-level scoring report based on a self-calibrating spatial topology integrator. like Figure 4As shown, in the actual clinical inference stage, routine H&E full-scan slides from patients completely devoid of exogenous amniotic membrane rolls are input into the trained model. First, the model automatically identifies normal gastric mucosal regions in the slides using the region segmentation mechanism in step 1 and extracts the corresponding H&E features. Subsequently, based on the cross-modal mapping relationship established in step 3, the normal gastric mucosal features are mapped to the shared latent space, dynamically restoring the individualized intensity reference scale corresponding to the current slide, thereby achieving online adaptive calibration of color differences caused by different slide batches, scanning equipment, and staining conditions.
[0110] After completing individualized intensity calibration, the system uses a sliding window approach to perform dense inference on the tumor parenchyma region in the entire WSI image. Specifically, using fixed-size image blocks as units (256×256 pixels in this embodiment), the system slides and cuts blocks within the entire tumor region according to a preset step size (128 pixels in this embodiment, i.e., 50% overlap), and outputs the Claudin18.2 staining intensity level, membrane expression pattern, and positive cell proportion for each image block.
[0111] This embodiment designs a self-calibrating spatial topology integrator to aggregate the prediction results of all image patches at the whole-patch level. For the i-th image patch, its prediction vector is represented as: ;
[0112] Where, p i I Let p be the intensity category probability vector, with 4 dimensions, where each element represents the probability of the corresponding intensity category; i L This is a probability vector for membrane mode categories, with four dimensions, where each element represents the probability of the corresponding membrane mode, p. i R The positive rate prediction value is a dimensionless real number, ranging from 0 to 1; conf i The confidence level for model prediction is a dimensionless real number ranging from 0 to 1, calculated using the softmax maximum probability value or MC Dropout uncertainty estimation.
[0113] A topological graph structure G=(V,E) is constructed based on the spatial adjacency relationships between image patches, where V is the set of all image patch nodes and E is the set of edges connecting spatially adjacent image patches. Spatial smoothing and topological fusion are then performed on the prediction results of adjacent image patches. ;
[0114] Where, p i smoothed Let represent the smoothed prediction vector of the i-th image patch; N(i) represents the set of image patches adjacent to the i-th image patch. Based on 4-neighborhood or 8-neighborhood definitions; w ij The adjacency weights are calculated using a Gaussian kernel based on spatial distance. , where pos i and pos j σ and η are the center coordinates of the i-th and j-th image blocks, respectively. σ is the Gaussian kernel bandwidth parameter, which is 1.0 in this embodiment. η is the smoothing coefficient (η=0.3 in this embodiment, dimensionless), used to control the smoothing intensity and eliminate local noise and isolated misjudged regions.
[0115] The number and spatial distribution of tumor cells with positive membrane expression characteristics in the entire WSI were statistically analyzed, and the percentage of positive tumor cells in the total number of tumor cells was calculated. ;
[0116] Where, N i positive N represents the number of tumor cells with positive membrane expression in the i-th image patch. i total Σ represents the total number of tumor cells in the i-th image patch. i R represents the summation over all image patches. WSI The percentage of positive cells on the entire WSI slice is a dimensionless real number, ranging from 0 to 100%.
[0117] The system combines the confidence level, spatial continuity, and region area proportion of image patch prediction results to construct a confidence-weighted topology mapping network: ;
[0118] Among them, Score final The final predicted category for the entire WSI is indicated by argmax: intensity level or membrane mode. c This indicates selecting the category c that yields the largest value within the parentheses, conf. i Let area be the confidence level of the i-th image patch. i p represents the area percentage of the i-th image patch within the tumor region (a dimensionless real number, the area of this image patch divided by the total area of the tumor region). i (c) represents the probability that the i-th image patch is predicted to be of category c. Global voting and weighted fusion are performed on different staining intensity levels and membrane expression patterns to finally determine the final Claudin18.2 staining intensity level (0, 1+, 2+, 3+) and main membrane expression patterns of the entire WSI.
[0119] Finally, the system automatically generates a standardized quantitative report containing the proportion of positive cells, staining intensity level, membrane expression location, spatial distribution heatmap, and comprehensive scoring results. The report includes: Patient basic information and sample information; Claudin 18.2 comprehensive expression score (positive rate, intensity level, membrane pattern); Heat map of spatial distribution across the entire area (marking the spatial distribution of different intensity levels); Details of local scoring for each tumor region; Clinical interpretation recommendations.
[0120] Through the above steps, this invention achieves precise quantitative assessment of the expression status of Claudin18.2 in gastric cancer at the whole-slice, cell-level, and multi-dimensional levels based solely on conventional H&E full-scan slices, providing a reliable and reproducible quantitative basis for clinical precision treatment decisions.
[0121] To verify the effectiveness of this invention, this embodiment collected H&E full scan slides and paired Claudin 18.2 immunohistochemical slides from 2448 gastric cancer patients. Of these, 1800 were used for training, 324 for validation, and 324 for independent testing. The test results showed that: In the task of staining intensity grading, the overall classification accuracy of the present invention reached 89.6%; in the task of membrane positioning pattern recognition, the average classification accuracy of the present invention reached 88.9%, with the highest recognition accuracy of 93.2% for complete circumferential membrane patterns.
[0122] In the positive proportion prediction task, the Pearson correlation coefficient between the Claudin18.2 positive tumor cell proportion predicted by this invention and the pathologist's manual interpretation result reached 0.92, with a mean absolute error of 5.6%.
[0123] In cross-center generalization tests, the staining intensity classification accuracy of this invention fluctuated within a range of less than 3% on datasets from different medical centers and scanning devices. This indicates that the constructed dual-anchor intensity calibration mechanism can effectively reduce the systematic differences caused by different slide batches, staining conditions and scanning devices, and improve the stability and generalization ability of the model in cross-center application scenarios.
[0124] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the implementation of the present invention is not limited to the above-described manner. Any improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.
Claims
1. A method for multidimensional quantitative assessment of Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchoring adaptive calibration, characterized in that, Includes the following steps: Step 1: Obtain H&E stained full-scan slice images of gastric cancer patients, perform color normalization processing, and use a lightweight semantic segmentation network to extract the tumor parenchymal epithelial region and the patient's own normal gastric mucosal gland region from the image, and remove acellular regions; introduce a cell center spatial perception resampling mechanism, use cell nucleus detection algorithm to lock the centroid of each cell nucleus as a physical anchor point, and adaptively expand outward to extract local image patches containing complete cytoplasm and cell membrane boundaries; Step 2: Construct a bidirectional cross-modal virtual staining adversarial network Bi-VCAN as a feature extractor, which includes a generator G that transforms HE modality features into virtual Claudin18.2 IHC modality features. H →I, and the generator G that inversely restores the virtual IHC mode to the HE mode. I →H;Introducing structural consistency loss, using edge detection operators to extract cell boundary structure information in HE images, constraining high-response regions in virtual IHC feature maps to maintain spatial consistency with cell membrane boundaries in HE images; Step 3: Construct an expression intensity calibration space based on a dual-anchoring comparison mechanism of IHC positive / negative control and endogenous normal gastric mucosa; during the training phase, the amniotic roll control region on the IHC slice is simultaneously extracted as the exogenous highest intensity anchor point, and the patient's own normal gastric mucosa region is used as the endogenous normal expression intensity anchor point; the amniotic roll IHC features are mapped into a global strong positive benchmark feature vector v. max The IHC features of normal gastric mucosa were constructed into a normal mucosal physiological baseline vector group v. n Using paired HE-IHC image patches of normal gastric mucosa from patients as cross-modal bridging samples, a cross-modal mapping relationship is established, enabling the model to learn and solidify the absolute colorimetric standard jointly defined by the amnion roll and normal gastric mucosa. Step 4: Input the high-order cross-modal transformation features extracted in Step 2 into the decoupled multi-task head, and map them to the staining intensity subspace S respectively. i , membrane staining location subspace S l and the coloring ratio subspace S r This method enables the joint prediction of Claudin18.2 expression intensity, membrane localization pattern, and the proportion of positive cells; orthogonal constraint loss is introduced to reduce information coupling between the three tasks. Step 5: Use a geometrically adaptive joint loss supervision mechanism to perform end-to-end joint training on Bi-VCAN, the dual-anchor calibration module, and the multi-task head; Step 6: In actual clinical reasoning, the HE full scan slice of the gastric cancer of the test patient without exogenous amniotic membrane roll is input into the trained model. The model automatically locates the endogenous normal gastric mucosal gland region in the slice and extracts its HE features. Based on the cross-modal mapping relationship established in Step 3, the individualized intensity reference scale corresponding to the current slice is dynamically restored, realizing online adaptive calibration across batches and devices. A sliding window strategy is used to intensively infer the entire WSI tumor parenchyma region. The prediction results of all image patches are aggregated at the whole-pattern level through a self-calibrated spatial topology integrator. The final Claudin18.2 staining intensity level, positive cell ratio and major membrane expression pattern of the entire WSI are output, and a standardized quantitative scoring report is generated.
2. The method for multidimensional quantitative assessment of Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchoring adaptive calibration as described in claim 1, characterized in that: The cell center spatial awareness resampling mechanism described in step 1 is as follows: the HoVer-Net cell nucleus detection algorithm is used to identify all cell nuclei in the tissue region and extract their centroid coordinates; the cell nucleus centroid is used as a physical anchor point to adaptively expand outward to form a local sampling region, and the image block size is dynamically adjusted according to the local cell density. The dense cell region uses a 256×256 pixel image block, and the sparse cell region uses a 512×512 pixel image block. The color normalization process described in step 1 specifically involves: converting the original RGB image to optical density space, extracting the principal component staining matrix through singular value decomposition, and mapping the staining features of the original image to a preset standard reference gastric cancer image.
3. The method for multidimensional quantitative assessment of Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchoring adaptive calibration as described in claim 1, characterized in that: The structural consistency loss mentioned in step 2 includes the edge consistency loss L. SC And skeleton consistency loss L SK : ; ; ; Where N is the number of image patch samples in a training batch. This represents the edge map extracted from the i-th HE image patch. This represents the edge map extracted from the i-th virtual IHC image patch. This represents the skeleton map extracted from the i-th HE image patch. represents the skeleton map extracted from the i-th virtual IHC image block; ‖·‖1 is the L1 norm, which is the sum of the absolute values of each element; ‖·‖2² is the square of the L2 norm, which is the sum of the squares of each element.
4. The method for multidimensional quantitative assessment of Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchoring adaptive calibration as described in claim 1, characterized in that: The dual-anchoring comparison mechanism described in step 3 specifically involves: dividing the amniotic membrane roll image block X... A Deep features extracted using a shared encoder: By anchoring losses ; Map it to the global strong positive prototype vector v max X-ray a normal gastric mucosa image block N Extracting deep features By anchoring losses ; Map it to the normal mucosal physiological reference vector group v n The combined loss of dual anchoring is: ;in, f A It is a depth feature representation of the amniotic roll image patch; E shared It is an encoder network that shares modes for HE and IHC; Proj It is a reference object aligned with the projection layer; v max It is a globally strong positive prototype vector; f N (i) is the depth feature of the i-th normal gastric mucosal image patch; ||·|2² is the square of the L2 norm, i.e., the square of the Euclidean distance.
5. The method for multidimensional quantitative assessment of Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchoring adaptive calibration as described in claim 1, characterized in that: The cross-modal mapping relationship described in step 3 is achieved through anchoring constraint loss based on margin sorting: ; Where m is the sorting margin parameter, d(·,·) is the Euclidean distance metric, and v t Let v be the feature vector of the gastric cancer region to be predicted. n v is the normal gastric mucosal physiological reference prototype vector. max Σ is the global strong positive prototype vector. t This sums all gastric cancer samples in the current training batch. max(0, ·) is the cutoff function, taking the larger value between 0 and the value in parentheses.
6. The method for multidimensional quantitative assessment of Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchoring adaptive calibration as described in claim 1, characterized in that: The decoupled multi-task header described in step 4 uses three task-specific projection networks to share the feature F. shared Mapped to three subspaces: f I = P I (F shared ) ∈S I ; f L = P L (F shared ) ∈S L ; f R = P R (F shared ) ∈S R ; Among them, P I P L P R These represent mappings to the staining intensity subspace S, respectively. I Membrane expression location subspace S L and positive proportion subspace S R Projection network; f I f L f R These are task-specific feature vectors mapped to each subspace; The orthogonal constraint loss is: ; Where the superscript T denotes vector transpose, ||·|| F 2 This represents the square of the Frobenius norm, which is the sum of the squares of the elements in the matrix.
7. The method for multidimensional quantitative assessment of Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchoring adaptive calibration as described in claim 1, characterized in that: The staining intensity prediction in step 4 uses a Softmax classifier: pi = softmax(Wi⋅fi + bi); The intensity classification loss is the cross-entropy loss: ; The membrane expression pattern prediction uses a multi-class classifier: p l = softmax(W l · f l + b l ) ; The membrane pattern classification loss is: ; The positive cell proportion prediction was performed using the Smooth L1 loss function: L R = SmoothL1(r pred - r gt ); SmoothL1(x) = { 0.5·x², if |x| < 1; |x| - 0.5, otherwise}; The combined loss from multiple tasks is: W i b is the weight matrix of the intensity classifier; i It is the bias vector of the intensity classifier; p i It is the output intensity category probability vector; W l This is the weight matrix of the membrane pattern classifier; b l It is the bias vector of the membrane pattern classifier; p l It is the output membrane pattern category probability vector; r pred It is the positive rate predicted by the model; r gt It represents the true positive rate; SmoothL1 is the smoothed L1 loss function; λ I , λ L , λ R , λ O These are the weighting coefficients for the losses of each subtask.
8. The method for multidimensional quantitative assessment of Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchoring adaptive calibration as described in claim 1, characterized in that: The overall loss function of the geometrically adaptive joint loss supervision mechanism described in step 5 is: ; Among them, L GAN To generate adversarial loss across modalities, L Cycle For the cycle consistency loss, L DACIC For the dual-anchored expression intensity calibration loss, L Task For multi-task joint loss, L Heter This is a heterogeneous adaptive loss; The heterogeneity adaptive loss dynamically adjusts the training weights based on the cell density, tissue complexity, and model prediction uncertainty of the image patch: ; ; ; Among them, h i D is the comprehensive heterogeneity score for the i-th image patch. i As a cell density indicator, C i U is an indicator of organizational structure complexity. i To represent the model's prediction uncertainty, α, β, and γ are weighting coefficients that satisfy α + β + γ = 1, w i Let be the training weights for the i-th image patch; L i It represents the basic task loss for the i-th image patch.
9. The method for multidimensional quantitative assessment of Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchoring adaptive calibration according to claim 1, characterized in that: The specific method by which the self-calibrating spatial topology integrator performs whole-patch aggregation of image patch prediction results in step 6 is as follows: A topology graph structure is constructed based on the spatial adjacency relationships between image patches, and spatial smoothing and topological fusion are performed on the prediction results of adjacent image patches. ; The percentage of positive cells in the entire WSI was: ; The final staining intensity level is determined using a confidence-weighted topology mapping network: ; Where, p i Let w be the original prediction vector of the i-th image patch, and N(i) be the set of image patches adjacent to the i-th image patch. ij Here, adjacency weights are based on spatial distance, η is the smoothing coefficient, and conf i For the model prediction confidence of the i-th image patch, area i N represents the proportion of the area of the i-th image patch to the total area of the tumor region. i positive N is the number of tumor cells with positive membrane expression in the i-th image patch; i total It is the total number of tumor cells in the i-th image patch; R WSI It represents the percentage of positive cells across the entire WSI layer.
10. A multidimensional quantitative assessment system for Claudin 18.2 expression in gastric cancer based on bidirectional cross-modal virtual staining and dual-anchoring adaptive calibration, characterized in that, include: The image preprocessing module is used to normalize the color of the acquired H&E stained full scan slice images of gastric cancer patients, extract the tumor parenchymal epithelial region and the patient's own normal gastric mucosal gland region, and extract local image patches containing intact cytoplasm and cell membrane boundaries through the cell center spatial perception resampling mechanism. A bidirectional cross-modal virtual staining feature extraction module is used to construct a Bi-VCAN network, which transforms the HE modality into higher-order features of the virtual Claudin18.2 IHC modality, and constrains the virtual staining signal to expand along the real physical boundary of the cell membrane through structural consistency loss. The dual-anchored expression intensity calibration module is used to introduce the amniotic roll control region as the exogenous highest intensity anchor point and the patient's own normal gastric mucosa as the endogenous normal expression intensity anchor point during the training phase, establish a cross-modal mapping relationship, and dynamically restore the individualized intensity reference scale of the current slice during the inference phase. A multi-dimensional decoupled multi-task decision module is used to map cross-modal transformation features to the staining intensity subspace, membrane staining location subspace, and staining ratio subspace, respectively, to achieve joint prediction of Claudin18.2 expression intensity, membrane localization pattern, and positive cell ratio; The geometrically adaptive joint loss supervised training module is used for end-to-end joint training of the entire network; The self-calibrated spatial topology integration module is used to perform sliding window dense inference on the entire WSI in actual clinical reasoning, perform whole-area topological aggregation and confidence-weighted fusion of prediction results for all image patches, and output a standardized quantitative scoring report.
