Colorectal cancer HER2 state classification method based on double-branch collaborative optimization network

Abstract

The invention discloses a colorectal cancer HER2 state classification method based on a double-branch collaborative optimization network, and belongs to the technical field of pathological image processing. According to the method, a double-path collaborative optimization architecture is adopted, knowledge migration and cross-modal feature re-projection are combined, features related to the HER2 state are efficiently extracted from a conventional hematoxylin-eosin (Hamp; E) dyed section, and the HER2 state can be directly and accurately judged. According to the method, through priori knowledge migration and a cross-modal feature reprojection mechanism, the bottlenecks of insufficient priori knowledge utilization, modal isomerism interference and the like in a traditional method are broken through, and the precision and robustness of rectal cancer HER2 state classification are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of pathological image processing, and specifically relates to a colorectal cancer HER2 status classification method based on a dual-branch collaborative optimization network. Background Art

[0002] Colorectal cancer (CRC) is a major global public health challenge. Its early screening mainly relies on fecal immunochemical testing (FIT) and colonoscopy. The latter is a pathological diagnosis through tissue biopsy combined with H&E staining. However, traditional histological testing can only confirm the presence of malignant tumors and cannot provide information on tumor characteristics at the molecular level, which poses a key limitation to prognosis assessment and precision treatment. In this context, the importance of HER2 testing has become increasingly prominent. The overexpression or gene amplification of this receptor protein is closely related to the aggressiveness and poor prognosis of colon cancer. Although immunohistochemistry (IHC) and fluorescence in situ hybridization (FISH) are still the gold standard for HER2 detection, their clinical application faces bottlenecks such as strict tissue sample quality requirements, complex detection procedures, and inter-observer differences in interpretation results.

[0003] Current research on HER2 status classification is gradually expanding from breast cancer to colon cancer, and deep learning technology has shown the potential to replace traditional detection. Existing studies have achieved tumor detection and preliminary classification of HER2 status through H&E image analysis. Some work has attempted to integrate morphological features such as DAB staining density and cell nuclear distribution to improve model performance. It is worth noting that the constraint matrix optimization method based on the ASCO / CAP guidelines and the technological innovation of using generative adversarial networks (GANs) to virtually convert H&E slices into IHC images have provided new ideas for reducing detection costs. However, existing algorithms still rely mainly on low-level image features such as color and texture, and fail to fully integrate medical prior knowledge to build a deep cognitive system.

[0004] The lack of integration of medical prior knowledge has become a core issue that restricts classification accuracy. The multi-dimensional knowledge that pathologists rely on in HER2 interpretation - including the spatial distribution characteristics of IHC-positive areas, the correlation pattern between DAB color intensity and cellular substructures (such as cell membrane integrity and nuclear-cytoplasmic ratio abnormalities), and key diagnostic clues such as tissue structural heterogeneity, has not yet been effectively quantified into operational model parameters. The morphological association cognition accumulated by these human experts through long-term practice, especially the interaction between cell membrane-specific staining intensity and the tumor microenvironment, is of decisive value in distinguishing true and false positive cases. Future research urgently needs to establish a knowledge embedding mechanism to transform expert experience such as quantitative analysis of cell morphology and tissue spatial topological relationships into constraints for deep learning, so as to break through the current classification bottleneck based on appearance features. Summary of the Invention

[0005] The purpose of the present invention is to provide a colorectal cancer HER2 status classification method based on a dual-branch collaborative optimization network to solve key problems in the existing technology, such as insufficient HER2 status classification accuracy and insufficient cross-modal feature fusion.

[0006] In order to achieve the above object, the specific technical solution adopted by the present invention is:

[0007] A method for classifying HER2 status of colorectal cancer based on a dual-branch collaborative optimization network comprises the following steps:

[0008] S1: Acquire H&E whole-slide images and IHC landmark images, perform spatial registration, and obtain an affine transformation matrix to spatially align the H&E image with the IHC image.

[0009] S2: Using the registered annotation information, a knowledge transfer feature encoder is constructed to transfer the annotation information from the IHC image to the H&E image, thereby achieving knowledge sharing between the two image modalities; the knowledge transfer feature encoder includes a HER2 block-level classifier using a ResNet18 backbone network consisting of a linear layer, a ReLU activation function, a dropout layer, a linear layer, a ReLU activation function, a dropout layer, a linear layer, a ReLU activation function, a dropout layer, and an output layer, and the classifier is optimized using a binary cross-entropy loss;

[0010] S3: Based on the affine transformation matrix, IHC and HE image pairs are sampled, and a cross-modal feature reprojection encoder is constructed and trained to achieve effective fusion of the two modal image features, further improving the expressiveness and consistency of the features. The cross-modal feature reprojection encoder adopts a cross-modal joint embedding architecture based on a symmetric dual-branch encoder. Its core consists of two ImageNet pre-trained ResNet50 backbone networks, followed by a shared projection layer, which is composed of a linear layer, a ReLU activation function, a Dropout layer, another linear layer, and a ReLU activation function.

[0011] S4: extracting image block-level features using the trained knowledge transfer feature encoder and the cross-modal feature reprojection encoder;

[0012] S5: Generate full-slice-level prediction results through a multi-instance aggregation mechanism, and integrate local features into global features through an aggregation algorithm to obtain the final HER2 status classification prediction results.

[0013] Furthermore, the S1 specifically includes:

[0014] S1-1: Mark matching landmarks on paired H&E images (denoted as X) and IHC images (denoted as Y) for subsequent calculation of the affine transformation matrix;

[0015] S1-2: Manually select and annotate obvious and easily identifiable matching landmarks on H&E and IHC images;

[0016] S1-3: Calculate the affine transformation matrix describing the spatial relationship between the H&E image and the IHC image;

[0017] S1-4: Map the annotated region in the IHC image to the H&E image space using an affine transformation matrix;

[0018] S1-5: Generate image block level label L based on the registration result i , providing supervisory signals for subsequent feature extraction and classification.

[0019] Furthermore, the S2 specifically includes:

[0020] S2-1: Train the HER2 block-level classifier, using the image block-level label L obtained in S1-5 i Training is performed, where the negative image block is P i X- , the positive image block is P i X+ ;

[0021] S2-2: The classifier uses a customized ResNet18 backbone network. By replacing its original fully connected layers with a multi-layer perceptron structure containing multiple linear layers, ReLU activation functions, and Dropout layers, the model's feature learning ability and generalization performance are enhanced. Specifically, it consists of a linear layer (512, 512), ReLU activation function, Dropout layer, linear layer (512, 256), ReLU activation function, Dropout layer, linear layer (256, 128), ReLU activation function, Dropout layer, and output layer (128, 1). Finally, a linear layer with an output dimension of 1 is used for binary classification problems to generate a single prediction value.

[0022] S2-3: Use regularized binary cross entropy loss to optimize the classifier:

[0023]

[0024] Where N represents the batch size, and Represents the positive sample P i X+ and negative samples P i X- The true label, and Represents the positive sample P i X+and negative samples P i X- The predicted label of .

[0025] Furthermore, the S3 includes:

[0026] S3-1: Align the H&E and IHC image patches using the affine transformation matrix M and the matching control points between the manually annotated H&E and IHC image pairs to obtain a large number of image patch pairs for training the feature reprojection encoder.

[0027] S3-2: A cross-modal joint embedding architecture based on a symmetric dual-branch encoder is used. Its core consists of two ImageNet pre-trained ResNet50 backbone networks, followed by a shared projection layer, which is composed of a linear layer (2048, 1024), a ReLU activation function, a Dropout layer, another linear layer (1024, 1024), and a ReLU activation function. Deep feature extraction is performed on H&E and IHC images respectively.

[0028] S3-3: To align features between H&E and IHC modalities, label-aware loss and cross-modal triplet loss are used to jointly optimize the cross-modal feature embedding space.

[0029] Furthermore, the S3-3 includes:

[0030] S3-31: Label-aware loss: To enhance the discriminability of features, a loss based on the category label L is applied. i The cross entropy constraint is:

[0031]

[0032] where σ(·) represents the softmax function; L cls Label-aware discriminative losses representing H&E and IHC modalities, ensuring that the learned features are semantically aligned with their respective class labels;

[0033] S3-32: Cross-modal triplet loss: To further strengthen cross-modal alignment, a triplet based on L2 distance is designed.

[0034] loss:

[0035]

[0036] Here, X and Y are used as anchor points to calculate the triplet loss. i X ,P i Y} are cross-modal samples from the same control point, and negative samples are are cross-modal samples from different control points with different class labels.

[0037] S3-33: The total loss function is:

[0038] L total =L cls +αL cross-triplet

[0039] where α is the balance coefficient.

[0040] Furthermore, the S4 is:

[0041] S4-1: First, we use the SLFCD model to accurately segment tumor regions, effectively distinguishing between tumors and normal tissues and reducing computational redundancy. The SLFCD model is a supervised learning-based cancer detection framework primarily used for cancer region localization and image classification in full-slide cancer pathology images.

[0042] S4-2: The knowledge transfer encoder and the cross-modal feature reprojection encoder are used to extract features respectively, and the extracted features are concatenated to obtain a feature vector F.

[0043] Furthermore, in S5, the classification prediction is expressed as:

[0044] L her2 =Multi-Instance(F)

[0045] Among them, L her2 is the predicted full-slice-level HER2 status, and Multi-Instance(·) is the multi-instance aggregator.

[0046] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0047] This invention, through prior knowledge transfer and cross-modal feature reprojection mechanisms, overcomes bottlenecks such as insufficient prior knowledge utilization and modality heterogeneity interference in traditional methods, significantly improving the accuracy and robustness of colorectal cancer HER2 status classification. The invention has the following advantages:

[0048] 1. Prior Knowledge Transfer and Enhancement: This paper constructs a knowledge transfer feature encoder to transfer annotation information from IHC images to H&E images. This fully utilizes the prior knowledge of IHC images and compensates for the lack of information in H&E images for HER2 status classification. This knowledge transfer mechanism effectively enhances the feature expression capabilities of H&E images and improves classification accuracy.

[0049] 2. Cross-modal feature reprojection and fusion: This paper uses affine transformation matrices to sample H&E and IHC images, constructs a cross-modal feature reprojection encoder, and achieves effective fusion of the two modal image features, significantly improving the consistency of feature expression and classification performance.

[0050] 3. Multi-Instance Aggregation and Global Prediction: This invention utilizes a multi-instance aggregation mechanism to integrate block-level features into full-slice-level features, generating the final HER2 status classification prediction. This innovative dual-modal joint representation mechanism addresses the shortcomings of traditional methods in leveraging prior knowledge and handling modality heterogeneity, significantly improving the accuracy and robustness of HER2 status classification in colorectal cancer. This provides new technical insights and solutions for pathological image analysis and clinical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flowchart of the present invention.

[0052] Figure 2 Schematic diagram of the knowledge transfer module of the present invention.

[0053] Figure 3 Schematic diagram of the cross-modal feature reprojection structure of the present invention.

[0054] Figure 4 Schematic diagram of the overall classification process of the present invention.

[0055] Figure 5 Schematic diagram of input data of the present invention.

[0056] Figure 6 This is a comparison chart of the density effects of H&E, IHC and DAB (3,3'-Diaminobenzidine) according to the method of the present invention. DETAILED DESCRIPTION

[0057] In the following description, various aspects of the present invention will be described. However, it will be apparent to those skilled in the art that the present invention may be practiced utilizing only some or all of the structures or processes of the present invention. Specific numbers, configurations, and sequences are described for clarity of explanation, but it is apparent that the present invention may be practiced without these specific details. In other cases, well-known features will not be described in detail to avoid obscuring the present invention.

[0058] Example 1:

[0059] Reference Figure 1 、 Figure 4 , a colorectal cancer HER2 status classification method based on a dual-branch collaborative optimization network, comprising the following steps:

[0060] Step 1: Perform spatial registration on the H&E whole-slide image and the IHC landmark image to obtain an affine transformation matrix so that the H&E image and the IHC image are spatially aligned.

[0061] Step 1.1: Mark matching landmarks on the paired H&E image (denoted as X) and IHC image (denoted as Y) to facilitate subsequent calculation of the affine transformation matrix.

[0062] Step 1.2: Manually select and annotate approximately 40 pairs of obvious, easily identifiable matching landmarks on the H&E and IHC images. The coordinates of the landmarks in the H&E images are expressed as The coordinates of the landmark points in the IHC image are expressed as These landmarks should have sufficient contrast and uniqueness to ensure accurate identification in images from both modalities. Each pair of landmarks should be distributed in different areas of the image as much as possible to improve the accuracy and robustness of the registration.

[0063] Step 1.3: Calculate the affine transformation matrix that describes the spatial relationship between the H&E image and the IHC image. Expand the IHC coordinate y to a homogeneous form, that is, add a constant term 1, expressed as The affine transformation matrix M is solved by minimizing the geometric distortion objective function:

[0064]

[0065] in represents the transformed y-coordinate, and · represents matrix multiplication.

[0066] Step 1.4: Map the annotated region in the IHC image to the H&E image space using an affine transformation matrix.

[0067] A H&E =M·A IHC

[0068] A H&E Indicates the corresponding HER2 annotated area in the H&E image.

[0069] Step 1.5: Generate image block-level labels L based on the registration results i , providing supervisory signals for subsequent feature extraction and classification. i If it is 1, the corresponding area is HER2 positive, L i If it is 0, the corresponding area is HER2 negative.

[0070] Step 2: Use the registered annotation information to construct a knowledge transfer feature encoder. The annotation information in the IHC image is transferred to the H&E image, thus achieving knowledge sharing between the two image modalities.

[0071] Step 2.1: Since H&E images are captured by whole-slice imaging (WSI), the images are too large to be processed as a whole. First, a HER2 block-level classifier is trained to achieve block-level HER2 classification. The block-level label L of the H&E image obtained in step 1.4 is used. i Training is performed, where the negative image block is P i X- , the positive image block is P i X+ .

[0072] Step 2.2: If Figure 2 As shown, the classifier uses a customized ResNet18 backbone network. By replacing its original fully connected layers with a multilayer perceptron structure consisting of multiple linear layers, ReLU activation functions, and Dropout layers, the model's feature learning and generalization capabilities are enhanced. This multilayer perceptron structure consists of three linear layers, each followed by a ReLU activation function and a Dropout layer. Finally, a linear layer with an output dimension of 1 is used for binary classification problems to generate a single prediction value.

[0073] Step 2.3: Optimize the classifier using regularized binary cross entropy loss:

[0074]

[0075] Where N represents the batch size, and Represents the positive sample P i X+ and negative samples P i X- The true label, and Represents the positive sample P i X+ and negative samples P i X- The predicted label of .

[0076] Step 3: Based on the affine transformation matrix, sample IHC and HE image pairs and construct a cross-modal feature reprojection encoder. This effectively fuses the features of the two modal images, further improving the expressiveness and consistency of the features.

[0077] Step 3.1: Align the H&E and IHC image patches using the affine transformation matrix M and approximately 40 manually annotated matching control points between the H&E and IHC image pairs. Specifically, each control point in the IHC image is mapped to its corresponding spatial position in the H&E image to ensure accurate patch alignment. Then, 25 non-overlapping patches are extracted by uniform grid partitioning centered on these transformation points to obtain a large number of image patch pairs {P iX ,P i Y} for training of feature reprojection encoder.

[0078] Step 3.2: This module adopts a cross-modal joint embedding architecture based on a symmetric dual-branch encoder, such as Figure 3 As shown in the figure, its core consists of two ImageNet pre-trained ResNet50 backbone networks, which perform deep feature extraction on H&E and IHC images respectively. To eliminate the feature distribution differences between the modalities, a learnable nonlinear mapping module is introduced at the end of the two branches. Through joint optimization, the feature vectors of the two modalities are projected into a common subspace, achieving geometric structure and semantic relevance alignment of cross-modal features.

[0079] Step 3.3: To align features between H&E and IHC modalities, define a common subspace Z, as well as mapping functions f:X→Z and g:Y→Z; optimize the cross-modal feature embedding space using the following joint loss function:

[0080] Step 3.3.1: Label-aware loss: To enhance the discriminability of features, a loss based on the class label L is applied. i The cross entropy constraint is:

[0081]

[0082] where σ(·) represents the softmax function; L cls Representing label-aware discriminative losses for H&E and IHC modalities, ensuring that the learned features are semantically aligned with their respective class labels.

[0083] Step 3.3.2: Cross-modal triplet loss: To further strengthen cross-modal alignment, a triplet loss based on L2 distance is designed:

[0084]

[0085] Here, X and Y are used as anchor points to calculate the triplet loss. i X ,P i Y} are cross-modal samples from the same control point, and negative samples are are cross-modal samples from different control points with different class labels.

[0086] Step 3.3.3: The total loss function combines the above components:

[0087] L total =L cls +αL cross-triplet

[0088] where α is the balancing coefficient. This formulation ensures the alignment of labels and positions, making the subspace Z discriminative and robust.

[0089] Step 4: Use the two trained feature encoders to extract image block-level features.

[0090] Step 4.1: HER2 is strongly expressed in colorectal cancer tumor tissues but weakly expressed in some normal tissues, which can lead to false positives in whole-slice analysis. To address this issue, the SLFCD model is used to accurately segment the tumor region, effectively distinguishing between tumor and normal tissue and reducing computational redundancy.

[0091] Step 4.2: Use the explicit knowledge transfer and implicit cross-modal feature reprojection modules as encoders to extract feature vectors from the image processed in step 4.1; use the explicit knowledge transfer encoder and the implicit cross-modal feature reprojection encoder for feature extraction respectively, and concatenate the extracted features to obtain the feature vector F.

[0092] Step 5: Generate full-slice-level prediction results through a multi-instance aggregation mechanism. Aggregation algorithms are used to integrate local features into global features to obtain the final HER2 status classification prediction. The classification prediction can be expressed as:

[0093] L her2 =Multi-Instance(F)

[0094] Among them, L her2 is the predicted full-slice-level HER2 status, and Multi-Instance(·) is the multi-instance aggregator.

[0095] Example 2:

[0096] This embodiment further illustrates the effects of the present invention in combination with simulation experiments:

[0097] The simulation experiments of the present invention were carried out in a hardware environment with Intel Xeon Silver 4110, GeForce RTX 2080, 64GB of memory and a software environment with Ubuntu 18.04.6 and PyTorch 1.12.1. Figure 5 As shown, Figure 5 (a) is the original IHC image; Figure 5 (b) is the IHC annotated image; Figure 5 (c) is the original H&E image; Figure 5(d) is an annotated image of H&E after knowledge transfer. The dataset of the present invention is ColoHER2, which contains 320 high-quality digitized pathology slides. These slides were scanned using the Pannoramic SCAN system with a resolution of 0.12 μm / pixel and a 40x objective lens. The dataset covers all HER2 score categories (0 / 1+ / 2+ / 3+) and is divided into two groups: negative (119 slides) and positive (201 slides), ensuring that the data is representative and comprehensive.

[0098] Table 1 compares the proposed method with state-of-the-art change detection methods. Three feature extraction methods were evaluated: ResNet50 (proposed in the paper "Deep residual learning for image recognition"), CTransPath (proposed in the paper "Transpath: Transformer-based self-supervised learning for histopathological image classification"), and the proposed method. The performance of five multi-instance learning feature aggregators (AB_MIL, CLAM_MIL, DS_MIL~\cite{ref18}, DTFD_MIL~\cite{ref19}, and RRT_MIL~\cite{ref20}) was also compared. Among them, AB_MIL was proposed in the article "Attention-based deep multiple instance learning"; CLAM_MIL was proposed in the article "Data-efficient and weakly supervised computational pathology on whole-slide images"; DS_MIL was proposed in the article "Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning"; DTFD_MIL was proposed in the article "Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification"; RRT_MIL was proposed in the article "Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology".

[0099] This method evaluates the proposed method based on the evaluation indicators of the HEROHE Challenge. These indicators include four parts: AUC (area under the curve), precision, recall, and F1 score. AUC measures the overall discriminative ability of the model by calculating the area under the receiver operating characteristic curve (ROC Curve). It reflects the model's ability to distinguish between positive and negative samples at different classification thresholds. The higher the value, the better the model performance. Precision measures the proportion of samples predicted by the model to be positive that are actually positive. The calculation formula is:

[0100]

[0101] Among them, TP represents true positive examples and FP represents false positive examples.

[0102] The recall rate measures the proportion of positive samples that the model can correctly identify to all positive samples. The calculation formula is:

[0103]

[0104] Among them, FN represents false negative examples.

[0105] The F1 score is the harmonic mean of precision and recall, which is used to comprehensively measure the performance of the model and is particularly suitable for dealing with class imbalance problems. The calculation formula is:

[0106]

[0107] All evaluation metrics are calculated in a class-weighted manner to ensure fair assessment of the contribution of each class in multi-class scenarios.

[0108] Table 1 Classification results of ResNet50 feature extractor

[0109]

[0110] Table 2 Classification results of CTransPath feature extractor

[0111]

[0112] Table 3 Classification results of the feature extractor of the present invention

[0113]

[0114] As shown in Tables 1, 2, and 3, the present invention achieves the current best performance in terms of AUC and F1 score indicators, significantly surpassing existing methods, and significantly improving both precision and recall. The core advantage of the present invention is that it can effectively preserve HER2-specific morphological features. This ability stems from migrating HER2 heterogeneity information from IHC images to H&E-stained images, thereby explicitly guiding the model to identify discriminative pathological features in weakly labeled areas. Figure 6 As shown in the figure, the HER2 spatial distribution heat map presented by the feature extraction results generated by this method is highly consistent with the actual HER2 expression spectrum and DAB color development pattern, verifying the ability of this method to accurately capture the heterogeneous distribution of HER2 biomarkers and has strong consistency with the IHC annotation results.

[0115] While the above describes in detail the method for colorectal cancer HER2 status classification based on a dual-branch collaborative optimization network provided by the present invention, it is apparent that the specific implementation of the present invention is not limited thereto. Any obvious modifications to the method that do not depart from the scope of the claims of the present invention will be readily apparent to those skilled in the art.

Claims

1. A method for classifying HER2 status of colorectal cancer based on a dual-branch collaborative optimization network, characterized in that: The following steps are involved: S1: Acquire H&E whole-slide images and IHC landmark images, perform spatial registration, and obtain an affine transformation matrix to spatially align the H&E image with the IHC image. S2: Using the registered annotation information, a knowledge transfer feature encoder is constructed to transfer the annotation information from the IHC image to the H&E image; the knowledge transfer feature encoder includes a HER2 block-level classifier, which uses a ResNet18 backbone network consisting of a linear layer, a ReLU activation function, a dropout layer, a linear layer, a ReLU activation function, a dropout layer, a linear layer, a ReLU activation function, a dropout layer, and an output layer, and uses a binary cross-entropy loss to optimize the classifier; S3: Based on the affine transformation matrix, IHC and HE image pairs are sampled to construct and train a cross-modal feature reprojection encoder. The cross-modal feature reprojection encoder adopts a cross-modal joint embedding architecture based on a symmetric dual-branch encoder. Its core consists of two ImageNet pre-trained ResNet50 backbone networks, followed by a shared projection layer, which is composed of a linear layer, a ReLU activation function, a dropout layer, another linear layer, and a ReLU activation function. S4: extracting image block-level features using the trained knowledge transfer feature encoder and the cross-modal feature reprojection encoder; S5: Generate full-slice-level prediction results through a multi-instance aggregation mechanism, and integrate local features into global features through an aggregation algorithm to obtain the final HER2 status classification prediction results.

2. The method for classifying HER2 status of colorectal cancer according to claim 1, wherein: Said S1 specifically includes: S1-1: Mark matching landmarks on a paired H&E image denoted as X and an IHC image denoted as Y; S1-2: Manually select and annotate obvious matching landmarks on H&E and IHC images; S1-3: Calculate the affine transformation matrix describing the spatial relationship between the H&E image and the IHC image; S1-4: Map the annotated region in the IHC image to the H&E image space using an affine transformation matrix; S1-5: Generate image block level label L based on the registration result i , providing supervisory signals for subsequent feature extraction and classification.

3. The method for classifying HER2 status of colorectal cancer according to claim 1, wherein: The S2 specifically includes: S2-1: Train the HER2 block-level classifier, using the image block-level label L obtained in S1-5 i Training is performed, where the negative image block is P i X- , the positive image block is P i X+ ; S2-2: The classifier uses a customized ResNet18 backbone network. By replacing its original fully connected layers with a multi-layer perceptron structure consisting of multiple linear layers, ReLU activation functions, and Dropout layers, a linear layer with an output dimension of 1 is used for binary classification problems to generate a single prediction value. S2-3: Use regularized binary cross entropy loss to optimize the classifier: Where N represents the batch size, and Represents the positive sample P i X+ and negative samples P i X- The true label, and Represents the positive sample P i X+ and negative samples P i X- The predicted label of .

4. The method for classifying HER2 status of colorectal cancer according to claim 1, wherein: The S3 includes: S3-1: Align the H&E and IHC image patches using the affine transformation matrix M and the matching control points between the manually annotated H&E and IHC image pairs to obtain a large number of image patch pairs for training the feature reprojection encoder. S3-2: A cross-modal joint embedding architecture based on a symmetric dual-branch encoder is used. Its core consists of two ImageNet pre-trained ResNet50 backbone networks, followed by a shared projection layer, which consists of a linear layer, a ReLU activation function, a dropout layer, another linear layer, and a ReLU activation function. Deep feature extraction is performed on H&E and IHC images respectively. S3-3: To align features between H&E and IHC modalities, label-aware loss and cross-modal triplet loss are used to jointly optimize the cross-modal feature embedding space.

5. The method for classifying HER2 status of colorectal cancer according to claim 4, wherein: The S3-3 includes: S3-31: Label-aware loss is: where σ(·) represents the softmax function; L cls Label-aware discriminative losses representing H&E and IHC modalities, ensuring that the learned features are semantically aligned with their respective class labels; S3-32: Cross-modal triplet loss: Design triplet loss based on L2 distance: X and Y are used as anchor points to calculate the triplet loss; the positive sample pair {P i X ,P i Y } are cross-modal samples from the same control point, and negative samples are are cross-modal samples from different control points with different class labels; S3-33: The total loss function is: L total =L cls +αL cross-triplet where α is the balance coefficient.

6. The method for classifying HER2 status of colorectal cancer according to claim 1, wherein: The S4 is: S4-1: First, the SLFCD model is used to accurately segment the tumor area and effectively distinguish between tumors and normal tissues; S4-2: The knowledge transfer encoder and the cross-modal feature reprojection encoder are used to extract features respectively, and the extracted features are concatenated to obtain a feature vector F.

7. The method for classifying HER2 status of colorectal cancer according to claim 1, wherein: In S5, the classification prediction is expressed as: L her2 =Multi-Instance(F) Among them, L her2 is the predicted full-slice-level HER2 status, and Multi-Instance(·) is the multi-instance aggregator.

Images