Colorectal cancer her2 status classification method based on double-branch cooperative optimization network

Abstract

The application discloses a colorectal cancer HER2 state classification method based on a double-branch cooperative optimization network and belongs to the technical field of pathological image processing. The application adopts a double-path cooperative optimization framework, combines knowledge transfer and cross-modal feature re-projection, efficiently extracts features related to the HER2 state from a conventional hematoxylin-eosin (H&E) staining section, and can directly accurately judge the HER2 state. Through the prior knowledge transfer and cross-modal feature re-projection mechanism, the application breaks through the bottlenecks such as insufficient utilization of prior knowledge and modal heterogeneity interference in the traditional method, and significantly improves the accuracy and robustness of the colorectal cancer HER2 state classification.

Description

Technical Field

[0001] This invention belongs to the field of pathological image processing technology, specifically relating to a method for classifying the HER2 status of colorectal cancer based on a bi-branch collaborative optimization network. Background Technology

[0002] Colorectal cancer (CRC) is a major global public health challenge. Early screening primarily relies on fecal immunochemical testing (FIT) and colonoscopy, the latter involving tissue biopsy combined with H&E staining for pathological diagnosis. However, traditional histological examinations can only confirm the presence of malignant tumors and cannot provide molecular-level tumor characteristic information, which poses a key limitation to prognostic assessment and precision treatment. Against this backdrop, the importance of HER2 testing is increasingly evident, as overexpression or gene amplification of this receptor protein is closely related to the invasiveness and poor prognosis of colorectal cancer. Although immunohistochemistry (IHC) and fluorescence in situ hybridization (FISH) remain the gold standard for HER2 testing, their clinical application faces bottlenecks such as stringent requirements for tissue sample quality, complex testing procedures, and inter-observer variability in interpreting results.

[0003] Current research on HER2 status classification is gradually expanding from breast cancer to colorectal cancer, with deep learning technology showing the potential to replace traditional detection methods. Existing studies have used H&E image analysis to achieve tumor detection and preliminary HER2 status classification, with some works attempting to integrate morphological features such as DAB staining density and nuclear distribution to improve model performance. Notably, constraint matrix optimization methods based on the ASCO / CAP guidelines, and the innovative technique of using generative adversarial networks (GANs) to virtually convert H&E slices into IHC images, offer new ideas for reducing detection costs. However, existing algorithms still primarily rely on low-level image features such as color and texture, failing to fully integrate prior medical knowledge to build a deep cognitive system.

[0004] The lack of integration of prior medical knowledge has become a core issue restricting classification accuracy. The multi-dimensional knowledge pathologists rely on in HER2 interpretation—including the spatial distribution characteristics of IHC-positive regions, the correlation patterns between DAB staining intensity and cellular substructures (such as cell membrane integrity and abnormal nucleocytoplasmic ratio), and key diagnostic clues such as tissue heterogeneity—has not yet been effectively quantified into operational model parameters. These morphological correlations accumulated by human experts through long-term practice, particularly the interaction patterns between cell membrane-specific staining intensity and the tumor microenvironment, are crucial for distinguishing between true and false positive cases. Future research urgently needs to establish knowledge embedding mechanisms to transform expert experience, such as quantitative analysis of cell morphology and tissue spatial topological relationships, into constraints for deep learning, thereby breaking through the current bottleneck of classification based on superficial features. Summary of the Invention

[0005] The purpose of this invention is to provide a HER2 status classification method for colorectal cancer based on a dual-branch collaborative optimization network, so as to solve key problems such as insufficient accuracy of HER2 status classification and insufficient fusion of cross-modal features in the prior art.

[0006] To achieve the above objectives, the specific technical solution adopted by the present invention is as follows:

[0007] A method for classifying the HER2 status of colorectal cancer based on a bi-branch collaborative optimization network includes the following steps:

[0008] S1: Obtain the full slice image of H&E and the IHC logo image, and perform spatial registration to obtain the affine transformation matrix so that the H&E image and the IHC image are spatially aligned.

[0009] S2: A knowledge transfer feature encoder is constructed using the registered annotation information to transfer the annotation information in the IHC image to the H&E image, thereby realizing knowledge sharing between the two image modalities. The knowledge transfer feature encoder includes a HER2 block classifier, which adopts a ResNet18 backbone network and consists of a linear layer, a ReLU activation function, a Dropout layer, a linear layer, a ReLU activation function, a Dropout layer, a linear layer, a ReLU activation function, a Dropout layer, and an output layer. Binary cross-entropy loss is used to optimize the classifier.

[0010] S3: Based on the affine transformation matrix, sample IHC and HE image pairs, construct a cross-modal feature reprojection encoder and train it to achieve effective fusion of image features from the two modalities, further improving the expressive power and consistency of the features; the cross-modal feature reprojection encoder adopts a cross-modal joint embedding architecture based on a symmetric dual-branch encoder, the core of which consists of two ImageNet pre-trained ResNet50 backbone networks, followed by a shared projection layer, which is composed of a linear layer, a ReLU activation function, a Dropout layer, another linear layer and a ReLU activation function connected in sequence;

[0011] S4: Extract image block-level features using the trained knowledge transfer feature encoder and the cross-modal feature reprojection encoder;

[0012] S5: Generate full-slice-level prediction results through a multi-instance aggregation mechanism, and integrate local features into global features through an aggregation algorithm to obtain the final HER2 state classification prediction results.

[0013] Furthermore, S1 specifically includes:

[0014] S1-1: Mark matching points on the paired H&E images (represented as X) and IHC images (represented as Y) for subsequent calculation of the affine transformation matrix;

[0015] S1-2: Manually select and mark obvious, easily identifiable matching markers on the H&E and IHC images;

[0016] S1-3: Calculate the affine transformation matrix that describes the spatial relationship between the H&E image and the IHC image;

[0017] S1-4: Map the labeled regions in the IHC image to the H&E image space using an affine transformation matrix;

[0018] S1-5: Generate image block-level labels L based on the registration results. i This provides a supervisory signal for subsequent feature extraction and classification.

[0019] Furthermore, S2 specifically includes:

[0020] S2-1: Train the HER2 block classifier using the image block labels L obtained in S1-5. i Training is performed, where the negative image patch is P. i X- The positive image patch is P. i X+ ;

[0021] S2-2: The classifier uses a custom ResNet18 backbone network. Its original fully connected layers are replaced with a multi-layer perceptron structure containing multiple linear layers, ReLU activation functions, and Dropout layers to enhance the model's feature learning ability and generalization performance. Specifically, it consists of a linear layer (512, 512), a ReLU activation function, a Dropout layer, a linear layer (512, 256), a ReLU activation function, a Dropout layer, a linear layer (256, 128), a ReLU activation function, a Dropout layer, and an output layer (128, 1). Finally, a linear layer with an output dimension of 1 is used for binary classification to generate a single predicted value.

[0022] S2-3: Optimize the classifier using regularized binary cross-entropy loss:

[0023]

[0024] Where N represents the batch size. and P represents the positive sample respectively. i X+ and negative sample P i X- The true label, and P represents the positive sample respectively. i X+and negative sample P i X- Predicted labels.

[0025] Furthermore, S3 includes:

[0026] S3-1: Align H&E and IHC image patches using affine transformation matrix M and manually labeled matching control points between H&E and IHC image pairs to obtain a large number of image patch pairs for training the feature reprojection encoder.

[0027] S3-2: A cross-modal joint embedding architecture based on a symmetric dual-branch encoder is adopted. Its core consists of two ImageNet pre-trained ResNet50 backbone networks, followed by a shared projection layer, which is composed of a linear layer (2048, 1024), a ReLU activation function, a Dropout layer, another linear layer (1024, 1024), and a ReLU activation function connected in sequence; deep features are extracted from H&E and IHC images respectively.

[0028] S3-3: To align features between H&E and IHC modalities, label-aware loss and cross-modal triplet loss are used to jointly optimize the cross-modal feature embedding space.

[0029] Furthermore, S3-3 includes:

[0030] S3-31: Label-aware loss: To enhance the discriminability of features, a category label-based loss function is applied. i Cross-entropy constraint:

[0031]

[0032] Where σ(·) represents the softmax function; L cls The label-aware discriminant loss for H&E and IHC modalities ensures that the learned features are semantically aligned with their respective category labels.

[0033] S3-32: Cross-modal triplet loss: To further enhance cross-modal alignment, a triplet based on L2 distance is designed.

[0034] loss:

[0035]

[0036] Here, X and Y are used as anchor points to calculate the triplet loss. Positive sample pairs {P} i X ,P i Y} represents cross-modal samples from the same control point, and negative sample pairs These are cross-modal samples from different control points with different category labels.

[0037] S3-33: The total loss function is:

[0038] L total =L cls +αL cross-triplet

[0039] Where α is the balance coefficient.

[0040] Furthermore, S4 is:

[0041] S4-1: First, the SLFCD model is used for accurate tumor region segmentation, effectively distinguishing tumors from normal tissue and reducing computational redundancy. The SLFCD model is a supervised learning-based cancer detection framework primarily used for cancer region localization and image classification in full-slide cancer pathology images.

[0042] S4-2: Then, use the knowledge transfer encoder and the cross-modal feature reprojection encoder to extract features respectively, and concatenate the extracted features to obtain the feature vector F.

[0043] Furthermore, in S5, the classification prediction is represented as:

[0044] L her2 =Multi-Instance(F)

[0045] Among them, L her2 It is the predicted full-slice HER2 state, and Multi-Instance(·) is the multi-instance aggregator.

[0046] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0047] This invention overcomes the bottlenecks of traditional methods, such as insufficient utilization of prior knowledge and interference from modal heterogeneity, by employing prior knowledge transfer and cross-modal feature reprojection mechanisms. This significantly improves the accuracy and robustness of HER2 status classification for colorectal cancer. This invention has the following advantages:

[0048] 1. Prior Knowledge Transfer and Enhancement: This invention constructs a knowledge transfer feature encoder to transfer annotation information from IHC images to H&E images, fully utilizing the prior knowledge of IHC images to compensate for the lack of information in H&E images for HER2 state classification. This knowledge transfer mechanism effectively enhances the feature representation capability of H&E images and improves classification accuracy.

[0049] 2. Cross-modal feature reprojection and fusion: This invention uses an affine transformation matrix to sample H&E and IHC images, constructs a cross-modal feature reprojection encoder, and achieves effective fusion of features from the two modalities, significantly improving the consistency of feature representation and classification performance.

[0050] 3. Multi-Instance Aggregation and Global Prediction: This invention integrates image block-level features into full-slice-level features through a multi-instance aggregation mechanism, generating the final HER2 status classification prediction result. Through an innovative dual-modal joint representation mechanism, it overcomes the shortcomings of traditional methods in utilizing prior knowledge and handling modal heterogeneity, significantly improving the accuracy and robustness of HER2 status classification in colorectal cancer, and providing new technical ideas and solutions for pathological image analysis and clinical diagnosis. Attached Figure Description

[0051] Figure 1 This is a flowchart of the present invention.

[0052] Figure 2 This is a schematic diagram of the knowledge transfer module of the present invention.

[0053] Figure 3 This is a schematic diagram of the cross-modal feature reprojection structure of the present invention.

[0054] Figure 4 This is a schematic diagram of the overall classification process of the present invention.

[0055] Figure 5 This is a schematic diagram of the input data for the present invention.

[0056] Figure 6 This is a comparison chart of the density effects of H&E, IHC, and DAB (3,3'-Diaminobenzidine) in the method of this invention. Detailed Implementation

[0057] In the following description, various aspects of the invention will be described; however, those skilled in the art will recognize that the invention can be practiced using only some or all of its structures or processes. For clarity, specific numbers, arrangements, and sequences are set forth, but it will be apparent that the invention can be practiced without these specific details. In other instances, well-known features will not be described in detail to avoid obscuring the invention.

[0058] Example 1:

[0059] Reference Figure 1 , Figure 4 A method for classifying the HER2 status of colorectal cancer based on a two-branch collaborative optimization network includes the following steps:

[0060] Step 1: Spatial registration is performed on the H&E full slice image and the IHC marker image to obtain the affine transformation matrix, so that the H&E image and the IHC image are spatially aligned.

[0061] Step 1.1: Mark matching points on the paired H&E images (denoted as X) and IHC images (denoted as Y) for subsequent calculation of the affine transformation matrix.

[0062] Step 1.2: Manually select and label approximately 40 pairs of obvious and easily identifiable matching landmarks on the H&E and IHC images. The coordinates of the landmarks in the H&E image are represented as follows: The coordinates of markers in an IHC image are represented as follows: These markers should have sufficient contrast and uniqueness to ensure accurate identification in images of both modalities. Each pair of markers should be distributed in different regions of the image as much as possible to improve the accuracy and robustness of registration.

[0063] Step 1.3: Calculate the affine transformation matrix describing the spatial relationship between the H&E image and the IHC image. Extend the IHC coordinates y to homogeneous form, i.e., add a constant term 1, expressed as... The affine transformation matrix M is solved by minimizing the geometric distortion objective function:

[0064]

[0065] in · represents the transformed y-coordinate, and · represents matrix multiplication.

[0066] Step 1.4: Map the labeled regions in the IHC image to the H&E image space using an affine transformation matrix.

[0067] A H&E =M·A IHC

[0068] A H&E This indicates the corresponding HER2-labeled region in the H&E image.

[0069] Step 1.5: Generate image block-level labels L based on the registration results. i This provides a supervisory signal for subsequent feature extraction and classification. i If the value is 1, the corresponding region is HER2 positive, L i If the value is 0, the corresponding region is HER2 negative.

[0070] Step 2: Construct a knowledge transfer feature encoder using the registered annotation information. Transfer the annotation information from the IHC image to the H&E image to achieve knowledge sharing between the two image modalities.

[0071] Step 2.1: Since the H&E images are captured using whole-slice imaging (WSI), the images are too large to process as a whole. First, a HER2 block-level classifier is trained to achieve block-level HER2 classification. The block-level labels L of the H&E images obtained in Step 1.4 are then used. i Training is performed, where the negative image patch is P. i X- The positive image patch is P. i X+ .

[0072] Step 2.2: As Figure 2 As shown, the classifier employs a custom ResNet18 backbone network. Its original fully connected layers are replaced with a multi-layer perceptron structure containing multiple linear layers, ReLU activation functions, and Dropout layers to enhance the model's feature learning ability and generalization performance. This multi-layer perceptron structure includes three linear layers, each followed by a ReLU activation function and a Dropout layer. Finally, a linear layer with an output dimension of 1 is used for the binary classification problem, generating a single predicted value.

[0073] Step 2.3: Optimize the classifier using regularized binary cross-entropy loss:

[0074]

[0075] Where N represents the batch size. and P represents the positive sample respectively. i X+ and negative sample P i X- The true label, and P represents the positive sample respectively. i X+ and negative sample P i X- Predicted labels.

[0076] Step 3: Based on the affine transformation matrix, sample IHC and HE image pairs to construct a cross-modal feature reprojection encoder. This achieves effective fusion of image features from the two modalities, further enhancing the expressive power and consistency of the features.

[0077] Step 3.1: Align the H&E and IHC image patches using an affine transformation matrix M and approximately 40 manually labeled matching control points between H&E and IHC image pairs. Specifically, each control point in the IHC image is mapped to its corresponding spatial location in the H&E image to ensure accurate patch alignment. Then, 25 non-overlapping patches are extracted by dividing the image into a uniform grid centered on these transformation points to obtain a large number of image patch pairs {P}. iX ,P i Y Training for the feature reprojection encoder.

[0078] Step 3.2: This module adopts a cross-modal joint embedding architecture based on a symmetric dual-branch encoder, such as... Figure 3 As shown, its core consists of two ImageNet pre-trained ResNet50 backbone networks, which extract deep features from H&E and IHC images respectively. To eliminate the difference in feature distribution between modalities, a learnable nonlinear mapping module is introduced at the end of the dual branches. Through joint optimization, the feature vectors of the two modalities are projected onto a common subspace, thereby aligning the geometric structure and semantic relevance of cross-modal features.

[0079] Step 3.3: To align features between H&E and IHC modalities, define a common subspace Z, and mapping functions f:X→Z and g:Y→Z; optimize the cross-modal feature embedding space using the following joint loss function:

[0080] Step 3.3.1: Label-aware loss: To enhance the discriminability of features, a category label-based loss method is applied. i Cross-entropy constraint:

[0081]

[0082] Where σ(·) represents the softmax function; L cls The label-aware discriminant loss for H&E and IHC modalities is used to ensure that the learned features are semantically aligned with their respective category labels.

[0083] Step 3.3.2: Cross-modal triplet loss: To further enhance cross-modal alignment, a triplet loss based on L2 distance is designed:

[0084]

[0085] Here, X and Y are used as anchor points to calculate the triplet loss. Positive sample pairs {P} i X ,P i Y} represents cross-modal samples from the same control point, and negative sample pairs These are cross-modal samples from different control points with different category labels.

[0086] Step 3.3.3: The total loss function combines the above components:

[0087] L total =L cls +αL cross-triplet

[0088] Where α is the balance coefficient. This formula ensures the alignment of labels and positions, making the subspace Z discriminative and robust.

[0089] Step 4: Extract image block-level features using the two feature encoders that have been trained above.

[0090] Step 4.1: HER2 is strongly expressed in colorectal cancer tumor tissues, but weakly expressed in some normal tissues, which may lead to false positives in whole-slice analysis. To address this issue, the SLFCD model is used for accurate tumor region segmentation, effectively distinguishing tumors from normal tissues and reducing computational redundancy.

[0091] Step 4.2: Use the explicit knowledge transfer and implicit cross-modal feature reprojection modules as encoders to extract feature vectors from the image processed in Step 4.1; use the explicit knowledge transfer encoder and the implicit cross-modal feature reprojection encoder to extract features respectively, and concatenate the extracted features to obtain the feature vector F.

[0092] Step 5: Generate full-slice-level prediction results through a multi-instance aggregation mechanism, and integrate local features into global features using an aggregation algorithm to obtain the final HER2 state classification prediction; the classification prediction can be expressed as:

[0093] L her2 =Multi-Instance(F)

[0094] Among them, L her2 It is the predicted full-slice HER2 state, and Multi-Instance(·) is the multi-instance aggregator.

[0095] Example 2:

[0096] This embodiment, combined with simulation experiments, further illustrates the effectiveness of the present invention:

[0097] The simulation experiments of this invention were conducted in a hardware environment of Intel Xeon Silver 4110, GeForce RTX 2080, and 64GB of memory, and a software environment of Ubuntu 18.04.6 and PyTorch 1.12.1. The experimental data of this invention are as follows: Figure 5 As shown, where, Figure 5 (a) is the original IHC image; Figure 5 (b) is the IHC-annotated image; Figure 5 (c) is the original H&E image; Figure 5(d) shows the annotated H&E images after knowledge transfer. The dataset used in this invention is ColoHER2, which contains 320 high-quality digitized pathology slides. These slides were scanned using the Pannoramic SCAN system at a resolution of 0.12 μm / pixel with a 40x objective lens. The dataset covers all HER2 scoring categories (0 / 1+ / 2+ / 3+) and is divided into two groups: negative (119 slides) and positive (201 slides) to ensure data representativeness and comprehensiveness.

[0098] Table 1 shows the comparison results between the method of this invention and existing advanced change detection methods. Three feature extraction methods were evaluated: ResNet50, CTransPath, and the method proposed in this invention. ResNet50 was proposed in the paper "Deep residual learning for image recognition," and CTransPath was proposed in the paper "Transpath: Transformer-based self-supervised learning for histopathological image classification." Furthermore, the performance of five multi-instance learning feature aggregators was compared: AB_MIL, CLAM_MIL, DS_MIL (ref 18), DTFD_MIL (ref 19), and RRT_MIL (ref 20). Among them, AB_MIL was proposed in the paper "Attention-based deep multiple instance learning"; CLAM_MIL was proposed in the paper "Data-efficient and weakly supervised computational pathology on whole-slide images"; DS_MIL was proposed in the paper "Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning"; DTFD_MIL was proposed in the paper "Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology wholeslide image classification"; and RRT_MIL was proposed in the paper "Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology".

[0099] This method evaluates the proposed approach based on the HEROHE Challenge's evaluation metrics, which include four components: AUC (Area Under the Curve), Precision, Recall, and F1 Score. AUC measures the model's overall discriminative ability by calculating the area under the Receiver Operating Characteristic (ROC) curve. It reflects the model's ability to distinguish between positive and negative samples at different classification thresholds; a higher value indicates better model performance. Precision measures the proportion of samples predicted as positive that are actually positive, and is calculated using the following formula:

[0100]

[0101] In this context, TP represents a true positive and FP represents a false positive.

[0102] Recall measures the proportion of positive samples that a model can correctly identify out of all positive samples. The formula is:

[0103]

[0104] Here, FN represents a false negative.

[0105] The F1 score is the harmonic mean of precision and recall, used to comprehensively measure the performance of a model, and is particularly suitable for handling class imbalance problems. The calculation formula is:

[0106]

[0107] All evaluation metrics are calculated using a category-weighted approach to ensure a fair assessment of the contribution of each category in multi-category scenarios.

[0108] Table 1. Classification results of ResNet50 feature extractor

[0109]

[0110] Table 2 Classification results of CTransPath feature extractor

[0111]

[0112] Table 3. Classification results of the feature extractor of the present invention.

[0113]

[0114] As shown in Tables 1, 2, and 3, this invention achieves state-of-the-art performance in both AUC and F1 scores, significantly surpassing existing methods, and also demonstrates a marked improvement in precision and recall. The core advantage of this invention lies in its ability to effectively preserve HER2-specific morphological features. This capability stems from transferring HER2 heterogeneity information from IHC images to H&E-stained images, thereby explicitly guiding the model to identify discriminative pathological features in weakly labeled regions. Figure 6 As shown, the HER2 spatial distribution heatmap generated by this method is highly consistent with the real HER2 expression spectrum and DAB colorimetric mode, verifying the method's ability to accurately capture the heterogeneous distribution of HER2 biomarkers, and it also shows strong consistency with the IHC annotation results.

[0115] The above provides a detailed description of the HER2 status classification method for colorectal cancer based on a bi-branch collaborative optimization network provided by this invention. However, it is obvious that the specific implementation of this invention is not limited thereto. For those skilled in the art, various obvious modifications made to this invention without departing from the scope of the claims are all within the protection scope of this invention.

Claims

1. A method for classifying HER2 status of colorectal cancer based on a bi-branch collaborative optimization network, characterized in that, Includes the following steps: S1: Obtain the full slice image of H&E and the IHC logo image, and perform spatial registration to obtain the affine transformation matrix so that the H&E image and the IHC image are spatially aligned. S2: Construct a knowledge transfer feature encoder using the registered annotation information to transfer the annotation information from the IHC image to the H&E image; the knowledge transfer feature encoder includes a HER2 block classifier, which adopts a ResNet18 backbone network and consists of a linear layer, a ReLU activation function, a Dropout layer, a linear layer, a ReLU activation function, a Dropout layer, a linear layer, a ReLU activation function, a Dropout layer, and an output layer connected together, and uses binary cross-entropy loss to optimize the classifier; S3: Based on the affine transformation matrix, sample IHC and HE image pairs, construct a cross-modal feature reprojection encoder, and train it; the cross-modal feature reprojection encoder adopts a cross-modal joint embedding architecture based on a symmetric dual-branch encoder, the core of which consists of two ImageNet pre-trained ResNet50 backbone networks, followed by a shared projection layer, which is composed of a linear layer, a ReLU activation function, a Dropout layer, another linear layer, and a ReLU activation function connected in sequence; S4: Extract image block-level features using the trained knowledge transfer feature encoder and the cross-modal feature reprojection encoder; S5: Generate full-slice-level prediction results through a multi-instance aggregation mechanism, and integrate local features into global features through an aggregation algorithm to obtain the final HER2 state classification prediction results.

2. The HER2 status classification method for colorectal cancer as described in claim 1, characterized in that, S1 specifically includes: S1-1: Mark matching markers on the paired H&E images represented as X and IHC images represented as Y; S1-2: Manually select and mark obvious matching markers on the H&E and IHC images; S1-3: Calculate the affine transformation matrix that describes the spatial relationship between the H&E image and the IHC image; S1-4: Map the labeled regions in the IHC image to the H&E image space using an affine transformation matrix; S1-5: Generate image block-level labels based on the registration results. This provides a supervisory signal for subsequent feature extraction and classification.

3. The HER2 status classification method for colorectal cancer as described in claim 1, characterized in that, S2 specifically includes: S2-1: Train the HER2 block classifier using the image block labels obtained in S1-5. Training is performed, where the negative image patch is Positive image patch is ; S2-2: The classifier uses a custom ResNet18 backbone network. By replacing its original fully connected layers with a multilayer perceptron structure containing multiple linear layers, ReLU activation functions and Dropout layers, a single linear layer with an output dimension of 1 is used for binary classification to generate a single predicted value. S2-3: Optimize the classifier using regularized binary cross-entropy loss: in Indicates batch size. and These represent positive samples respectively. and negative samples The true label, and These represent positive samples respectively. and negative samples Predicted labels.

4. The HER2 status classification method for colorectal cancer as described in claim 1, characterized in that, S3 includes: S3-1: Using affine transformation matrices Matching control points between manually labeled H&E and IHC image pairs are used to align H&E and IHC image patches to obtain a large number of image patch pairs for training the feature reprojection encoder. S3-2: A cross-modal joint embedding architecture based on a symmetric dual-branch encoder is adopted. Its core consists of two ImageNet pre-trained ResNet50 backbone networks, followed by a shared projection layer, which is composed of a linear layer, a ReLU activation function, a Dropout layer, another linear layer, and a ReLU activation function connected in sequence; deep features are extracted from H&E and IHC images respectively. S3-3: To align features between H&E and IHC modalities, label-aware loss and cross-modal triplet loss are used to jointly optimize the cross-modal feature embedding space.

5. The HER2 status classification method for colorectal cancer as described in claim 4, characterized in that, S3-3 includes: S3-31: The label-aware loss is: ； in This represents the softmax function; The label-aware discriminant loss for H&E and IHC modalities ensures that the learned features are semantically aligned with their respective category labels. S3-32: Cross-modal triplet loss: Designing a triplet loss based on L2 distance: and They are used as anchor points to calculate the triplet loss; positive sample pairs These are cross-modal samples from the same control point, negative sample pairs. These are cross-modal samples from different control points with different category labels; S3-33: The total loss function is: in It is the balance coefficient.

6. The method for classifying HER2 status of colorectal cancer as described in claim 1, characterized in that, S4 is: S4-1: First, the SLFCD model is used for accurate tumor region segmentation, effectively distinguishing tumors from normal tissues; S4-2: Then, use the knowledge transfer feature encoder and the cross-modal feature reprojection encoder to extract features respectively, and concatenate the extracted features to obtain a feature vector. .

7. The HER2 status classification method for colorectal cancer as described in claim 1, characterized in that, In S5, the classification prediction is expressed as: in, It is the predicted full-slice HER2 state. It is a multi-instance aggregator.

Images