Method for grading tumor her2 expression based on he-stained images

Abstract

The present application relates to a kind of based on HE staining image tumor HER2 expression grading method.It includes: each target HE image block in target HE image block group is respectively loaded into IHC image generation model, to generate the IHC image block to be examined with current target HE image block in structure alignment state using IHC image model;Each IHC image block to be examined is loaded into multimodal semantic classification model, to use the multimodal semantic classification model to carry out expression grading processing, and the HER2 expression grade prediction information of current IHC image block to be examined is generated after grading processing.The present application can realize tumor HER2 expression grading prediction based on HE staining image, reduce the dependence on IHC staining technology, while improving the accuracy, stability and clinical availability of HER2 expression grading prediction.

Description

Technical Field

[0001] This invention relates to a method for grading tumor HER2 expression, and more particularly to a method for grading tumor HER2 expression based on HE staining images. Background Technology

[0002] Molecular subtyping of tumors is of great significance in clinical treatment. Among them, the expression level of HER2 protein (0, 1+, 2+, 3+) is an important indicator for determining the applicability of targeted therapy in some tumor types. Traditional HER2 subtyping relies on immunohistochemistry (IHC) staining of sections, which is mainly observed and interpreted by professional pathologists under a microscope. However, obtaining HER2 expression status by IHC staining is costly, time-consuming, and requires high quality of stained sections and strict operational procedures, limiting its application in large-scale screening and automated analysis.

[0003] In contrast, hematoxylin and eosin (HE) staining images, as a standard tissue staining technique, are inexpensive to obtain and widely available in medical institutions at all levels, possessing greater practicality and scalability. Therefore, how to indirectly predict HER2 expression levels using HE staining images has become a research focus. However, inferring HER2 expression levels from HE staining images still faces several key technical challenges, specifically:

[0004] First, when aligning HE-stained images with IHC-stained images in space, existing methods mostly adopt a single-scale registration strategy, which makes it difficult to achieve high-precision registration at both the whole-image level and the patch level. Especially when there are tissue structure deformations, slice thickness differences, or image occlusions between HE-stained images and IHC-stained images, using only coarse registration or single-patch-level registration often fails to meet the positional consistency requirements of subsequent image generation and classification tasks.

[0005] Secondly, although some methods have attempted to generate IHC-stained images from HE-stained images using image generation models, these methods generally lack a complete HE→IHC→hierarchical task chain, resulting in a lack of practical usability and inter-module coordination mechanisms. Furthermore, during image generation, due to the high resolution and rich structural details of pathological images, existing generative adversarial networks often suffer from mode collapse, manifesting as repetitive structures, missing details, or abnormal color distribution in the output image, leading to distorted staining features and severely impacting the clinical interpretability and reliability of the generated images. Existing literature rarely provides systematic modeling for the structural consistency of membrane-stained regions, sample diversity control, and high-frequency detail restoration, thus limiting the application performance of image generation models in medical imaging scenarios.

[0006] When classifying HER2 expression levels, manual annotation is required. Specifically, manual annotation is characterized by strong subjectivity and blurred inter-class boundaries. In particular, misjudgments are prone to occur between 1+ and 2+, and between 2+ and 3+, resulting in insufficient generalization performance of traditional CNN or shallow classification models in HER2 expression classification.

[0007] Multimodal visual language models that have emerged in recent years possess strong semantic modeling capabilities. However, there is currently a lack of a complete mechanism to effectively introduce them into HER2 expression grading tasks. Especially when HER2 grade semantics are not covered by pre-trained models, direct application often leads to the failure of text supervision. Existing research usually neglects to introduce domain knowledge (such as HER2 interpretation criteria) into the language encoding process, or fails to design prototype guidance mechanisms with semantic transfer capabilities, making it difficult to accurately model fine-grained grade differences in tumor pathology images.

[0008] Application CN114820555A discloses a breast cancer pathological image classification method based on SENet channel attention and transfer learning. This method first extracts features from breast cancer pathological images by combining wavelet transform and transfer learning, then fuses the features using the SENet channel attention mechanism, and finally constructs a classifier to achieve effective classification of pathological images. This application improves the accuracy of benign and malignant breast cancer classification by enhancing feature expression capabilities, thus solving the problems of feature loss and overfitting in traditional CNN models. However, this application can only classify single-stained images, does not consider the registration and conversion between different stained sections, and does not involve the prediction task of HER2 expression level, making it difficult to meet the needs of molecular-level auxiliary diagnosis.

[0009] Application CN117765252A discloses a breast cancer identification system and method based on Swing Transformer and contrastive learning, including the following steps: First, acquire breast cancer X-ray image data; then, crop the original image into a 224×224 pixel three-channel image to form input image samples; next, build a Swing Transformer neural network structure and introduce the SwingCLR framework, and perform unsupervised pre-training of the model using unlabeled images in the SimCLR manner; then, perform supervised training of the model using labeled images; finally, output the probability of breast cancer corresponding to the image.

[0010] As explained above, application CN117765252A combines the advantages of contrastive learning and SwingTransformer, improving model performance and enhancing the accuracy and training efficiency of breast cancer identification even in the absence of labeled data. However, this application primarily focuses on the overall identification and analysis of breast cancer X-ray images, with its task being the detection of the presence of the disease (i.e., whether cancer is present). It does not involve the analysis of HE-stained and IHC-stained images at the tissue section level, nor does it include a HER2 expression grading task. Furthermore, this method does not consider staining conversion or registration issues between images, making it difficult to apply to molecular pathology-assisted diagnosis scenarios based on HE image inference of HER2 expression.

[0011] Application CN119006942A discloses a method and system for identifying and classifying breast cancer pathological images, comprising the following steps: collecting breast cancer pathological images and preprocessing them; extracting multi-scale texture features of the images based on convolutional neural networks, and capturing local and global texture information in breast tissue by designing convolutional kernels with different receptive field sizes; then using an attention mechanism to adaptively weight the extracted texture features, dynamically adjusting the feature weights according to the importance of different regions, and extracting key texture features; subsequently using a sequence feature selection algorithm to screen the key texture features, and obtaining the optimal feature subset through iterative search and cross-validation; finally, constructing an ensemble learning classification model based on the selected features to identify and classify the images to be predicted.

[0012] As explained above, the application with publication number CN119006942A has a certain effect on improving the accuracy and interpretability of breast cancer pathology image classification. However, this application only performs image-level classification of texture structures in breast cancer pathology images and does not involve tasks such as image registration between staining methods, staining image generation, or HER2 expression level classification. It is difficult to use it to replace expression prediction scenarios for IHC staining, and it does not have image generation and multimodal semantic reasoning capabilities, so its application scope is relatively limited.

[0013] In summary, existing technologies have significant shortcomings in terms of slice image registration accuracy, stained image generation quality, and HER2 expression level discrimination stability. There is an urgent need for a method that covers image preprocessing, generation, and classification to meet the application needs in medical imaging scenarios. Summary of the Invention

[0014] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a tumor HER2 expression grading method based on HE staining images. This method can predict the HER2 expression grading of tumors based on HE staining images, reduce the dependence on IHC staining images, and improve the accuracy, stability and clinical usability of HER2 expression grading prediction.

[0015] According to the technical solution provided by the present invention, a tumor HER2 expression grading method based on HE staining images is provided, the tumor HER2 expression grading method comprising:

[0016] A reference HE staining image is provided, and the reference HE staining image is at least segmented and screened to generate a target HE image block group after segmentation and screening, wherein the target HE image blocks in the target HE image block group all contain tumor tissue information;

[0017] Each target HE image block in the target HE image block group is loaded into the IHC staining image generation model to generate an IHC image block to be examined that is structurally aligned with the current target HE image block.

[0018] Each IHC image patch to be inspected is loaded into a multimodal semantic classification model for expression grading. After grading, HER2 expression level prediction information for the current IHC image patch to be inspected is generated.

[0019] When performing expression grading processing, the multimodal semantic classification model extracts the visual embedding features of the IHC image patch to be inspected, and calculates and determines the target HER2 grade information that is optimally approximated with the visual embedding features;

[0020] The target HER2 level information is configured as the expression level prediction information for the current IHC image block to be examined.

[0021] When calculating the target HER2 level information that best approximates the visual embedding features, the following steps are included:

[0022] Calculate the feature similarity between the visual embedded features and each HER2-level semantic encoded feature within the reference semantic encoded feature group;

[0023] For any visual embedding feature, when the calculated feature similarity is at its maximum, the visual embedding feature and the corresponding HER2 level semantic coding feature are in the optimal approximation state. At this time, the corresponding HER2 level semantic coding feature is configured as the optimal HER2 level semantic coding feature.

[0024] Configure the HER2 level information in the optimal HER2 level semantic coding features as the target HER2 level information.

[0025] The multimodal semantic classification model includes at least a visual encoder module, a language encoder module, and a graph-text alignment inference module, wherein...

[0026] Visual embedding features of the IHC image patch to be inspected are extracted using a visual encoder module.

[0027] A reference semantic coding feature group is generated using the language encoder module. When generating the reference semantic coding feature group, the HER2 level bootstrap information group is loaded into the language encoder module so that the language encoder module can perform semantic coding on each HER2 level bootstrap information group and generate the corresponding HER2 level semantic coding feature after language coding. The HER2 level semantic coding features in the reference semantic coding feature group are independent of each other.

[0028] The image-text alignment inference module is used to calculate the feature similarity between the visual embedded features and the semantic coding features of each HER2 level. Based on the calculated feature similarity, the target HER2 level information is determined, and the expression level prediction information of the current IHC image block to be inspected is output.

[0029] The multimodal semantic classification model also includes a category prototype matching module, wherein,

[0030] When the feature similarity calculated by the image-text alignment reasoning module is in a fuzzy approximation state, the category prototype matching module is configured to perform prototype similarity calculation processing to determine the target HER2 level information through prototype similarity calculation processing.

[0031] When performing prototype similarity calculation, the prototype similarity between the visual embedded features and the category prototype features within the category prototype matching module is calculated separately, and the optimal prototype similarity and the target category prototype feature corresponding to the optimal prototype similarity are determined.

[0032] Target HER2 level information is generated based on the target category prototype features.

[0033] The target HER2 level information is configured as the expression level prediction information of the current IHC image block to be inspected, and the category prototype matching module is configured to output the expression level prediction information of the current IHC image block to be inspected.

[0034] The feature similarity calculated by the image-text alignment reasoning module includes cosine similarity, where...

[0035] When cosine similarity is used for feature similarity, then:

[0036]

[0037] in, For feature similarity, For visual embedding features, For category HER2 level semantic encoding features For visual embedding features and categories HER2 level semantic encoding features vector dot product, For visual embedding features and categories HER2 level semantic encoding features Modulo multiplication of vectors.

[0038] When constructing a multimodal semantic classification model, the following are included:

[0039] Select a pre-trained framework model and configure it as the base model for multimodal semantic classification. The base model for multimodal semantic classification includes a visual encoder module, a language encoder module, a cue enhancement module, a graph-text alignment reasoning module, and a basic module for category prototype matching.

[0040] A classification model training dataset is constructed to train a multimodal semantic classification base model. The classification model training dataset includes several classification training samples. Each classification training sample includes a training IHC image patch and training expression level text information corresponding to the current training IHC image patch.

[0041] Configure the training conditions for the multimodal semantic classification base model until it reaches the target state. Then, generate a multimodal semantic classification model based on the trained base model that has reached the target state.

[0042] During model training, the corresponding weight parameters of the visual encoder module and the language encoder module are frozen, the cue enhancement module fine-tunes the corresponding weight parameters through the backpropagation mechanism, and the category prototype matching basic module updates the corresponding category prototype features in the training of each batch of classification training samples through the momentum update mechanism.

[0043] During model training, the prompting enhancement module concatenates each HER2 level description text in the HER2 level description text group into a learnable training vector to generate HER2 level training source information corresponding to each HER2 level description text. Subsequently, the language encoder module performs semantic encoding on the HER2 level training source information to generate HER2 level training encoded features.

[0044] The visual encoder module is used to extract the training visual features of the training IHC image patch within each classification training sample. Then, the image-text alignment inference module is configured to calculate the feature similarity between the training visual features and all HER2 level training encoded features. Finally, the category prototype matching module is used to calculate and update the category prototype features corresponding to each training expression level information.

[0045] When calculating the prototype features of each training representation level information category, we have:

[0046]

[0047] in, Classify training samples within each batch The number of training samples for classification. Within each annotation classification training sample Training visual features of each classification training sample Within each batch of training samples for classification The training representation level text information of each classification training sample, For category The category prototype features corresponding to the training samples;

[0048] For each category Update category prototype features through momentum update mechanism Then:

[0049]

[0050] in: To update the momentum coefficient.

[0051] The IHC staining image generation model includes at least an image generator. The target HE image block is loaded into the IHC staining image generator, which then generates the IHC image block to be examined.

[0052] The IHC staining image generation model is generated based on the IHC staining image generation base model after model training. The IHC staining image generation base model includes an image generator and an image discriminator. The IHC staining image generation base model is trained using the generated model training dataset.

[0053] The generative model training dataset includes several generated training samples. Each generated training sample includes a training HE image patch and a corresponding baseline IHC image patch. The training HE image patch and the baseline IHC image patch within the same generated training sample are formed based on the same tumor tissue slice. At the same time, both the training HE image patch and the baseline IHC image patch contain tumor tissue information, and the training HE image patch and the baseline IHC image patch within the same training sample are at least aligned at the tissue structure level.

[0054] Configure the IHC staining image generation base model to train the required generative model until the training effect of the IHC staining image generation base model reaches the target state of the generative model. After that, generate the IHC staining image generation model based on the IHC staining image generation base model that has reached the target state of the generative model.

[0055] When creating the training dataset for the generative model, generate training samples within the training dataset for the generative model, where,

[0056] When creating training samples, the following steps are included:

[0057] Obtain source images of HE-stained sections and paired IHC-stained sections from the same tumor tissue section;

[0058] The source images of HE-stained and IHC-stained sections were registered in multiple stages to generate HE-stained and IHC-stained registration images.

[0059] When performing multi-stage registration, it should include at least the following sequential steps: full-map coarse registration, mesoscale regional non-rigid registration, and patch-level fine registration.

[0060] After performing full-image coarse registration, the source images of HE-stained sections and IHC-stained sections are initially aligned.

[0061] After performing non-rigid registration in the mesoscale region, local offsets of tissue structures are corrected.

[0062] After performing tile-level fine registration, IHC staining registration map and HE staining registration map are generated respectively, and the IHC staining registration map and HE staining registration map correspond to each other at the pixel scale in terms of cell boundaries, nuclear structures and membrane staining areas.

[0063] HE staining registration maps and IHC staining registration maps are segmented and screened to generate training HE image blocks and corresponding baseline IHC image blocks within the training samples. The training HE image blocks and baseline IHC image blocks are at least aligned at the tissue structure level.

[0064] When segmenting and screening HE staining registration maps and IHC staining registration maps, the following steps are included:

[0065] HE training candidate regions are selected sequentially on the HE staining registration map, and tumor region identification is performed on the HE training candidate regions. After tumor region identification, the binary mask of each pixel in the HE training candidate region is determined. When the binary mask is 1, it indicates that the current pixel belongs to the tumor region. When the binary mask is 0, it indicates that the current pixel does not belong to the tumor region.

[0066] Based on the binary mask of each pixel in the HE training candidate region, the proportion of the tumor region is calculated. When the calculated proportion of the tumor region matches the region selection threshold, the current HE training candidate region is configured as the training HE image patch.

[0067] Based on the positional state of the training HE image blocks, the corresponding baseline IHC image blocks are segmented on the IHC staining registration map.

[0068] The advantages of this invention are as follows: When constructing training samples for the basic model of IHC staining image generation, multi-stage registration is performed on the source images of HE staining sections and IHC staining sections. After multi-stage registration, IHC staining registration maps and HE staining registration maps that achieve spatial consistency of key tissue morphology at the WSI (Whole Slice Imaging) level can be generated. Subsequently, training HE image blocks and benchmark IHC image blocks within the corresponding training samples can be segmented based on the IHC staining registration maps and HE staining registration maps. After training the basic model of IHC staining image generation using the training samples prepared in this way and obtaining the IHC staining image generation model, the IHC image blocks to be detected generated by the IHC staining generation model can be structurally aligned with the target HE image blocks. This provides favorable support for subsequent expression grading processing using a multimodal semantic classification model, improving the accuracy and reliability of obtaining expression level prediction information.

[0069] When generating expression level prediction information, IHC staining image patches obtained by scanning or other means are not directly used, thereby reducing the dependence on IHC staining technology, significantly reducing the frequency of IHC staining technology use, and reducing medical costs.

[0070] By introducing a dual discriminator structure of local discriminator and global discriminator into the basic model for IHC staining image generation, and by introducing frequency domain structure preservation loss and sample feature diversity regularization loss into the generation loss function, the pattern collapse problem in medical image generation can be effectively alleviated, the ability to restore details of membrane staining regions can be enhanced, and the fidelity and consistency of generated IHC image patches can be improved.

[0071] When the feature similarity calculated by the image-text alignment inference module is in a fuzzy approximation state, the category prototype matching module is configured to perform prototype similarity calculation processing to determine the target HER2 level information through prototype similarity calculation processing. In this way, when there are fuzzy boundaries in the HER2 expression level, the model's discrimination and generalization ability can be improved, which is particularly suitable for the discrimination of 2+ and 1+ / 3+ boundary samples.

[0072] A complete system was constructed, from multi-stage registration preparation to generate training samples, training to obtain an IHC staining image generation model, to a multimodal semantic classification model. It supports modular deployment and end-to-end inference, and can choose whether to use the generation path or directly perform classification based on the reference HE staining according to the deployment scenario, thereby improving the convenience of deployment. Attached Figure Description

[0073] Figure 1This is a flowchart of one embodiment of the tumor HER2 expression grading method of the present invention.

[0074] Figure 2 This is a schematic diagram of an embodiment where multi-stage registration of HE-stained and IHC-stained source images is not performed.

[0075] Figure 3 This is a schematic diagram of an embodiment of the present invention that generates HE staining registration map and IHC staining registration map after multi-stage registration.

[0076] Figure 4 This is a schematic diagram of an embodiment of the present invention that generates training HE image blocks and benchmark IHC image blocks through segmentation and screening.

[0077] Figure 5 This is a schematic diagram of an embodiment of the basic model for generating HER2 IHC staining images according to the present invention.

[0078] Figure 6 This is a structural block diagram of one embodiment of the global discriminator of the present invention.

[0079] Figure 7 This is a structural block diagram of one embodiment of the local discriminator of the present invention.

[0080] Figure 8 This is a structural block diagram of one embodiment of the multimodal semantic classification model of the present invention. Detailed Implementation

[0081] The present invention will be further described below with reference to specific accompanying drawings and embodiments.

[0082] To reduce reliance on IHC staining technology and predict tumor HER2 expression grading based on HE staining images, this invention provides a tumor HER2 expression grading method based on HE staining images. Specifically, the tumor HER2 expression grading method includes:

[0083] A reference HE staining image is provided, and the reference HE staining image is at least segmented and screened to generate a target HE image block group after segmentation and screening, wherein the target HE image blocks in the target HE image block group all contain tumor tissue information;

[0084] Each target HE image block in the target HE image block group is loaded into the IHC staining image generation model to generate an IHC image block to be examined that is structurally aligned with the current target HE image block.

[0085] Each IHC image patch to be inspected is loaded into a multimodal semantic classification model for expression grading. After grading, HER2 expression level prediction information for the current IHC image patch to be inspected is generated.

[0086] When performing expression grading processing, the multimodal semantic classification model extracts the visual embedding features of the IHC image patch to be inspected, and calculates and determines the target HER2 grade information that is optimally approximated with the visual embedding features;

[0087] The target HER2 level information is configured as the expression level prediction information for the current IHC image block to be examined.

[0088] It should be noted that this invention achieves tumor HER2 expression grading based on HE staining images. Specifically, it means that the tumor HER2 expression level can be predicted using only HE staining images, i.e., it does not rely on tumor IHC staining images, or tumor IHC staining technology, when predicting the tumor HER2 expression level. Specifically, the provided HE staining images should be HE staining images of tumor tissue to be graded for HER2 expression. Currently, tumor types for which HER2 expression detection is clinically significant include breast cancer, cervical cancer, and gastric cancer.

[0089] Depend on Figure 1 It is understood that when implementing the tumor HER2 expression grading method of the present invention, a reference HE staining image should be provided. The reference HE staining image should be the HE staining image corresponding to the tumor HER2 expression level to be predicted. For example, for a tumor tissue sample, when it is necessary to determine the tumor HER2 expression level, only the HE staining image of the tumor tissue sample can be provided. That is, the obtained HE staining image is used as the reference HE staining image, and the tumor HER2 expression level can be finally predicted using the reference HE staining image.

[0090] It should be understood that reference HE-stained images can be provided using techniques commonly used in this field, such as acquiring them through a full-field pathological slide scanning system and generating a pyramid-structured digital image file. The preferred resolution of the reference HE-stained image is 100,000 × 80,000 pixels, and the image format is .tiff or other formats that support multi-resolution access. To predict the HER2 expression level of tumors, [further details are needed]. Figure 1 It is known that the reference HE-stained image should be segmented and screened at least so that a target HE image block group can be generated after segmentation and screening. The target HE image block group includes several target HE image blocks, and each target HE image block contains tumor tissue information. That is, the prediction of tumor HER2 expression level is only necessary when the target HE image block contains tumor tissue information.

[0091] Specifically, the target HE image patch contains tumor tissue information, meaning that all target HE-stained images contain tumor tissue, which is the type of tumor mentioned above, such as breast cancer, cervical cancer, or gastric cancer. In the following description, "containing tumor tissue information" has the same meaning and can be found here. The method for segmenting and filtering the reference HE-stained image can be found in the corresponding description below. Furthermore, all target HE image patches have the same scale.

[0092] Depend on Figure 1 It is known that after generating the target HE image block group, each target HE image block within the target HE image block should be loaded into the IHC staining image generation model. Each target HE image block, after being loaded into the IHC image block generation model, can generate a corresponding IHC image block to be examined. The generated IHC image block to be examined should be structurally aligned with the target HE image currently loaded into the IHC image block generation model. Therefore, when the target HE image block group includes multiple target HE image blocks, loading them into the IHC image block generation model will generate multiple corresponding IHC image blocks to be examined. The number of IHC image blocks to be examined should be consistent with the number of target HE image blocks loaded into the IHC image block generation model and should correspond one-to-one. Generally, the scale of the IHC image block to be examined is consistent with the scale of the target HE image block.

[0093] It should be noted that the tumor HER2 expression grading of this invention specifically refers to determining the HER2 expression level of each IHC image block to be examined. Therefore, after generating the IHC image block to be examined, it should be loaded into a multimodal semantic classification model for HER2 expression grading prediction. The multimodal semantic classification model is used to perform expression grading processing on the current IHC image block to be examined. After expression grading processing, the expression level prediction information of the current IHC image block to be examined can be generated. The generated expression level prediction information is the tumor HER2 expression grading prediction result of the IHC image block to be examined. This realizes the prediction of tumor HER2 expression level. Since the IHC image block obtained by scanning is not directly used when grading tumor HER2 expression, the dependence on IHC staining images / IHC staining technology is reduced.

[0094] In one embodiment of the present invention, when performing expression grading processing, the visual embedding features of the IHC image block to be inspected should be extracted. Then, the target HER2 grade information that is optimally approximated with the visual embedding features is calculated and determined. After the target HER2 grade information is determined, the determined target HER2 grade information can be configured as the expression grade prediction information of the current IHC image block to be inspected.

[0095] As explained above, for each target HE image patch, the IHC staining image generation model can generate a corresponding IHC image patch to be examined. Subsequently, a multimodal semantic classification model is used to perform expression grading processing on a target HER2 image patch, thereby determining the predicted expression level information of the target IHC image patch. It should be understood that determining the predicted expression level information for each target IHC image patch also determines the predicted expression level information for each target HE image patch. Once the predicted expression level information for all target HE image patches is determined, partial / complete prediction information can be used to provide medical diagnostic references for clinical pathology experts.

[0096] In one embodiment of the present invention, calculating and determining the target HER2 level information that best approximates the visual embedding features includes:

[0097] Calculate the feature similarity between the visual embedded features and each HER2-level semantic encoded feature within the reference semantic encoded feature group;

[0098] For any visual embedding feature, when the calculated feature similarity is at its maximum, the visual embedding feature and the corresponding HER2 level semantic coding feature are in the optimal approximation state. At this time, the corresponding HER2 level semantic coding feature is configured as the optimal HER2 level semantic coding feature.

[0099] Configure the HER2 level information in the optimal HER2 level semantic coding features as the target HER2 level information.

[0100] It should be noted that when determining the target HER2 level information that best approximates the visual embedding features, a reference semantic coding feature group should also be generated within the multimodal semantic classification model. The method for generating the reference semantic coding feature group will be explained in detail below. Specifically, the reference semantic coding feature group includes multiple HER2 level semantic coding features, and each HER2 level coding semantic feature is independent of the other HER2 level coding semantic features.

[0101] After generating the reference semantic coding feature group, a multimodal semantic classification model should be configured to calculate the feature similarity between the visual embedding feature and each HER2 level semantic coding feature. Specifically, after calculating the feature similarity between the visual embedding feature and each HER2 level semantic coding feature, the maximum feature similarity can be determined. At this point, the visual embedding feature and the corresponding HER2 level semantic coding feature are in the optimal approximation state. The corresponding HER2 level semantic coding feature here is the HER2 level semantic coding feature with the maximum feature similarity determined. Afterward, the corresponding HER2 level semantic coding feature can be configured as the optimal HER2 level semantic coding feature.

[0102] After determining the optimal HER2 level semantic coding features, the HER2 level information within the optimal HER2 level semantic coding features can be extracted, and the extracted and determined HER2 level information can be configured as the target HER2 level information, thus realizing the determination of the target HER2 level information.

[0103] In practice, feature similarity can be cosine similarity. When cosine similarity is used, the feature similarity can be calculated as follows:

[0104]

[0105] in, For feature similarity, For visual embedding features, For category HER2 level semantic encoding features For visual embedding features and categories HER2 level semantic encoding features vector dot product, For visual embedding features and categories HER2 level semantic encoding features The modular multiplication of vectors. Generally, HER2-level semantic encoding features and visual embedding features have the same dimensionality for categories. For the meaning of , please refer to the following explanation. For the HER2 level semantic encoding features, please refer to the corresponding explanation below.

[0106] In one embodiment of the present invention, the IHC staining image generation model includes at least an image generator, which loads the target HE image block into the IHC staining image generator so that the IHC staining image generator can generate the IHC image block to be examined.

[0107] Specifically, the image generator can adopt existing common forms, such as an image generator consisting of an encoder network and a decoder network. The encoder network can adopt a residual network structure (such as ResNet-101). Within the image generator, the encoder network extracts features from the target HE image block. The decoder network should generally be structurally symmetrical with the encoder network so that the encoded features output by the encoder network are upsampled layer by layer and reconstructed into IHC image blocks. Skip connections are used to retain the low-level spatial information in the encoder network. Of course, the image generator can also adopt other forms, as long as it can meet the requirements for generating the required HER2 image block to be detected.

[0108] In one embodiment of the present invention, the IHC staining image generation model is generated based on the IHC staining image generation basic model after model training. The IHC staining image generation basic model includes an image generator and an image discriminator. The IHC staining image generation basic model is trained using the generated model training dataset.

[0109] The generative model training dataset includes several generated training samples. Each generated training sample includes a training HE image patch and a corresponding baseline IHC image patch. The training HE image patch and the baseline IHC image patch within the same generated training sample are formed based on the same tumor tissue slice. At the same time, both the training HE image patch and the baseline IHC image patch contain tumor tissue information, and the training HE image patch and the baseline IHC image patch within the same training sample are at least aligned at the tissue structure level.

[0110] Configure the training conditions for the generative model to train the basic model for IHC staining image generation until the basic model for IHC staining image generation reaches the target state of the generative model. After that, generate the IHC staining image generation model based on the basic model for IHC staining image generation that has reached the target state of the generative model.

[0111] It should be understood that when constructing the above-mentioned IHC staining image generation model, a basic IHC staining image generation model should first be constructed, and the corresponding IHC staining image generation model can be generated after training the basic IHC staining image generation model. Figure 5 The diagram illustrates one embodiment of the constructed IHC staining image generation basic model. As shown in the diagram, compared with the trained IHC staining image generation model, the IHC staining image generation basic model should also include an image discriminator. The image discriminator is used to determine the authenticity of the generated training IHC image patches. That is, the image discriminator is mainly used to participate in the model training of the IHC staining image generation basic model. The image generator in the IHC staining image generation basic model should adopt the same structural form as the image generator in the IHC staining image generation basic model. For details on the image generator in the IHC staining image generation basic model, please refer to the above description.

[0112] To train the basic model for generating IHC staining images, a training dataset should be created. Generally, this dataset should include several training samples, each containing a training HE image patch and a baseline IHC image patch. To ensure the image generator produces IHC image patches structurally aligned with the target HE image patch, specifically, the training HE image patch and the baseline IHC image patch within the same training sample should originate from the same tumor tissue slice, and they should be at least structurally aligned. The tumor tissue slice can be taken from cancerous tissues such as breast cancer, gastric cancer, or cervical cancer, as mentioned above. Furthermore, to ensure the generated IHC image patch contains tumor tissue information, both the training HE image patch and the baseline IHC image patch should contain tumor tissue information.

[0113] When training the IHC staining image generation base model using the generative model training dataset, the training HE image patch in each generated training sample is loaded into the image generator in the IHC staining image generation base model so that the image generator can generate the corresponding pseudo IHC image patch. The benchmark IHC image patch can be used to compare the generated pseudo IHC image patch. That is, the benchmark IHC image patch is mainly used to calculate the loss function of the generative model.

[0114] It should be understood that when training the basic model for IHC staining image generation, training conditions should be configured. These conditions typically include a loss function, and may also include: using the Adam optimizer, setting the initial learning rate to 2e-4, a batch size of 16, and 200 training epochs. Specifically, the meanings of the training epochs and batch size are consistent with existing techniques. After training the basic IHC staining image generation model on the training dataset for 200 epochs, training can be terminated. At this point, an IHC staining image generation model can be generated based on the basic model trained for 200 epochs. In other words, after 200 epochs, the basic IHC staining image generation model reaches its target state. At this point, the image discriminator within the basic model trained for the 200th epoch is stopped, and only the image generator is configured to operate, thus generating the IHC staining image generation model.

[0115] In addition, other methods can be used to determine whether the basic model for generating IHC-stained images has reached the target state of the generation model. For example, after training the basic model for generating IHC-stained images for 200 rounds, it can be validated on a validation set. When the loss function of the generation model validated on the validation set is less than the generation training loss threshold and tends to be stable, it is considered that the corresponding basic model for generating IHC-stained images has reached the target state of the generation model. Other cases will not be listed here. Generally, the size of the generation training loss threshold can be selected according to actual needs.

[0116] To address the common mode collapse problem encountered during the generation of the IHC image block to be detected from the target HE image block, in one embodiment of the present invention, the image discriminator includes a local discriminator and a global discriminator, wherein...

[0117] The global discriminator is based on a discriminative architecture using spatial pyramid pooling and channel-space self-attention mechanisms, configured to determine the authenticity of image organization structures.

[0118] The local discriminator is based on a discriminant architecture using an edge-guided lightweight residual mechanism, which is configured to perform fine-grained discrimination of membrane-stained regions and cell nucleus edges.

[0119] In practice, the global discriminator and the local discriminator are independent of each other. Based on the discrimination structure of the global discriminator and the local discriminator, the global discriminator and the local discriminator can form significant differences in feature extraction mechanism and receptive field processing, forming a complementary mechanism of macroscopic topological verification and microscopic pathological feature discrimination.

[0120] Figure 6 An embodiment of the global discriminator is shown in the figure. Figure 6 The diagram shows that spatial pyramid pooling uses a four-level pyramid pooling layer, specifically including a global first convolutional layer, a global second convolutional layer, a global third convolutional layer, and a global average layer. The global first convolutional layer can use a 1×1 convolution, the global second convolutional layer can use a 3×3 convolution, and the global third convolutional layer can use a 6×6 convolution.

[0121] During model training, the pseudo-IHC image patches generated by the image generator are convolved using the global first convolutional layer, the global second convolutional layer, and the global third convolutional layer, respectively. At the same time, the global average difference is used for averaging. Then, the features after convolution and averaging are concatenated using a feature concatenation layer. The concatenated features are then gated and reweighted using a channel-spatial self-attention mechanism. Finally, the global discriminative output is obtained through a fully connected layer.

[0122] Specifically, global averaging, feature stitching, channel-spatial self-attention mechanism, and fully connected layer can adopt existing common forms. Spatial pyramid pooling (SPP) can be used to extract multi-scale tissue topological features in parallel from pseudo-IHC image patches. The channel-spatial self-attention mechanism can learn the deep-level dependencies of tissue structure.

[0123] Figure 7 The image illustrates one embodiment of a local discriminator that employs an edge-guided lightweight residual architecture, innovatively fusing traditional edge detection operators with deep feature extraction. Figure 7 In the local discriminator, there are three units: Sobel convolutional layer, Res Block1 unit, and Res Block2 unit. The Sobel convolutional layer extracts the edge prior information of the pseudo HER2 image patch. The Res Block1 and Res Block2 units constitute a two-level residual compression unit. The Res Block1 unit is a shallow residual module that preserves edge details through residual connections and initially fuses low-order features. The Res Block2 unit is a deep residual module that further compresses features and enhances semantic expressiveness. Its output is connected to a global max pooling layer to focus on key regions. Specifically, the local discriminator can avoid gradient degradation through the residual structure and improve local discrimination accuracy by utilizing edge guidance.

[0124] During model training, for the generated pseudo-IHC image patches, a trainable Sobel convolutional layer is used to actively enhance the high-frequency signal of the cell membrane boundary and strengthen the gradient response of the discontinuous region of the nuclear membrane. Subsequently, a two-level residual compression unit is used to reduce the number of parameters while preserving the key details of chromatin distribution through residual connections. Finally, a global max pooling layer is used to replace the fully connected layer, forcing the network to focus on the most discriminative local pathological features (such as nuclear cracks or abnormal chromatin aggregation areas). In addition, local discriminative outputs can also be obtained through the global max pooling layer.

[0125] To further suppress the mode collapse problem, the loss function of the generative model in this invention can be:

[0126]

[0127] in, To generate the model loss function, To reconstruct the loss weights, To rebuild the losses, To combat the losses, To counteract the loss of weight, In order to perceive loss, To perceive loss weights, To preserve the loss in the frequency domain structure, To preserve the loss weights in the frequency domain structure, The loss is a regularization term for sample feature diversity. The loss weights are the regularization term for sample feature diversity. To preserve constraint loss at the cell edge, Constraint loss weights are preserved at cell edges.

[0128] In practice, the loss weights are reconstructed. Counter-loss weights Perceived loss weights Frequency domain structure preservation loss weights Sample feature diversity regularization term loss weight and cell edge preservation constraint loss weights You can select and confirm as needed; further details will not be provided here.

[0129] Specifically, a reconstruction loss is added to the loss function of the generative model. At that time, it can be used to maintain the basic structure, for reconstruction losses Then we have:

[0130]

[0131] in, The pseudo-IHC image patch output by the generator. As a reference IHC image block, To represent the L1 norm, the following... All of these have the same meaning; please refer to the explanation here.

[0132] Add adversarial loss to the loss function of the generative model. This can improve the realism of the training IHC image patches generated by the image generator, particularly for adversarial loss. Then we have:

[0133]

[0134] in, For the local discriminant output, This represents the global discriminant output. The corresponding values ​​for the local and global discriminant outputs are typically [0,1]. It represents the mathematical expectation.

[0135] Calculate perceived loss In this invention, a VGG network can be utilized. Specifically, training IHC image blocks and baseline IHC image blocks are loaded into the VGG network respectively, after which perceptual loss can be performed. For perceived loss Then we have:

[0136]

[0137] in, This indicates that the feature extraction network trained using VGG is the first... Layer output feature map.

[0138] Add frequency domain structure preservation loss to the generative model loss function Constraining the high-frequency components of pseudo-IHC image patches and reference IHC image patches in the Fourier frequency domain preserves the sharpness and morphological consistency of the film staining edges, and improves the detail sharpness of the edges and staining areas; while maintaining the loss of frequency domain structure. Then we have:

[0139]

[0140] in, This refers to the two-dimensional Fourier transform (FFT) operation performed on the pseudo-IHC image patch. This is a two-dimensional Fourier transform operation performed on a reference IHC image block.

[0141] Add a sample feature diversity regularization term to the loss function of the generative model. To prevent generated images from becoming homogenized in the later stages of training, and thus to prevent the IHC staining image generation model from falling into pattern collapse during training by outputting a single image; this is achieved through a loss function that regularizes the loss term for sample feature diversity. Then we have:

[0142]

[0143] in, , To extract intermediate features from images generated by the encoder network within the same batch during the training phase, To calculate intermediate features of an image Image intermediate features The cosine similarity. Specifically, when extracting intermediate features from an image, the intermediate features should be those located at the output of the bottom-level encoder within the encoder network.

[0144] Add cell edge preservation constraint loss to the generative model loss function At this time, cell edge preservation constraints can be applied, and the ability to restore cell nucleus edges and membrane staining contours can be enhanced through image gradient comparison. This addresses the loss due to cell edge preservation constraints. Then we have:

[0145]

[0146] in, The edge map of the image can be obtained by gradient operators such as Sobel and Laplacian.

[0147] As can be seen from the above description, the image discriminator employs both local and global discriminators, and the loss function of the generation model adopts the above form, which can effectively prevent the common pattern collapse.

[0148] In one embodiment of the present invention, when creating the generative model training dataset, generated training samples are created within the generative model training dataset, wherein...

[0149] When creating training samples, the following steps are included:

[0150] Obtain source images of HE-stained sections and paired IHC-stained sections from the same tumor tissue section;

[0151] The source images of HE-stained and IHC-stained sections were registered in multiple stages to generate HE-stained and IHC-stained registration images.

[0152] When performing multi-stage registration, it should include at least the following sequential steps: full-map coarse registration, mesoscale regional non-rigid registration, and patch-level fine registration.

[0153] After performing full-image coarse registration, the source images of HE-stained sections and IHC-stained sections are initially aligned.

[0154] After performing non-rigid registration in the mesoscale region, local offsets of tissue structures are corrected.

[0155] After performing tile-level fine registration, IHC staining registration map and HE staining registration map are generated respectively, and the IHC staining registration map and HE staining registration map correspond to each other at the pixel scale in terms of cell boundaries, nuclear structures and membrane staining areas.

[0156] The HE staining registration map and the IHC staining registration map are segmented and screened to generate training HE image blocks and corresponding reference IHCN image blocks within the training samples. The training HE image blocks and the reference IHC staining image blocks are at least aligned at the tissue structure level.

[0157] As explained above, the generative model training dataset includes several generated training samples. These samples should be created using the same method. Generally, when creating a training sample, both HE-stained and IHC-stained slide source images should be obtained. These source images should originate from the same tumor tissue section. The method for obtaining the HE-stained and IHC-stained slide source images can be found in the description of the reference HE-stained images provided above, and will not be repeated here. It should be noted that the type of tumor tissue corresponding to the reference HE-stained image should be consistent with the type of tumor tissue corresponding to the original HE-stained and IHC-stained slide images, such as breast tumor tissue, gastric tumor tissue, or cervical tumor tissue. Figure 2 The image shows an example of HE-stained and IHC-stained sections originating from the breast. Figure 2 In the image, the left image is a source image of an HE-stained section, and the right image is a source image of an IHC-stained section. It should be understood that the source breast tissue should include tumor tissue.

[0158] To address the issue of insufficient spatial registration accuracy, this invention performs multi-stage registration on HE-stained and IHC-stained source images. This multi-stage registration enables precise spatial correspondence at both the macroscopic structural and cellular tissue scales. Specifically, multi-stage registration can include sequential full-image coarse registration, mesoscale region-level non-rigid registration, and patch-level fine registration. The methods and processes of full-image coarse registration, mesoscale region-level non-rigid registration, and patch-level fine registration are illustrated below.

[0159] When performing full-map coarse registration, one feasible approach is as follows:

[0160] Since the source images of HE-stained and IHC-stained sections have high resolution, they are first downsampled to a lower resolution level (e.g., Level 3) to reduce computational complexity and suppress high-frequency noise interference. Then, key feature points are extracted from the downsampled images. If a feature point detection algorithm (e.g., SIFT, SURF, or ORB) is preferred to extract the set of key points from the two images, then:

[0161]

[0162] in, This is the set of key points within the image after HE staining downsampling. Key points within the image after HE staining and downsampling. This is the set of key points within the image after IHC staining downsampling. These are key points within the image after IHC staining and downsampling. Set of key points The number of internal key points Set of key points The number of internal key points.

[0163] Key points Includes its corresponding image location , main direction ,scale and local descriptors (Taking SIFT as an example, SIFT extracts stable keypoint locations by detecting local extrema in the Gaussian pyramid and constructs a gradient histogram descriptor (usually a 128-dimensional vector) with consistent orientation for each keypoint, which is used for feature matching and geometric alignment between subsequent images. SIFT features are robust to image scaling, rotation, and certain degrees of brightness changes, and are suitable for stable detection of structural regions such as cell nucleus edges and gland contours in pathological images.)

[0164] For any key point By calculating its relationship with key points Local descriptor Euclidean distance: ,in, The Euclidean distance is the local descriptor.

[0165] Establish a preliminary set of matching point pairs ,in, This represents the matching threshold, used to filter keypoint pairs where the Euclidean distance between descriptors is less than this threshold. To eliminate the impact of false matches, the RANSAC algorithm is used to refine the set of matching point pairs. Perform interior point filtering: In each round of sampling, select from the set of matching points. The affine transformation matrix is ​​estimated by randomly selecting 3 to 4 pairs of matching points. For the first Key points and the Key points , so that:

[0166]

[0167] For the affine transformation matrix obtained above, calculate the geometric reprojection error and determine whether the set reprojection error is less than the threshold. When the geometric reprojection error is less than the threshold When the time is right, the matching point pair is determined to be an interior point; where, for geometric reprojection error, we have: .

[0168] In practice, the transformation matrix with the most interior points is selected as the coarse registration output. Then, the affine transformation matrix can be used to affinely transform the IHC stained slide source image to the HE stained slide source image so that the HE stained slide source image and the IHC stained slide source image are initially aligned. That is, the HE stained slide image is used as the alignment reference, so that the preliminary aligned HE stained image and the preliminary aligned IHC stained image can be formed.

[0169] When performing non-rigid registration at the mesoscale regional level, one feasible approach is as follows:

[0170] The preliminary aligned images of HE staining and IHC staining are divided into several medium-sized image window regions (e.g., 10000×10000 pixels). Local non-rigid registration is performed within each window to correct local offsets caused by tissue deformation, distortion, or differences in slice thickness. In practice, an elastic deformation model based on B-spline interpolation is preferred for local offset correction.

[0171] Specifically, a regular grid of control points is defined in each medium-sized image window area. The two-dimensional deformation vector for each control point is defined as follows: ,in, These represent the horizontal and vertical displacements of the control point, respectively. (Any pixel) The overall deformation vector is obtained by weighting the difference between the control point displacements using the B-spline basis functions, then: ,in, Represents pixels cubic B-spline basis functions in the x-direction, Represents pixels cubic B-spline basis functions in the y-direction, It is a pixel-level non-rigid deformation field.

[0172] It should be noted that defining grid control points At this time, it is necessary to divide the current medium-sized image window area into a uniform grid, with grid control points. That is, the vertex in the i-th row and j-th column of the grid is obtained; the horizontal direction mentioned above is generally the length direction of the grid obtained, and the vertical direction is generally the width direction of the grid obtained.

[0173] To solve for the optimal deformation parameters, the following mesoscale registration loss function is constructed to jointly minimize pixel registration error and deformation field smoothness:

[0174]

[0175] in, The loss function is for mesoscale registration. This represents the pixel values ​​of a medium-sized image window region in HE. This represents the pixel values ​​of a medium-sized image window region in HER2. This represents the second derivative of the deformation vector at the control point, used to constrain the smoothness of the deformation field. This is the regularization coefficient that balances the two terms.

[0176] It should be noted that, based on the preliminary HE staining alignment image, the position of the preliminary IHC staining alignment image is adjusted. When the mesoscale registration loss function is minimized, the non-rigid registration of the mesoscale region is completed, thereby correcting the local offset of the tissue structure. At this time, a mesoscale HE staining alignment image is formed based on the preliminary HE staining alignment image, and a mesoscale IHC staining alignment image is formed based on the adjusted preliminary IHC staining alignment image.

[0177] When performing detailed registration at the tile level, one feasible approach is as follows:

[0178] On the HE staining mesoscale registration image and the IHC staining mesoscale registration image, select the patch fine registration region (preferably the size of the patch fine registration region is 1024×1024 pixels), and then use the dense optical flow estimation algorithm based on multi-resolution pyramid to perform high-precision registration operation.

[0179] When performing high-precision registration, the optical flow target is to estimate the two-dimensional displacement vector field of each pixel, then: ,in, The horizontal displacement of the fine registration region of the patch on the IHC staining mesoscale registration image to the fine registration region of the patch on the HE staining mesoscale registration image. The vertical displacement of the fine-registration region of a patch on the IHC-stained mesoscale registered image to the fine-registration region of a patch on the HE-stained mesoscale registered image; furthermore, the registered position should satisfy the photometric consistency assumption, then: , To calculate the grayscale values ​​of pixels (x, y) within the fine registration region of the patch in the HE-stained mesoscale registration image, similarly... The corresponding gray values ​​within the fine registration area of ​​the patch in the IHC staining mesoscale registration image.

[0180] Construct a high-precision registration loss function. For the high-precision registration loss function, we have:

[0181]

[0182] in, The regularization coefficient is . , The gradient of the optical flow field is used to suppress local discontinuities or non-physical deformations.

[0183] It should be noted that, using the HE staining mesoscale registration image as a reference, the positional state of the IHC staining mesoscale registration image is adjusted. When the high-precision registration loss function is minimized, patch-level fine registration is completed, thus ensuring accurate correspondence between cell boundaries, nuclear structures, and membrane-stained regions at the pixel scale. Furthermore, after performing patch-level fine registration, the IHC staining registration image and the HE staining registration image are made to correspond at the pixel scale in terms of cell boundaries, nuclear structures, and membrane-stained regions. Figure 3 The figure shows an embodiment of IHC staining registration map and HE staining registration map, wherein the left figure is the HE staining registration map and the right figure is the IHC staining registration map.

[0184] In one embodiment of the present invention, the segmentation and screening process for HE staining registration maps and IHC staining registration maps includes:

[0185] HE training candidate regions are selected sequentially on the HE staining registration map, and tumor region identification is performed on the HE training candidate regions. After tumor region identification, the binary mask of each pixel in the HE training candidate region is determined. When the binary mask is 1, it indicates that the current pixel belongs to the tumor region. When the binary mask is 0, it indicates that the current pixel does not belong to the tumor region.

[0186] Based on the binary mask of each pixel in the HE training candidate region, the proportion of the tumor region is calculated. When the calculated proportion of the tumor region matches the region selection threshold, the current HE training candidate region is configured as the training HE image patch.

[0187] Based on the positional state of the training HE image blocks, the corresponding baseline IHC image blocks are segmented on the IHC staining registration map.

[0188] In practice, a fixed-step sliding window strategy can be used to sequentially select HE training candidate regions on the HE staining registration map. To ensure the independence of training and test samples, overlap of HE training candidate regions should be avoided during selection. Preferably, the size of the HE training candidate region is 1024×1024 pixels, and the sliding step size is 1024 pixels.

[0189] To eliminate areas without tumor tissue or background and improve the relevance and effectiveness of training data, this invention employs a patch selection mechanism based on cancer region masks. Specifically, the pathological image data used are all derived from self-collected tumor tissue slice samples, and qualified pathologists perform pixel-level fine delineation of the tumor regions to form a cancer region mask map for modeling. Based on this, a tumor region recognition model based on U-Net or other semantic segmentation architecture is trained to automatically predict potential cancer regions at the full-image scale. The details of the tumor region recognition model and the training methods and processes for it are consistent with existing technologies and will not be elaborated here.

[0190] For HE training candidate regions, the HE training candidate regions are loaded into the tumor region recognition model, so that the tumor region recognition model can output a binary mask for each pixel in each HE training candidate region. When the binary mask is 1, it indicates that the current pixel belongs to the tumor region. When the binary mask is 0, it indicates that the current pixel does not belong to the tumor region. The non-tumor region can be non-cancer tissue or background region.

[0191] Based on the binary mask of each pixel within the candidate region trained using HE, the proportion of the tumor region is calculated, and then:

[0192]

[0193] in, The percentage of tumor area. This represents the area of ​​the current HE training candidate region. For example, if the size of the HE training candidate region is 1024×1024 pixels, then... It should be 1024×1024.

[0194] After calculating the tumor region proportion of the current HE training candidate region, this proportion should be compared with a region selection threshold. If the tumor region proportion is greater than the threshold, it is considered a match. In this case, the current HE training candidate region is configured as a training HE image patch. Since the IHC staining registration map and the HE staining registration map are highly spatially registered, a corresponding baseline IHC image patch can be formed on the IHC staining registration map based on the positional state of the training HE image patch. This results in a training HE image patch and a baseline IHC image patch within the generated training samples. Figure 4 The output shows an embodiment of generating training HE image patches and benchmark IHC image patches within the training samples. In the figure, the left side shows the training HE image patch, and the right side shows the benchmark IHC image patch.

[0195] In practice, the region selection threshold can be 0.25. Of course, other values ​​can also be selected, depending on the needs. It should be noted that when performing segmentation and filtering on the reference HE image, the above segmentation and filtering instructions can be used as a reference. The difference is that during segmentation and filtering, only the corresponding target HE image block needs to be obtained.

[0196] It should be understood that, based on the above-described multi-stage registration and segmentation / screening process, the following can be obtained: the training HE image block and the reference IHC image block within the same generated training sample should originate from the same tumor tissue slice, and the training HE image block and the reference IHC image block should be at least aligned at the tissue structure level. Furthermore, both the training HE image block and the reference IHC image block contain tumor tissue information. Further, when the generated training sample is prepared through the above-described multi-stage registration and segmentation / screening process, and the reference HE-stained image is segmented and screened, the consistency requirements of the generation and classification tasks of this invention can be met.

[0197] To perform the aforementioned hierarchical expression processing, in one embodiment of the present invention, the multimodal semantic classification model includes at least a visual encoder module, a language encoder module, and a graph-text alignment inference module, wherein...

[0198] Visual embedding features of the IHC image patch to be inspected are extracted using a visual encoder module.

[0199] A reference semantic coding feature group is generated using the language encoder module. When generating the reference semantic coding feature group, the HER2 level bootstrap information group is loaded into the language encoder module so that the language encoder module can perform semantic coding on each HER2 level bootstrap information group and generate the corresponding HER2 level semantic coding feature after language coding. The HER2 level semantic coding features in the reference semantic coding feature group are independent of each other.

[0200] The image-text alignment inference module is used to calculate the feature similarity between the visual embedded features and the semantic coding features of each HER2 level. Based on the calculated feature similarity, the target HER2 level information is determined, and the expression level prediction information of the current IHC image block to be inspected is output.

[0201] Figure 8The diagram illustrates an embodiment of a multimodal semantic classification model. As shown, the model may include at least a visual encoder module, a language encoder module, and an image-text alignment inference module. The visual encoder module can employ structures such as ViT (Vision Transformer) or ResNet-101, specifically designed to extract visual embedding features from the IHC image patches to be detected. The language encoder module generates a reference semantic encoded feature set. For a given multimodal semantic classification model, the reference semantic encoded feature set generated by the language encoder module remains unchanged during inference; that is, the HER2-level semantic encoded features within the reference semantic encoded feature set remain stable. Of course, the reference semantic encoded feature set generated by the language encoder module may differ for different multimodal semantic classification models, and can be selected as needed. The method and process of generating the reference semantic encoded feature set by the language encoder module can be found in the following description.

[0202] In practice, the feature similarity between the visual embedding features and each HER2 level semantic coding feature can be calculated through the image-text alignment inference module. The target HER2 level information is then determined based on the calculated feature similarity. Subsequently, the expression level prediction information of the current IHC image block to be inspected is output. Specifically, the method and process of calculating feature similarity and determining the target HER2 level information can be referred to the above description, which will not be repeated here.

[0203] In one embodiment of the present invention, the multimodal semantic classification model further includes a category prototype matching module, wherein,

[0204] When the feature similarity calculated by the image-text alignment reasoning module is in a fuzzy approximation state, the category prototype matching module is configured to perform prototype similarity calculation processing to determine the target HER2 level information through prototype similarity calculation processing.

[0205] When performing prototype similarity calculation, the prototype similarity between the visual embedded features and the category prototype features within the category prototype matching module is calculated separately, and the optimal prototype similarity and the target category prototype feature corresponding to the optimal prototype similarity are determined.

[0206] Target HER2 level information is generated based on the target category prototype features.

[0207] The target HER2 level information is configured as the expression level prediction information of the current IHC image block to be inspected, and the category prototype matching module is configured to output the expression level prediction information of the current IHC image block to be inspected.

[0208] As explained above, after calculating the feature similarity between the visual embedded features and each HER2-level semantic coding feature, the image-text alignment inference module needs to select and determine the largest feature similarity, and configure the HER2-level semantic coding feature corresponding to the largest feature similarity as the optimal HER2-level semantic coding feature. It is understandable that the difference between the selected largest feature similarity and the second-best feature similarity may be small. For example, if the difference between the largest and second-best feature similarity is not greater than the switching threshold, the feature similarity calculated by the image-text alignment inference module is in a fuzzy approximation state. The second-best feature similarity specifically refers to a feature similarity less than the largest feature similarity, but greater than other feature similarities. The size of the switching threshold can be selected as needed, and will not be elaborated here.

[0209] When the target HER2 level information is determined using the above method in a fuzzy approximation state, the accuracy of the output expression level prediction information will be low. To further improve the accuracy of the generated expression level prediction information, this invention also includes a category prototype matching module within the multimodal semantic classification model, such as... Figure 8 As shown, in specific implementation, the category prototype matching module will only be started to perform prototype similarity calculation when the feature similarity calculated by the image-text alignment reasoning module is in a fuzzy approximation state. That is, during reasoning, it is preferable to use the image-text alignment reasoning module to perform the above-mentioned expression hierarchical processing.

[0210] In practice, the category prototype matching module stores several category prototype features. When performing prototype similarity calculation, the prototype similarity between the visual embedding feature and each category prototype feature is calculated. After calculating the prototype similarity between the visual embedding feature and all category prototype features, the optimal prototype similarity and the target category prototype feature corresponding to the optimal prototype similarity can be determined. The optimal prototype similarity is the maximum value of the calculated prototype similarity. After determining the optimal prototype similarity, the corresponding target category prototype feature can be determined.

[0211] It should be noted that when calculating the prototype similarity between visual embedded features and category prototype features, the above-described form of feature similarity calculation can be used. Specifically, when using the above-described method for calculating feature similarity, the following applies:

[0212]

[0213] in, For visual embedding features and categories Prototype similarity of category prototype features For category Category prototype features, Visual embedding features and category prototype features Dot product of vectors between them Visual embedding features and category prototype features Modulo multiplication of vectors between them.

[0214] Specifically, after obtaining the target category prototype features, target HER2 level information can be generated based on the target category prototype features. Subsequently, the target HER2 level information is configured as the expression level prediction information of the current IHC image block to be inspected, and the category prototype matching module is configured to output the expression level prediction information of the current IHC image block to be inspected. That is, after obtaining the target category prototype features and the corresponding expression level prediction information through the category prototype matching module, the category prototype matching module outputs the level prediction information.

[0215] In one embodiment of the present invention, constructing a multimodal semantic classification model includes:

[0216] Select a pre-trained framework model and configure it as the base model for multimodal semantic classification. The base model for multimodal semantic classification includes a visual encoder module, a language encoder module, a cue enhancement module, a graph-text alignment reasoning module, and a basic module for category prototype matching.

[0217] A classification model training dataset is constructed to train a multimodal semantic classification base model. The classification model training dataset includes several classification training samples. Each classification training sample includes a training IHC image patch and training expression level text information corresponding to the current training IHC image patch.

[0218] Configure the training conditions for the multimodal semantic classification base model until it reaches the target state. Then, generate a multimodal semantic classification model based on the trained base model that has reached the target state.

[0219] During model training, the corresponding weight parameters of the visual encoder module and the language encoder module are frozen, the cue enhancement module fine-tunes the corresponding weight parameters through the backpropagation mechanism, and the category prototype matching basic module updates the corresponding category prototype features in the training of each batch of classification training samples through the momentum update mechanism.

[0220] During model training, the prompting enhancement module concatenates each HER2 level description text in the HER2 level description text group into a learnable training vector to generate HER2 level training source information corresponding to each HER2 level description text. Subsequently, the language encoder module performs semantic encoding on the HER2 level training source information to generate HER2 level training encoded features.

[0221] The visual encoder module is used to extract the training visual features of the training IHC image patch within each classification training sample. Then, the image-text alignment inference module is configured to calculate the feature similarity between the training visual features and all HER2 level training encoded features. Finally, the category prototype matching module is used to calculate and update the category prototype features corresponding to each training expression level information.

[0222] It should be understood that for the above-mentioned multimodal semantic classification model, a multimodal semantic classification base model and a classification model training dataset should generally be constructed first. Then, the multimodal semantic classification base model is trained using the classification model training dataset. After the multimodal semantic classification base model reaches the target state of the classification model, the multimodal semantic classification model is generated based on the multimodal semantic classification base model that has reached the target state of the classification model.

[0223] In one embodiment of the present invention, a pre-trained framework model can be selected and used as the base model for multimodal semantic classification. Specifically, the pre-trained framework model refers to a model that has undergone pre-training. This pre-trained framework model can be a MUSK (Multimodal transformer with Unified masked modeling) model. The MUSK model has achieved cross-modal alignment of pathological images and text descriptions through contrastive learning on a large-scale medical image dataset (50 million pathological images from 11,577 patients and 1 billion pathology-related text tags). Based on the above description of the MUSK model, the visual encoder module within the multimodal semantic classification base model can possess visual embedding feature extraction capabilities. For example, after loading training IHC image patches from the classification training samples into the visual encoder module, the visual encoder module can extract the training visual features of the training IHC image patches. The training visual features can be referenced to the aforementioned visual embedding features, and will not be elaborated further here.

[0224] Furthermore, the language encoder module within the multimodal semantic classification base model possesses semantic encoding capabilities. Therefore, during model training of the multimodal semantic classification base model, the corresponding weight parameters of the visual encoder module and the language encoder module can be frozen. That is, during the model training process, the corresponding weight parameters of the visual encoder module and the language encoder module remain unchanged; only the weight parameters of the cue enhancement module need to be fine-tuned. The category prototype matching module does not directly participate in backpropagation, but in the training of each batch of classification training samples, the corresponding category prototype features within the category prototype matching base module are updated through a momentum update mechanism. As explained above, the image-text alignment inference module mainly calculates and determines the target HER2 level information that best approximates the visual embedding features; therefore, the image-text alignment inference module does not have any weight parameters that need to be trained.

[0225] Specifically, the classification model training dataset should include several classification training samples. Each classification training sample includes a training IHC image patch and training expression level text information corresponding to the current training IHC image patch. The training IHC image patch can be a pseudo IHC image patch generated by the IHC image generation model and / or a benchmark IHC image patch obtained through segmentation and filtering. Preferably, the classification model training dataset can simultaneously include training IHC image patches using pseudo IHC image patches and training IHC image patches using benchmark IHC image patches.

[0226] For each classification training sample, the training expression level text information is mainly used to describe the HER2 expression level of the training IHC image patch. This training expression level text information can be a pre-set fixed template text used to guide the alignment of the semantic information between the training visual features and the training expression level text information. The text description of the training expression level text information may include: Level 1) "0: No obvious membrane staining or only nuclear staining"; Level 2) "1+: Incomplete membrane staining, weakly positive, discontinuous staining"; Level 3) "2+: Moderate membrane staining, more than 10% of cells show incomplete or partially strong positive membrane staining"; Level 4) "3+: More than 10% of cells show complete, strong positive membrane staining, clear boundaries, continuous staining". Figure 8 The texts at levels 1 to 4 shown represent the training expression level text information of the four categories. The texts at levels 1 to 4 can be the text descriptions corresponding to 1) to 4) here. It can be seen that the training expression level text information mainly describes the HER2 expression level of the training IHC image patch as one of the categories at levels 1) to 4) above.

[0227] It should be noted that the aforementioned training expression level text information can be used as standard input to the language encoding module. Subsequently, within the multimodal semantic classification base model, image-text feature matching is performed with IHC image patches to achieve expression level prediction training. Preferably, the training expression level text information can be flexibly expanded according to clinical classification standards, but consistency should be maintained during model training and inference to ensure alignment stability.

[0228] To enhance the language encoder module's ability to perceive task semantics, this invention incorporates a cue enhancement module within the multimodal semantic classification base model. This module appends a set of learnable training vectors before the HER2 expression level for each class, generating HER2 level training source information corresponding to the descriptive text for each HER2 level. Figure 8As explained above, when there are texts from level 1 to level 4, the prompt enhancement module randomly generates four learnable training vectors, and concatenates these four learnable training vectors before the texts from level 1 to level 4 to form four corresponding HER2 level training source information. Figure 8 In this process, four learnable training vectors can form a sequence of promptable learning vectors.

[0229] It should be noted that the dimension of the learnable training vectors generated by the cue enhancement module should be consistent with the original word vectors of the grade description text. The weight parameters of the cue enhancement module are optimized during the training phase to guide the multimodal semantic classification base model to understand the task context.

[0230] During model training, for each classification training sample, the HER2-level training source information is loaded into the language encoder module. The language encoder then performs semantic encoding on the current HER2-level training source information, generating HER2-level training encoded features. Therefore, during model training, when four HER2-level training source information exist, semantic encoding of all HER2-level training source information can generate four corresponding HER2-level training encoded features. Of course, the training IHC image patches within each classification training sample should be loaded into the visual encoder module so that the corresponding training visual features can be extracted.

[0231] It is understandable that after training the multimodal semantic classification base model to the target state of the classification model, four corresponding HER2-level semantic coding features can be formed based on the four HER2-level training coding features mentioned above. That is, during inference, the reference semantic coding feature group includes four independent HER2-level semantic coding features. When the training expression level text information is other, the HER2-level semantic coding features can be determined, and the reference semantic coding feature group includes a corresponding number of HER2-level semantic coding features. Examples will not be given here.

[0232] During model training, classification training samples are typically fed into the multimodal semantic classification base model in batches. For each classification training sample within the same batch, the visual encoder module is used to extract the training visual features of the training IHC image patches within each classification training sample. Subsequently, the image-text alignment inference module is configured to calculate the feature similarity between the training visual features and all HER2-level training encoded features. The method for calculating the feature similarity between the training visual features and the HER2-level training encoded features can be found in the above description of feature similarity calculation, which will not be repeated here.

[0233] For each batch of classification training samples, when calculating the prototype features of each training expression level information category, we have:

[0234]

[0235] in, Classify training samples within each batch The number of training samples for classification. Within each annotation classification training sample Training visual features of each classification training sample Within each batch of training samples for classification The training representation level text information of each classification training sample, For category The category prototype features corresponding to the training samples;

[0236] For each category Update category prototype features through momentum update mechanism Then:

[0237]

[0238] in: To update the momentum coefficient.

[0239] In practice, the momentum coefficient is updated. The size can be selected as needed, such as allowing for updates to the momentum coefficient. It can be 0.1. During the above update, " Specifically, this refers to assigning the calculation result on the right to the coordinates, thus realizing the category prototype feature. Updates. Categories That is, one of the levels mentioned above (1) to (4).

[0240] It should be noted that when the text description of the training expression level text information is described as level 1) to level 4 above, the prototype similarity between the visual embedded features and the category prototype features is calculated by the category prototype matching module, and the target HER2 level information is determined. This can significantly improve the discrimination ability between boundary levels such as 1+ and 2+, 2+ and 3+, thereby further improving the accuracy of expression level prediction information.

[0241] It should be understood that when training a multimodal semantic classification base model, classification model training conditions should be configured. Generally, the configured classification model training conditions should at least include a classification training loss function. Of course, classification model training conditions can also include other necessary conditions, such as setting an AdamW optimizer with an initial learning rate of 3e-5, coupled with linear warm-up and cosine annealing strategies, and using dropout of 0.1 and weight decay of 0.01 to prevent overfitting. During training, the classification accuracy, recall, and F1 score should be monitored on the validation set. When the metrics do not improve for 5 consecutive epochs, an early stopping mechanism should be triggered. At this point, the model training of the multimodal semantic classification base model can be considered to have reached the target state of the classification model.

[0242] Specifically, when training the multimodal semantic classification base model, a contrastive learning-based image-text alignment loss function is introduced to improve the semantic consistency between the trained visual features and the HER2-level trained encoded features. That is, the classification training loss function can adopt the image-text alignment loss function. In one embodiment of the present invention, when the classification training loss function adopts the image-text alignment loss function, the classification training loss function adopts a CLIP-style bidirectional contrastive form, simultaneously maximizing the feature similarity between each pair of trained IHC image patches and their corresponding trained expression level text information, and minimizing the feature similarity with other trained expression level text information. For the classification training loss function, the following applies:

[0243]

[0244] In the formula, For the number of categories, To train visual features, Features are encoded for HER2 level training. Encoding features for HER2 level training With training visual features The cosine similarity function between them This refers to temperature hyperparameters.

[0245] In the embodiments described above, the number of categories The value is 4. The temperature hyperparameter τ is used to scale the distribution of image-text similarity scores. Generally, the smaller the temperature hyperparameter τ is (e.g., 0.01), the sharper the distribution, and the more the model focuses on high-similarity samples, but it is sensitive to noise. The larger the temperature hyperparameter τ is (e.g., 0.2), the smoother the distribution, and the more tolerant the model is to low-similarity samples, enhancing generalization. It can be set to 0.07 based on experience.

[0246] As can be seen from the above classification training loss function, the classification training loss function can enable the multimodal semantic classification base model to learn the one-to-one correspondence between the training visual features and the HER2 level training encoded features, and significantly improve the alignment ability of HER2 level semantic expression.

[0247] In specific implementation, once the multimodal semantic classification base model is trained to the target state of the classification model, a multimodal semantic classification model can be generated based on the multimodal semantic classification base model that has reached the target state of the classification model. As can be seen from the above description, when generating the multimodal semantic classification model, the prompt enhancement module is in a non-working state, and the reference semantic coding feature group generated by the language encoder module remains fixed. That is, during the reasoning process, the HER2 level semantic coding features in the reference semantic coding feature group will not change.

[0248] Furthermore, a category prototype matching module can be generated from the basic category prototype matching module. This module stores corresponding category prototype features for subsequent prototype similarity calculations. These category prototype features essentially represent the central representation of each category in the feature space, and their mechanism of action is as follows:

[0249] During training, the sample features of each category are continuously aggregated. The feature center points of each category are calculated and saved through the dynamic update mechanism mentioned above, and finally a set of category prototype features corresponding to the number of categories is formed (e.g., four categories correspond to four prototype vectors).

[0250] As can be seen from the above description, the prompt enhancement module and the category prototype matching basic module are integrated into the multimodal semantic classification base model as lightweight additional modules. They have good task transfer and semantic adaptation capabilities, avoid damaging the backbone model, improve the performance of HER2 level classification, and are particularly suitable for tasks with fuzzy category boundaries.

[0251] It should be noted that after deploying the multimodal semantic classification model, the above-described method can be used to first utilize the image-text alignment inference module to calculate the feature similarity between the visual embedded features and each HER2 level semantic encoding feature, and then the image-text alignment inference module outputs the expression level prediction information of the IHC image patch to be inspected. However, when the feature similarity calculated by the image-text alignment inference module is in a fuzzy approximation state, the processor deploying the multimodal semantic classification model is configured with a category prototype matching module to perform prototype similarity calculation processing to determine the target HER2 level information.

[0252] Specifically, the processor deploying the multimodal semantic classification model can also obtain the feature similarity between the visual embedded features calculated by the image-text alignment inference module and the semantic encoding features of each HER2 level. Afterwards, it can be determined whether the calculated feature similarity is in a fuzzy approximation state. Therefore, the operation and output states of the image-text alignment inference module and the category prototype matching module can be controlled by the processor deploying the multimodal semantic classification model. Alternatively, the image-text alignment inference module can simultaneously output all calculated feature similarities. Then, the processor determines the maximum feature similarity and whether the calculated feature similarity is in a fuzzy approximation state. The operation and output states of the image-text alignment inference module and the category prototype matching module can be selected according to actual needs, which will not be elaborated here. It should be understood that when the feature similarity is determined to be in a fuzzy approximation state, the target HER2 level information determined by the image-text alignment inference module should be ignored, and the target HER2 level information determined by the category prototype matching module should be used instead.

Claims

1. A method for grading tumor HER2 expression based on HE staining images, characterized in that, The tumor HER2 expression grading method includes: A reference HE staining image is provided, and the reference HE staining image is at least segmented and screened to generate a target HE image block group after segmentation and screening, wherein the target HE image blocks in the target HE image block group all contain tumor tissue information; Each target HE image block in the target HE image block group is loaded into the IHC staining image generation model to generate an IHC image block to be examined that is structurally aligned with the current target HE image block. Each IHC image patch to be inspected is loaded into a multimodal semantic classification model for expression grading. After grading, HER2 expression level prediction information for the current IHC image patch to be inspected is generated. When performing expression grading processing, the multimodal semantic classification model extracts the visual embedding features of the IHC image patch to be inspected, and calculates and determines the target HER2 grade information that is optimally approximated with the visual embedding features; The target HER2 level information is configured as the expression level prediction information for the current IHC image block to be examined; The IHC staining image generation model includes at least an image generator. The target HE image block is loaded into the IHC staining image generator, which then generates the IHC image block to be examined. The IHC staining image generation model is generated based on the IHC staining image generation base model after model training. The IHC staining image generation base model includes an image generator and an image discriminator. The IHC staining image generation base model is trained using the generated model training dataset. The generative model training dataset includes several generated training samples. Each generated training sample includes a training HE image patch and a corresponding baseline IHC image patch. The training HE image patch and the baseline IHC image patch within the same generated training sample are formed based on the same tumor tissue slice. At the same time, both the training HE image patch and the baseline IHC image patch contain tumor tissue information, and the training HE image patch and the baseline IHC image patch within the same training sample are at least aligned at the tissue structure level. Configure the basic model for IHC staining image generation and train the required generative model until the training effect of the basic model for IHC staining image generation reaches the target state of the generative model. After that, generate the IHC staining image generation model based on the basic model for IHC staining image generation that has reached the target state of the generative model. The image discriminator includes a local discriminator and a global discriminator, wherein, The global discriminator is based on a discriminative architecture using spatial pyramid pooling and channel-space self-attention mechanisms, configured to determine the authenticity of image organization structures. The local discriminator is based on a discriminant architecture with an edge-guided lightweight residual mechanism, which can be configured to perform fine-grained discrimination of membrane-stained regions and cell nucleus edges. The local discriminator includes a Sobel convolutional layer, a Res Block1 unit, and a Res Block2 unit. The Sobel convolutional layer extracts the edge prior information of the pseudo-HER2 image patch. The Res Block1 and Res Block2 units constitute a two-level residual compression unit. The Res Block1 unit is a shallow residual module that preserves edge details through residual connections and initially fuses low-order features. The Res Block2 unit is a deep residual module that further compresses features and enhances semantic expressiveness. Its output is connected to a global max pooling layer to focus on key regions.

2. The tumor HER2 expression grading method based on HE staining images according to claim 1, characterized in that, When calculating the target HER2 level information that best approximates the visual embedding features, the following steps are included: Calculate the feature similarity between the visual embedded features and each HER2-level semantic encoded feature within the reference semantic encoded feature group; For any visual embedding feature, when the calculated feature similarity is at its maximum, the visual embedding feature and the corresponding HER2 level semantic coding feature are in the optimal approximation state. At this time, the corresponding HER2 level semantic coding feature is configured as the optimal HER2 level semantic coding feature. Configure the HER2 level information in the optimal HER2 level semantic coding features as the target HER2 level information.

3. The tumor HER2 expression grading method based on HE staining images according to claim 2, characterized in that, The multimodal semantic classification model includes at least a visual encoder module, a language encoder module, and a graph-text alignment inference module, wherein... Visual embedding features of the IHC image patch to be inspected are extracted using a visual encoder module. A reference semantic coding feature group is generated using the language encoder module. When generating the reference semantic coding feature group, the HER2 level bootstrap information group is loaded into the language encoder module so that the language encoder module can perform semantic coding on each HER2 level bootstrap information group and generate the corresponding HER2 level semantic coding feature after language coding. The HER2 level semantic coding features in the reference semantic coding feature group are independent of each other. The image-text alignment inference module is used to calculate the feature similarity between the visual embedded features and the semantic coding features of each HER2 level. Based on the calculated feature similarity, the target HER2 level information is determined, and the expression level prediction information of the current IHC image block to be inspected is output.

4. The tumor HER2 expression grading method based on HE staining images according to claim 3, characterized in that, The multimodal semantic classification model also includes a category prototype matching module, wherein, When the feature similarity calculated by the image-text alignment reasoning module is in a fuzzy approximation state, the category prototype matching module is configured to perform prototype similarity calculation processing to determine the target HER2 level information through prototype similarity calculation processing. When performing prototype similarity calculation, the prototype similarity between the visual embedded features and the category prototype features within the category prototype matching module is calculated separately, and the optimal prototype similarity and the target category prototype feature corresponding to the optimal prototype similarity are determined. Target HER2 level information is generated based on the target category prototype features. The target HER2 level information is configured as the expression level prediction information of the current IHC image block to be inspected, and the category prototype matching module is configured to output the expression level prediction information of the current IHC image block to be inspected.

5. The tumor HER2 expression grading method based on HE staining images according to claim 3, characterized in that, The feature similarity calculated by the image-text alignment reasoning module includes cosine similarity. in, When cosine similarity is used for feature similarity, then: in, For feature similarity, For visual embedding features, For category HER2 level semantic encoding features For visual embedding features and categories HER2 level semantic encoding features vector dot product, For visual embedding features and categories HER2 level semantic encoding features Modulo multiplication of vectors.

6. The tumor HER2 expression grading method based on HE staining images according to claim 4, characterized in that, When constructing a multimodal semantic classification model, the following are included: Select a pre-trained framework model and configure it as the base model for multimodal semantic classification. The base model for multimodal semantic classification includes a visual encoder module, a language encoder module, a cue enhancement module, a graph-text alignment reasoning module, and a basic module for category prototype matching. A classification model training dataset is constructed to train a multimodal semantic classification base model. The classification model training dataset includes several classification training samples. Each classification training sample includes a training IHC image patch and training expression level text information corresponding to the current training IHC image patch. Configure the training conditions for the multimodal semantic classification base model until it reaches the target state. Then, generate a multimodal semantic classification model based on the trained base model that has reached the target state. During model training, the corresponding weight parameters of the visual encoder module and the language encoder module are frozen, the cue enhancement module fine-tunes the corresponding weight parameters through the backpropagation mechanism, and the category prototype matching basic module updates the corresponding category prototype features in the training of each batch of classification training samples through the momentum update mechanism. During model training, the prompting enhancement module concatenates each HER2 level description text in the HER2 level description text group into a learnable training vector to generate HER2 level training source information corresponding to each HER2 level description text. Subsequently, the language encoder module performs semantic encoding on the HER2 level training source information to generate HER2 level training encoded features. The visual encoder module is used to extract the training visual features of the training IHC image patch within each classification training sample. Then, the image-text alignment inference module is configured to calculate the feature similarity between the training visual features and all HER2 level training encoded features. Finally, the category prototype matching module is used to calculate and update the category prototype features corresponding to each training expression level information.

7. The tumor HER2 expression grading method based on HE staining images according to claim 6, characterized in that, When calculating the prototype features of each training representation level information category, we have: in, Classify training samples within each batch The number of training samples for classification. Within each annotation classification training sample Training visual features of each classification training sample Within each batch of training samples for classification The training representation level text information of each classification training sample, For category The category prototype features corresponding to the training samples; For each category Update category prototype features through momentum update mechanism Then: in: To update the momentum coefficient.

8. The tumor HER2 expression grading method based on HE staining images according to claim 1, characterized in that, When creating the training dataset for the generative model, generate training samples within the training dataset for the generative model, where, When creating training samples, the following steps are included: Obtain source images of HE-stained sections and paired IHC-stained sections from the same tumor tissue section; The source images of HE-stained and IHC-stained sections were registered in multiple stages to generate HE-stained and IHC-stained registration images. When performing multi-stage registration, it should include at least the following sequential steps: full-map coarse registration, mesoscale regional non-rigid registration, and patch-level fine registration. After performing full-image coarse registration, the source images of HE-stained sections and IHC-stained sections are initially aligned. After performing non-rigid registration in the mesoscale region, local offsets of tissue structures are corrected. After performing tile-level fine registration, IHC staining registration map and HE staining registration map are generated respectively, and the IHC staining registration map and HE staining registration map correspond to each other at the pixel scale in terms of cell boundaries, nuclear structures and membrane staining areas. HE staining registration maps and IHC staining registration maps are segmented and screened to generate training HE image blocks and corresponding baseline IHC image blocks within the training samples. The training HE image blocks and baseline IHC image blocks are at least aligned at the tissue structure level.

9. The tumor HER2 expression grading method based on HE staining images according to claim 8, characterized in that, When segmenting and screening HE staining registration maps and IHC staining registration maps, the following steps are included: HE training candidate regions are selected sequentially on the HE staining registration map, and tumor region identification is performed on the HE training candidate regions. After tumor region identification, the binary mask of each pixel in the HE training candidate region is determined. When the binary mask is 1, it indicates that the current pixel belongs to the tumor region. When the binary mask is 0, it indicates that the current pixel does not belong to the tumor region. Based on the binary mask of each pixel in the HE training candidate region, the proportion of the tumor region is calculated. When the calculated proportion of the tumor region matches the region selection threshold, the current HE training candidate region is configured as the training HE image patch. Based on the positional state of the training HE image blocks, the corresponding baseline IHC image blocks are segmented on the IHC staining registration map.

Images