Ultrasound-guided cross-modality deep learning breast cancer molecular subtyping method, device and equipment

Abstract

This invention discloses a method, apparatus, and device for molecular subtyping of breast cancer based on ultrasound-guided cross-modal deep learning. The method includes: acquiring preoperative ultrasound images of breast tumors and corresponding postoperative immunohistochemically stained whole-section images; extracting ultrasound features from the ultrasound images and multi-scale pathological features from the whole-section images; constructing a cross-modal learning model including ultrasound and pathological branches; during the training phase, guided representation learning of ultrasound features is performed using multi-scale pathological features, including cross-modal attention interaction, knowledge distillation, contrastive learning, and bond consistency constraints; during the inference phase, only the preoperative ultrasound image is input, and the molecular subtyping prediction result is output by the ultrasound branch. This invention achieves high-precision molecular subtyping prediction based solely on ultrasound images by guiding ultrasound feature learning through multi-scale pathological features, providing a non-invasive and reliable computer-aided diagnostic tool for clinical practice.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence and computer-aided diagnostic technology, specifically to a method, apparatus, and device for molecular subtyping of breast cancer based on ultrasound-guided cross-modal deep learning. Background Technology

[0002] Currently, accurate molecular subtyping of breast cancer is crucial for developing personalized treatment plans. Traditional molecular subtyping mainly relies on biopsy or immunohistochemistry (IHC) staining analysis of postoperative pathological tissue. However, due to the widespread heterogeneity of tumors and the limited sampling range of biopsies, biopsies often fail to comprehensively reflect the molecular characteristics of the entire tumor, leading to inconsistencies between biopsy results and postoperative immunohistochemical results, posing challenges to clinical diagnosis and treatment decisions. While postoperative pathological immunohistochemical analysis can provide accurate molecular subtyping, it can only be performed after surgical resection of the tumor and cannot be used for preoperative treatment planning (such as neoadjuvant chemotherapy, targeted therapy, etc.), potentially leading to treatment delays or missing the optimal time for preoperative intervention. Furthermore, surgery itself is invasive and may cause complications such as bleeding, infection, and poor healing.

[0003] Ultrasound (US), as a non-invasive, real-time, and widely used imaging technique, plays a crucial role in breast cancer screening and diagnosis. Previous studies have shown that specific grayscale ultrasound features (such as tumor margins, morphology, and echogenicity) may be associated with specific molecular subtypes. Although current technologies attempt to predict molecular subtypes using ultrasound image features, the limited accuracy of existing prediction models based on a single ultrasound modality is due to the fact that ultrasound images primarily reflect the anatomical structure and echo characteristics of tissues, lacking microscopic molecular biological information. Furthermore, the subjective nature of image interpretation and its high dependence on operator experience make it difficult to meet the clinical demand for high-precision molecular subtype prediction. In addition, existing multimodal deep learning methods typically still require pathological image data during the inference phase, preventing truly non-invasive preoperative molecular subtype prediction. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device and equipment for molecular subtyping of breast cancer based on ultrasound-guided cross-modal deep learning, which uses multi-scale pathological features to guide ultrasound features for representation learning during the model training stage, so as to achieve high-precision molecular subtyping prediction based solely on preoperative ultrasound images during the inference stage.

[0005] To achieve the above objectives, the present invention employs the following technical means:

[0006] The first aspect of this invention proposes an ultrasound-guided cross-modal deep learning-based molecular subtyping method for breast cancer, comprising the following steps:

[0007] Obtain preoperative ultrasound images of breast tumors and postoperative immunohistochemical stained whole slide images corresponding to the tumors;

[0008] Ultrasound features from the ultrasound images and multi-scale pathological features from the whole slice images are extracted. The multi-scale pathological features include block-level pathological features and region-level pathological features.

[0009] A cross-modal learning model is constructed, the model including an ultrasound branch that processes the ultrasound features and a pathology branch that processes the multi-scale pathological features; during the training phase, guided representation learning is performed on the ultrasound features using the multi-scale pathological features, the guided representation learning including cross-modal attention interaction, knowledge distillation, contrastive learning and bond consistency constraints.

[0010] Based on the trained cross-modal learning model, only the preoperative ultrasound image of the breast tumor to be classified is input during the inference stage, and the molecular classification prediction result of breast cancer is output by the ultrasound branch.

[0011] Preferably, extracting ultrasound features from the ultrasound image includes:

[0012] The ultrasound images are preprocessed;

[0013] The tumor region in the pre-processed image is automatically located using a pre-trained segmentation or detection model to obtain the region of interest of the tumor.

[0014] Based on the tumor region of interest, manual features and depth features are extracted respectively, and the two are spliced ​​together to form ultrasound features. The spliced ​​feature vector is then linearly projected onto a feature space that matches the multi-scale pathological features.

[0015] Preferably, extracting multi-scale pathological features from the whole-slice image includes:

[0016] The whole-slice image is preprocessed to retain only the tumor tissue region;

[0017] The preprocessed full-slice image is divided into multiple image blocks to obtain an image block set;

[0018] Extracting block-level pathological features: A pre-trained block-level pathological feature extractor is used to extract the corresponding block-level embedding features for each image block in the image block set, forming a block-level feature matrix;

[0019] Extracting regional pathological features: A pre-trained regional pathological feature extractor is used to extract regional embedding features from the pre-processed whole-slice image to form a regional feature matrix; or, the regional pathological features are obtained by aggregating the block-level pathological features.

[0020] Preferably, the cross-modal attention interaction adopts a two-stage co-attention structure, including:

[0021] In the block-level co-attention stage, the ultrasound features are used as query vectors and the block-level pathological features are used as key and value vectors to calculate the ultrasound features after block-level interaction.

[0022] In the regional-level co-attention stage, the ultrasound features after block-level interaction are used as query vectors, and the regional-level pathological features are used as key and value vectors to calculate the fused cross-modal features.

[0023] Preferably, the knowledge distillation employs the Kullback-Leibler divergence loss function to make the predicted probability distribution of the ultrasound branch output approximate the predicted probability distribution of the pathology branch output.

[0024] Preferably, the contrastive learning adopts bag-level contrastive learning, which forms positive sample pairs by combining the global representation of ultrasound features of the same patient with the global representation of multi-scale pathological features, and forms negative sample pairs by combining the global representation of ultrasound features of different patients with the global representation of multi-scale pathological features, and optimizes the learning using the InfoNCE loss function.

[0025] Preferably, the key consistency constraint adopts an L2 regularized loss function, so that during cross-modal attention interaction, the key vector generated by the ultrasound branch under the self-attention mechanism approximates the key vector generated by the pathology branch.

[0026] Preferably, during the training phase, a weighted cross-entropy loss function is used to simultaneously supervise the predicted outputs of the ultrasound branch and the pathology branch.

[0027] Preferably, during the training phase, a joint loss function is used to optimize the cross-modal learning model. The joint loss function is a weighted sum of weighted cross-entropy loss, knowledge distillation loss, contrastive learning loss, and key consistency constraint loss. Specifically, the weighted cross-entropy loss is calculated using the aforementioned weighted cross-entropy loss function, the knowledge distillation loss is calculated using the aforementioned Kullback-Leibler divergence loss function, the contrastive learning loss is calculated using the aforementioned InfoNCE loss function, and the key consistency constraint loss is calculated using the aforementioned L2 regularization loss function.

[0028] Preferably, during the inference phase, the cross-modal learning model uses only the ultrasound branch for inference and prediction, and the ultrasound branch uses a self-attention mechanism based on ultrasound features to generate key vectors and calculate attention weights.

[0029] A second aspect of this invention proposes an ultrasound-guided cross-modal deep learning-based molecular subtyping device for breast cancer, comprising:

[0030] The image acquisition module is used to acquire preoperative ultrasound images of breast tumors and postoperative immunohistochemical stained whole-section images corresponding to the tumor.

[0031] The feature extraction module includes an ultrasound feature extraction unit and a pathological feature extraction unit. The ultrasound feature extraction unit is used to extract ultrasound features from the ultrasound image, and the pathological feature extraction unit is used to extract multi-scale pathological features from the whole slice image. The multi-scale pathological features include block-level pathological features and region-level pathological features.

[0032] A cross-modal guided learning module is used to construct a cross-modal learning model, which includes an ultrasound branch that processes the ultrasound features and a pathology branch that processes the multi-scale pathological features. During the training phase, guided representation learning is performed on the ultrasound features using the multi-scale pathological features. The guided representation learning includes cross-modal attention interaction, knowledge distillation, contrastive learning, and bond consistency constraints.

[0033] The inference prediction module is used to input only the preoperative ultrasound image of the breast tumor to be classified during the inference stage based on the cross-modal learning model that has been trained, and output the molecular classification prediction result of breast cancer by the ultrasound branch.

[0034] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described ultrasound-guided cross-modal deep learning molecular subtyping method for breast cancer.

[0035] The beneficial effects of this invention are as follows:

[0036] (1) The ultrasound-guided cross-modal deep learning molecular subtyping method for breast cancer provided by this invention constructs a cross-modal learning model. During the training phase, it utilizes high-precision multi-scale pathological features (block-level and region-level) after surgery to guide the representation learning of preoperative ultrasound features. This allows the model to output high-precision molecular subtyping prediction results by inputting only non-invasive ultrasound images during the inference phase. This method eliminates the need for puncture biopsy or obtaining postoperative pathological tissue, avoiding the risks of pain, bleeding, and infection associated with invasive procedures. At the same time, it shortens the diagnostic cycle from several days to several minutes, providing clinicians with non-invasive and timely reliable auxiliary decision-making basis for formulating preoperative treatment plans.

[0037] (2) This invention comprehensively utilizes various guiding techniques such as cross-modal attention interaction, knowledge distillation, contrastive learning, and bond consistency constraints to constrain and optimize ultrasound features from dimensions such as spatial alignment, distribution approximation, feature discrimination, and structural consistency. It successfully transfers and integrates microscopic pathological molecular knowledge into macroscopic ultrasound features, achieving multi-level and multi-angle semantic alignment from macroscopic images to microscopic molecules, significantly improving the representational ability of ultrasound features and the accuracy of molecular subtyping prediction. In particular, during the inference stage, even if the pathological branches are completely removed, the model can still maintain high-performance prediction, demonstrating excellent single-modal inference capabilities.

[0038] Experiments have shown that when using ultrasound alone, the AUROC value of the model of this invention reaches 0.966, significantly better than the traditional single ultrasound model (AUROC=0.861), and comparable to the performance of the model dependent on pathological images (AUROC=0.971) and the combined modality model (AUROC=0.980). Furthermore, the model demonstrates stability on both the internal validation set and multiple external test sets, proving its good robustness, scalability, and clinical adaptability. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the ultrasound-guided cross-modal deep learning-based molecular subtyping method for breast cancer in Example 1.

[0040] Figure 2 This is a flowchart illustrating the experimental design and verification of the ultrasound-guided cross-modal deep learning-based molecular subtyping method for breast cancer in Example 1.

[0041] Figure 3 This is a schematic diagram of the overall architecture of the cross-modal learning model (US-GPS) in Example 1;

[0042] Figure 4 This is a schematic diagram of the two-stage co-attention structure for cross-modal attention interaction in Example 1;

[0043] Figure 5 This is a schematic diagram of the process for extracting ultrasound features from ultrasound images in Example 1;

[0044] Figure 6 This is a flowchart illustrating the process of extracting multi-scale pathological features from whole-slice images in Example 1.

[0045] Figure 7 The graph shows the performance comparison and verification results between the cross-modal learning model (US-GPS) and the baseline model in Example 2. In this graph, A is the macro-mean receiver operating characteristic (ROC) curve of each model, and B is the DeLong test statistical result matrix of the macro-mean AUROC value of each model.

[0046] Figure 8 The figure shows the performance verification results of the cross-modal learning model (US-GPS) for the classification of various molecular subtypes of breast cancer in Example 2. In this figure, A is the one-to-many receiver operating characteristic (ROC) curve of the US-GPS model for the five molecular subtypes, and B is the precision-recall (PR) curve of the US-GPS model for the five molecular subtypes.

[0047] Figure 9 The following is a confusion matrix diagram of the cross-modal learning model (US-GPS) and each baseline model in Example 2. A is the confusion matrix diagram of the pure ultrasound baseline model (US), B is the confusion matrix diagram of the cross-modal learning model (US-GPS, labeled as US*), C is the confusion matrix diagram of the pure pathology baseline model (Pathology), and D is the confusion matrix diagram of the combined modality model (US+Pathology).

[0048] Figure 10 The image shows the verification results of the decision interpretability of the cross-modal learning model (US-GPS) in Example 2. In the image, A is an immunohistochemical staining section of breast cancer biomarkers, B is the visualization results of the model decision basis, the top row is a heatmap of collaborative attention and a magnified view of the key region of interest (ROI), and the bottom row is a visualization result of gradient class activation mapping (Grad-CAM). Detailed Implementation

[0049] To make the objectives and technical solutions of this invention clearer and more complete, the invention will be further described in detail below with reference to embodiments. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Modifications or equivalent substitutions made by those skilled in the art based on their understanding of the technical solutions of this invention, without departing from the spirit and scope of the technical solutions of this invention, are all within the scope of protection of this invention.

[0050] Example 1

[0051] See Figures 1-4This embodiment provides an ultrasound-guided cross-modal deep learning-based molecular subtyping method for breast cancer. This method is implemented by constructing and training a cross-modal learning model. In this embodiment, the model is named "Ultrasound-Guided Pathomic Subtyping" (US-GPS). The core of this method lies in introducing high-precision postoperative whole-section pathological images as "teachers" during the model training phase. These images guide deep representation learning of preoperative ultrasound image features, enabling the trained model to overcome its dependence on pathological data during the inference phase and achieve high-precision molecular subtyping prediction using only non-invasive preoperative ultrasound images. The method specifically includes the following steps:

[0052] Step S100: Obtain preoperative ultrasound images of the breast tumor and postoperative immunohistochemical staining whole slide images corresponding to the tumor.

[0053] Step S200: Extract ultrasound features from the ultrasound image and multi-scale pathological features from the whole slice image. The multi-scale pathological features include block-level pathological features and region-level pathological features.

[0054] Step S300: Construct a cross-modal learning model, the model including an ultrasound branch for processing the ultrasound features and a pathology branch for processing the multi-scale pathological features; during the training phase, guided representation learning is performed on the ultrasound features using the multi-scale pathological features, the guided representation learning including cross-modal attention interaction, knowledge distillation, contrastive learning and bond consistency constraints.

[0055] Step S400: Based on the trained cross-modal learning model, only the preoperative ultrasound image of the breast tumor to be classified is input during the inference stage, and the molecular subtyping prediction result of breast cancer is output by the ultrasound branch.

[0056] The following is a detailed explanation of each of the above steps.

[0057] Step S100: Obtain preoperative ultrasound images of the breast tumor and postoperative immunohistochemical stained whole-section images corresponding to the tumor.

[0058] Specifically, this step aims to construct a multimodal paired dataset for training and validating cross-modal learning models. Here, pairing refers to preoperative ultrasound images (US) and postoperative immunohistochemical (IHC) stained whole-slide images (WSIs) originating from the same tumor lesion in the same patient. Preoperative ultrasound images are typically acquired using conventional ultrasound equipment, including B-mode ultrasound and color Doppler imaging, to provide macroscopic imaging information of the tumor. Postoperative immunohistochemically stained whole-slide images (WSIs) originate from the corresponding surgically removed tumor tissue. Preferably, in this embodiment, multiple serial sections of tumor tissue are prepared, and each section is individually stained for four key breast cancer biomarkers: HER2 (Human Epidermal Growth Factor Receptor 2), ER (Estrogen Receptor), PR (Progesterone Receptor), and Ki-67 (a proliferation index marker). These sections are then digitized using a high-resolution scanner to obtain four sets of corresponding digital whole-slide images. In subsequent model training, ultrasound images of the same patient will be paired and guided with these four sets of pathological whole-section images containing different molecular information.

[0059] In one specific implementation, a retrospective collection of data from four centers totaling 1,283 patients was conducted. Of these, 454 cases (paired with both US and WSIs) were used for model training and validation, while 829 cases containing only US data and molecular subtyping labels were used for external testing. It should be noted that the collection and use of all data complied with relevant ethical review guidelines, and informed consent was obtained from all participating patients. Those skilled in the art should understand that in practical applications, data sources are not limited to this; they can also be extended to other retrospectively or prospectively collected multicenter data, or publicly available datasets, as long as the pairing principle of the same patient and the same lesion is met.

[0060] Step S200: Extract ultrasound features from the ultrasound image and multi-scale pathological features from the whole slice image. The multi-scale pathological features include block-level pathological features and regional-level pathological features.

[0061] In this step, the extraction of ultrasound features and the extraction of multi-scale pathological features are performed independently, and are described separately below.

[0062] like Figure 5 As shown, extracting ultrasound features from the ultrasound image specifically includes:

[0063] Step S201: Preprocess the ultrasound image.

[0064] This step preprocesses the acquired raw ultrasound images to eliminate noise interference during acquisition and standardize data specifications, providing consistent image input for subsequent feature extraction. Preprocessing operations include, but are not limited to, image size normalization (e.g., scaling to 512×512 pixels), image denoising (e.g., using median filtering or non-local mean denoising), image enhancement (e.g., using contrast-limited adaptive histogram equalization), and pixel value normalization (e.g., min-max normalization to the [0,1] interval). It should be understood that the above preprocessing operations all employ conventional image processing methods in this field, and specific parameters can be adjusted according to the actual data distribution; no limitations are imposed here.

[0065] Step S202: The tumor region in the preprocessed image is automatically located using a pre-trained segmentation or detection model to obtain the region of interest of the tumor.

[0066] This step utilizes a pre-trained segmentation or detection model to automatically identify and crop out the region of interest (ROI) containing the tumor lesion from the preprocessed image, eliminating interference from non-tumor areas and allowing subsequent feature extraction to focus more on the lesion itself. In some implementations, a segmentation network (such as U-Net or DeepLab) is used to perform pixel-level classification of the image, generating a binary mask of the tumor region, and then extracting the bounding rectangle or polygon of the mask as the ROI. In other implementations, a deep learning-based detection network (such as the YOLO series or Faster R-CNN) can be used to perform bounding box regression on the tumor in the ultrasound image, outputting the smallest rectangle containing the tumor as the ROI. It should be understood that the choice of the above segmentation or detection model is not limited to a specific architecture; any deep learning model capable of localizing tumor lesions can be applied to this step.

[0067] Step S203: Based on the region of interest of the tumor, extract manual features and depth features respectively, stitch them together to form ultrasound features, and linearly project the stitched feature vector onto a feature space that matches the multi-scale pathological features.

[0068] This step employs a strategy combining handcrafted features and deep features to extract and fuse multi-scale features from the Region of Interest (ROI) in ultrasound images. Handcrafted feature extraction (also known as radiomics features) extracts traditional radiomics features of the ROI, including shape features (such as tumor aspect ratio, edge spiculation, and irregularity index), texture features (such as contrast, correlation, and energy parameters extracted from the Gray-Level Co-occurrence Matrix (GLCM) and Gray-Level Run-Length Matrix (GLRLM)), and wavelet transform features. These features have clear clinical interpretability, corresponding to the visual perception of doctors when interpreting images. Deep feature extraction uses convolutional neural networks (such as ResNet50) or similar backbone networks to perform convolution operations on the ROI to extract high-dimensional deep semantic features. The two are then concatenated to form a comprehensive ultrasound feature vector. It integrates interpretable clinical radiomics information with deep semantic information, providing a rich representational basis for subsequent guidance on multi-scale pathological features.

[0069] It should be noted that, since the block-level and regional pathological features output by the pathology branches are usually in a high-dimensional embedding space, the spliced ​​ultrasound feature vectors... The dimensions may not be consistent with them. To enable cross-modal computation between ultrasound features and multi-scale pathological features, this step also includes processing the ultrasound feature vectors... Linear projection is performed onto a feature space that matches the multi-scale pathological features, making its dimensions consistent with the dimensions of block-level and regional-level pathological features, thus achieving alignment of the feature space.

[0070] like Figure 6 As shown, multi-scale pathological features are extracted from the whole-slice image, specifically including:

[0071] Step S211: Preprocess the whole slice image to retain only the tumor tissue region.

[0072] As mentioned earlier, the whole-section images are obtained through immunohistochemical staining and digital scanning, resulting in extremely high resolution, but also containing a large number of non-tumor tissue areas such as fat and blank areas. This step uses image processing methods such as color thresholding and morphological manipulation to remove these non-informative areas, retaining only the effective tumor tissue areas to reduce subsequent computation and improve feature purity. It should be noted that, to ensure biological consistency, the pathological tissue sections used for digital scanning must spatially correspond to the tumor lesion areas detected by preoperative ultrasound scanning.

[0073] Step S212: Divide the preprocessed full-slice image into multiple image blocks to obtain an image block set.

[0074] Because the whole-slice image has extremely high resolution, this step cuts the preprocessed image, which retains only the tumor tissue region, into multiple fixed-size image patches, such as 256×256 pixels or 512×512 pixels, forming an image patch set. This patch processing can preserve the microscopic details in the image.

[0075] Step S213, extract block-level pathological features: using a pre-trained block-level pathological feature extractor, extract the corresponding block-level embedding features for each image block in the image block set to form a block-level feature matrix.

[0076] This step employs a pre-trained, parameter-frozen block-level pathological feature extractor to extract features from each image block. In a preferred embodiment, the block-level pathological feature extractor uses a VisionTransformer (ViT) architecture, specifically a UNI encoder. This encoder has been pre-trained on large-scale histopathological data using self-supervised learning (e.g., the DINOv2 paradigm), enabling it to keenly capture tumor-specific morphological patterns, IHC staining features, etc. By inputting all the segmented pathological image blocks into the parameter-frozen UNI encoder, the block-level feature vector corresponding to each image block is output, and all feature vectors form a block-level pathological feature matrix. This process can be represented as:

[0077]

[0078] in, The mapping function represents the block-level pathological feature extractor. This represents the preprocessed full-slice image (which has been segmented into blocks). The number of image patches, The feature dimensions for each image patch.

[0079] Step S214, extracting regional pathological features: using a pre-trained regional pathological feature extractor, extracting regional embedding features from the preprocessed whole slice image to form a regional feature matrix; or, by aggregating the block-level pathological features, obtaining the regional pathological features.

[0080] To capture a wider range of tissue structure information, this step extracts regional pathological features that reflect the overall arrangement of tumor tissue, cell density distribution, stroma ratio, and other macroscopic pathological characteristics. These features can serve as high-level semantic supervision signals in subsequent cross-modal guided learning. In practice, one or a combination of two techniques can be used. In some implementations, a pre-trained and parameter-frozen regional pathological feature extractor can be used to perform end-to-end feature extraction on the preprocessed whole-slice image. Preferably, a hierarchical visual Transformer (Vit) architecture, such as the HIPT (Hierarchical Image Pyramid Transformer) model, can be used. This model uses a built-in multi-level self-attention mechanism to hierarchically process the whole-slice image, directly outputting regional feature vectors representing different spatial scales, forming a regional pathological feature matrix. This process can be represented as:

[0081]

[0082] in, This represents the preprocessed full-slice image. This represents the mapping function for the regional pathological feature extractor. The number of regional pathological features. The feature dimension is consistent with or aligned with the block-level pathological feature dimension via projection.

[0083] In other implementations, the block-level pathological features extracted in step S213 can be integrated through a separate feature aggregation module to obtain region-level pathological features. Specifically, this aggregation module can employ a visual Transformer encoder architecture (e.g., ViT). 4096 -256 encoder), will block-level feature matrix The feature sequence is input into this encoder, which uses its self-attention mechanism to perform global interaction and information fusion on the block-level features in the sequence, thereby reconstructing or aggregating a region-level feature matrix containing macro-contextual information. .

[0084] It is worth noting that this embodiment effectively addresses the problem of tumor heterogeneity by constructing a multi-scale pathological feature system of "block-level + region-level". Specifically, tumors often contain regions with different degrees of differentiation, which cannot be fully described by features at a single scale. Block-level pathological features focus on local cell morphology and atypia, while region-level pathological features focus on the overall tissue structure and spatial distribution. The combination of the two provides rich and hierarchical supervisory signals to guide ultrasound feature learning.

[0085] Step S300: Construct a cross-modal learning model, the model including an ultrasound branch for processing the ultrasound features and a pathology branch for processing the multi-scale pathological features; during the training phase, guided representation learning is performed on the ultrasound features using the multi-scale pathological features, the guided representation learning including cross-modal attention interaction, knowledge distillation, contrastive learning, and bond consistency constraints.

[0086] Specifically, combined Figure 2 , Figure 3 and Figure 4 The cross-modal learning model constructed in this step comprises two core branches: an ultrasound branch that processes ultrasound features and a pathology branch that processes multi-scale pathological features. Since ultrasound images lack direct molecular pathological information, directly utilizing ultrasound features for molecular subtyping prediction of breast cancer often results in limited accuracy. In this step, during the model training phase, the multi-scale pathological features processed by the pathology branch are used as strong supervisory signals to guide the representation learning of ultrasound features in the ultrasound branch.

[0087] Guided representation learning encompasses various methods, including cross-modal attention interaction, knowledge distillation, contrastive learning, and bond consistency constraints. Essentially, it utilizes multi-scale pathological features within the feature space to correct or enhance ultrasound features. For example, cross-modal attention interaction teaches the model which regions in ultrasound images are highly correlated with specific pathological features; knowledge distillation forces the ultrasound branch to mimic the prediction logic of the pathology branch; contrastive learning drives ultrasound features and multi-scale pathological features of the same patient to converge in the representation space, while features from different patients diverge; bond consistency constraints force the attention key vectors generated by the ultrasound branch during training to align with the key vectors provided by the pathology branch, ensuring that the model's attention focus pattern remains consistent with pathological guidance even when using only ultrasound-generated key vectors during inference. Through these guidance methods, ultrasound features are endowed with a previously unavailable pathological perspective, achieving semantic alignment from macroscopic imaging to microscopic molecular levels. Each guided representation learning method will be described in detail below.

[0088] (1) Cross-modal attention interaction

[0089] In cross-modal learning models, effectively integrating microscopic pathological features with macroscopic ultrasound features is a key technical challenge. Due to significant differences in imaging principles, resolution, and spatial dimensions between ultrasound images and whole-slice pathological images, simple feature stitching is insufficient to uncover the intrinsic correlation between them. Therefore, this embodiment employs cross-modal attention interaction, using ultrasound features as queries and multi-scale pathological features as keys and values. By calculating attention weights, the model automatically searches for ultrasound configuration features in ultrasound images that are highly correlated with the spatial heterogeneity of four IHC indices (HER2, ER, PR, Ki-67). As a preferred implementation, this embodiment uses a two-stage co-attention structure, achieving deep alignment of features through hierarchical interaction. Figure 3 and Figure 4 As shown, the two-stage co-attention structure includes a block-level co-attention stage and a region-level co-attention stage.

[0090] First, in the block-level co-attention stage, using ultrasound features as the query vector and block-level pathological features as the key-value vector, the ultrasound features after block-level interaction are calculated. The calculation process satisfies the following formula:

[0091]

[0092]

[0093] in, , , , This represents the learnable projective weight matrix. Indicates ultrasound characteristics, Indicates block-level pathological features. This represents the block-level common attention weight matrix. This represents the ultrasonic features after block-level interaction. Representing feature dimension, This represents the activation function.

[0094] At this stage, ultrasound features actively search for relevant microscopic pathological evidence in a block-level pathological feature database. For example, a hypoechoic region in an ultrasound image may correspond to a densely packed cell nucleus patch in a pathological image. This is achieved through an attention weight matrix. The model can filter out the pathological image blocks that have the most diagnostic value for the current ultrasound features, and aggregate these block-level pathological features into the ultrasound features, thereby obtaining enhanced features rich in microscopic pathological information. This achieves a preliminary mapping from the macro to the micro level.

[0095] Subsequently, in the regional-level co-attention stage, the model further integrates features at the tissue structure level. This stage uses the block-level interactive ultrasound features as the query vector and the regional-level pathological features as the key-value vector to calculate the fused cross-modal features. The calculation process satisfies the following formula:

[0096]

[0097]

[0098] in, , , , This represents the learnable projective weight matrix. This represents the ultrasonic features after block-level interaction. Indicates regional pathological features. This represents the region-level common attention weight matrix. This represents the fused cross-modal features. Representing feature dimension, This represents the activation function.

[0099] After block-level interaction, the ultrasound features already possess local pathological perception capabilities. Building upon this foundation, this stage further interacts them with regional-level pathological features that reflect the overall tumor tissue structure, such as the ratio of stroma to parenchyma and tumor growth patterns. This allows for the capture of the correlation between ultrasound features and the overall pathological structure, generating cross-modal features that integrate local details and global structural information. It should be understood that cross-modal attention interaction mechanisms are not limited to the two-stage co-attention structure mentioned above, and can also be implemented using other attention variants.

[0100] (2) Knowledge distillation

[0101] As a preferred embodiment, the knowledge distillation in this example employs the Kullback-Leibler (KL) divergence loss function to approximate the predicted probability distribution of the ultrasound branch output. The loss function satisfies the following formula:

[0102]

[0103] in, This represents the loss from knowledge distillation. Denotes KL divergence, Indicates molecular subtype category, This represents the probability distribution of molecular subtyping predictions output from the pathology branch. This represents the probability distribution of molecular subtyping predictions from the ultrasound branch output. and These represent the categories of pathological branches and ultrasound branches, respectively. The predicted probability.

[0104] In this embodiment, the pathology branch, due to its input of high-precision whole-slice images, has higher reliability in its predictions and is considered the teacher distribution. The ultrasound branch, as the student distribution, not only learns the true labels (hard labels) but also mimics the predicted distribution of the pathology branch (soft labels). KL divergence measures the difference between the two probability distributions. By minimizing this loss, the ultrasound branch is forced to approximate the pathology branch in the morphology of its predicted probabilities, thereby learning richer inter-class relationship information and improving classification robustness.

[0105] (3) Comparative learning

[0106] As a preferred implementation, this embodiment employs bag-level contrastive learning. Positive sample pairs are formed by pairing the global representation of ultrasound features of the same patient with the global representation of multi-scale pathological features, and negative sample pairs are formed by pairing the global representation of ultrasound features of different patients with the global representation of multi-scale pathological features. The InfoNCE loss function is used for optimization. In this embodiment, the global representation of ultrasound features is obtained by global average pooling of ultrasound features, and the global representation of multi-scale pathological features is preferably obtained by global average pooling of regional pathological features. The loss function satisfies the following formula:

[0107]

[0108] in, Indicates the contrast learning loss. The number of samples in a training batch. This is the cosine similarity calculation function. For temperature hyperparameters, and They represent the first Global representation of ultrasound features and multi-scale pathological features of individual samples. This represents the global representation of the multi-scale pathological features of the j-th sample.

[0109] In this embodiment, contrastive learning aims to bridge the gap between different modalities of the same patient in the feature space, while simultaneously widening the gap between features from different patients. This process constructs a shared feature space, bringing ultrasound features and pathological features with the same molecular subtype or similar biological characteristics closer together in space. Through this feature-level alignment, the model can learn cross-modal semantic consistency. Even if ultrasound images and pathological images appear visually distinct, the model can extract the inherent correlation features between them, thereby enhancing the ability of ultrasound branches to discriminate molecular subtypes.

[0110] (4) Key consistency constraint

[0111] As a preferred implementation, the key consistency constraint in this embodiment employs an L2 regularized loss function, making the key vector generated solely based on ultrasound features approximate the key vector generated from pathological features. The loss function satisfies the following formula:

[0112]

[0113] in, This represents the loss due to key consistency constraints. This indicates a bullish self-attention strategy. Indicates ultrasound characteristics, The key vector is generated from multi-scale pathological features.

[0114] It should be noted that in the cross-modal attention interaction during the training phase, the key vectors are derived from pathological features. If the pathological branch is directly removed, the ultrasound branch will be unable to obtain key vectors, causing the attention mechanism to fail during inference. Therefore, this embodiment introduces a key consistency constraint and trains a multi-head self-attention module based on ultrasound features to ensure that the key vectors it generates... As close as possible to the key vector generated by the pathological branch Through L2 regularization constraints, the model gradually learns the mapping from ultrasound features to pathological key vectors during training. During inference, even without pathological image input, the ultrasound branch can automatically generate high-quality key vectors, thus maintaining the normal operation of the attention mechanism and ensuring effective transfer of model performance.

[0115] It should be understood that, as a preferred implementation, this embodiment combines the above four guided learning strategies during the training of the cross-modal learning model to obtain the best model inference performance. It should also be understood that the four strategies can be flexibly combined according to actual application needs. This embodiment is not limited to using all four strategies simultaneously; rather, one or more guided strategies can be selected and combined based on computing resources, deployment environment, or accuracy requirements. For example, when computing resources are limited or rapid deployment is required, only contrastive learning and knowledge distillation strategies can be used; when fine feature fusion is required but a simplified loss function is desired, while ensuring that the ultrasonic branch in the inference stage can independently generate high-quality key vectors, cross-modal attention interaction, contrastive learning, and key consistency constraints can be used.

[0116] Furthermore, the cross-modal learning model also includes a classifier. In one specific implementation, the classifier is connected to the ends of the ultrasound and pathology branches, respectively. Its input is the final feature representation after deep feature encoding by a visual transformer (ViT) and aggregation by global average pooling (GAP), used to predict the specific molecular subtype of breast cancer. Preferably, in this embodiment, the classifier adopts a one-vs-Rest strategy for multi-classification tasks. Specifically, this embodiment decomposes the molecular subtyping prediction problem into binary subtasks targeting four key IHC markers (ER, PR, HER2, Ki-67). Figure 2 As shown, during the training phase, ultrasound features are cross-modal aligned with the corresponding pathological features for each biomarker, training four biomarker-specific binary classifiers. Each binary classifier outputs the positive probability of the corresponding biomarker. Simultaneously, the outputs of these four binary classifiers are connected to a multilayer perceptron (MLP). The predicted probabilities of the four biomarkers form a four-dimensional feature vector, which serves as the input to the MLP. The MLP is responsible for learning the complex nonlinear mapping from biomarker probability combinations to the final molecular subtype and outputting the probability distributions of five subtypes (Luminal A, Luminal B (HER2 negative), Luminal B (HER2 positive), HER2-enriched, and triple-negative).

[0117] During the inference phase, after inputting the ultrasound image to be classified into the trained model, the positive probability values ​​of the four markers can be obtained through the ultrasound branch alone. These are then further fused and mapped by the MLP to the final probability distribution of the five molecular subtypes. Through this one-vs-rest strategy of first predicting each marker separately and then making a fusion decision, the model can fully utilize multi-marker information. Each binary classifier focuses on a specific molecular marker, reducing the complexity of the multi-classification task. Simultaneously, by learning the nonlinear mapping relationship between marker combinations and subtypes through the MLP, the accuracy and flexibility of the final classification prediction are improved.

[0118] It should be noted that, in order to train the model to accurately predict breast cancer molecular subtypes, this embodiment uses a weighted cross-entropy loss function to supervise the predicted probabilities output by the model. Specifically, during the training phase, the model simultaneously outputs the predicted probability distributions of the ultrasound branch and the pathology branch. Both are used to calculate the cross-entropy loss against the true molecular subtype label, and the two losses are added together to obtain the total loss for this part. The weighted cross-entropy loss function satisfies the following formula:

[0119]

[0120] in, This represents the weighted cross-entropy loss. This represents the total number of molecular subtypes ( ), For category indexing, Indicate category Weighting coefficients; As an indicator variable, when the true class of the sample is hour, ,otherwise ; and The samples predicted by ultrasound branches and pathological branches belong to the following categories. The probability of.

[0121] It should be understood that in this embodiment, the prediction of the pathological branch itself has high accuracy, and applying supervision to it can stabilize the training process; while applying supervision to the ultrasound branch directly drives it to learn the mapping from ultrasound images to molecular subtypes, which can complement the aforementioned guided learning strategy and jointly improve the final classification performance of the cross-modal learning model.

[0122] Furthermore, in this embodiment, during the training phase, a joint loss function is used to perform end-to-end joint backpropagation optimization of the parameters of the cross-modal learning model until the model converges, resulting in a trained model. The joint loss function satisfies the following formula:

[0123]

[0124] in, For knowledge distillation loss, To compare learning loss, For the loss due to key consistency constraints, For weighted cross-entropy loss, These are the weighting coefficients corresponding to each loss term.

[0125] It should be noted that, in this embodiment, the above four parts are preferably weighted and summed to form the total training loss. However, in practical applications, the joint loss function should be adaptively adjusted according to the guided learning strategy adopted. The corresponding loss term is only included in the joint loss function when the corresponding guided technique is used. For example, if only cross-modal attention interaction, contrastive learning, and key consistency constraints are used, without knowledge distillation, the joint loss function will not include... Item. Other cases follow the same principle and will not be elaborated here.

[0126] Step S400: Based on the trained cross-modal learning model, only the preoperative ultrasound image of the breast tumor to be classified is input during the inference stage, and the molecular subtyping prediction result of breast cancer is output by the ultrasound branch.

[0127] After the model has undergone the above training and optimization, its ultrasound branch has the ability to independently predict molecular subtyping. In clinical applications (inference stage), it is no longer necessary to obtain the patient's pathological slides; only the ultrasound images acquired before surgery need to be input.

[0128] Specifically, preoperative ultrasound images of patients to be classified are acquired and input into a trained cross-modal learning model. The model processes the image using the ultrasound branch. Thanks to guided learning during the training phase, the extracted ultrasound features can be implicitly mapped to a feature space containing pathological IHC molecular information, possessing a representational capability comparable to the "ultrasound + pathology" combined modality. Subsequently, the classifier at the back end of the model outputs the probability of the lesion belonging to each molecular subtype based on the above features, providing the final molecular classification prediction result, such as Luminal A, Luminal B (HER2 negative / positive), HER2 enriched, or triple negative. Clinicians can directly formulate preoperative treatment plans based on this non-invasive prediction result, such as deciding whether to perform neoadjuvant chemotherapy or select targeted drugs.

[0129] As a preferred implementation, if key consistency constraints are used during the training phase, the model can utilize a self-attention mechanism during inference to simulate cross-modal attention interactions, thereby further improving prediction accuracy. Specifically, the ultrasound branch uses a self-attention mechanism based on ultrasound features to calculate attention weights, as shown in the formula:

[0130]

[0131] in, Represents the attention weight matrix. This represents a query vector generated from ultrasound features. This represents the key vector generated from ultrasound features through a multi-head self-attention operation. For feature dimension, This is the activation function.

[0132] It should be noted that the effectiveness of using self-attention mechanisms to simulate cross-modal attention interactions stems from the guidance of key consistency constraints during the training phase. This allows the model to generate self-attention keys that approximate pathological key vectors based solely on ultrasound features, thereby completing feature enhancement and fusion without relying on pathological input. It should be understood that in other implementations, other feature-internal interaction mechanisms capable of generating effective attention weights based on single-modal features can also be employed.

[0133] Example 2

[0134] This embodiment further validates and evaluates the effectiveness and generalization ability of the method described in Embodiment 1 and the cross-modal learning model (US-GPS) constructed thereon through a multi-center retrospective study.

[0135] 1. Validation and Evaluation Design

[0136] As mentioned above, the data used for validation and evaluation in this embodiment came from clinical data of 1283 breast cancer patients from four independent medical centers. Among them, 454 patients had paired preoperative ultrasound images and postoperative immunohistochemical stained whole slide images, which were divided into training and validation sets. The training set was used for model training in Example 1, and the validation set was used for internal validation in this embodiment. The other 829 patients only had preoperative ultrasound images and their postoperative pathological gold standard classification labels, which served as multiple external test sets in this embodiment to evaluate the generalization performance of the model in simulating real clinical scenarios (preoperative ultrasound images only).

[0137] To scientifically evaluate the performance of the cross-modal learning model of this invention, such as Figure 2 As shown in Table 1, this embodiment designs four comparison strategies for model validation and evaluation:

[0138] Table 1. Four Comparison Strategies for Model Validation and Evaluation

[0139]

[0140] 2. Verification and Evaluation Results

[0141] (1) Validation of the interpretability of model decisions

[0142] To verify the interpretability of the cross-modal attention interaction mechanism employed in the US-GPS model constructed in Example 1 of this invention, immunohistochemical staining of key molecular subtyping markers for breast cancer (ER, PR, Her2, Ki-67) was used as the gold standard to visualize and analyze the attention weight distribution and decision-making basis of the model. Figure 10 As shown, the co-attention heatmaps and magnified region of interest (ROI) results demonstrate that the model's high-weight regions accurately focus on tumor cell regions expressing positive biomarkers. Grad-CAM visualization further shows that the model's decision response regions (highlighted in red / yellow in the figure) highly match the distribution of pathological features. These results indicate that the US-GPS model's cross-modal attention interaction mechanism achieves semantic alignment between macroscopic ultrasound imaging features and microscopic pathological features. The model's predictive inference has a clear biological basis, rather than relying on irrelevant noise or spurious features, significantly improving the model's clinical interpretability and credibility.

[0143] (2) Validation and evaluation of model prediction performance

[0144] On the internal validation set, the area under the receiver operating characteristic (AUROC) curve was used as the primary metric to quantitatively compare the overall performance of the four strategy models. Figure 7 The diagram shows that the US-GPS model (Strategy 2, marked as...) The macro-mean AUROC value of the US-GPS model reached 0.966 (95% confidence interval: 0.961–0.991), significantly outperforming the pure ultrasound baseline model (Strategy 3, labeled US in the figure, macro-mean AUROC = 0.861, 95% confidence interval: 0.807–0.915, DeLong test p = 0.02). Furthermore, there was no statistically significant difference between the US-GPS model and the pure pathology baseline model (Strategy 4, labeled Pathology in the figure, macro-mean AUROC = 0.971, 95% confidence interval: 0.948–0.987) and the combined modality model (Strategy 1, labeled US+Pathology in the figure, macro-mean AUROC = 0.980, 95% confidence interval: 0.973–0.997) (DeLong test p > 0.05). This indicates that the US-GPS model, using only ultrasound images, achieves predictive performance comparable to models relying on pathology images.

[0145] Further evaluation of the US-GPS model's classification and predictive ability for five specific molecular subtypes (Luminal A, Luminal B (HER2-negative), Luminal B (HER2-positive), HER2-positive, and triple-negative). ROC curve ( Figure 8 Analysis A) in the figure shows that the model's AUROC values ​​for each subtype reached 0.942 (Luminal A, marked as Lumina A in the figure, 95% confidence interval: 0.895-0.979), 0.989 (Luminal B (HER2 negative), marked as Lumina B(-) in the figure, 95% confidence interval: 0.966-1.000), 0.985 (Luminal B (HER2 positive), marked as Lumina B(+) in the figure, 95% confidence interval: 0.949-1.000), 0.988 (HER2 positive, marked as HER2-enriched in the figure, 95% confidence interval: 0.963-1.000), and 0.986 (Triple negative, marked as Triple negative in the figure, 95% confidence interval: 0.960-1.000), indicating that the model has a high and balanced ability to identify all subtypes. Precision-Recall (PR) curve Figure 8 As shown in B), the area under the precision-recall curve (AUPRC) for each subtype is greater than 0.8, indicating that the model can still effectively identify the minority class even under imbalanced sample conditions. The corresponding confusion matrix ( Figure 9 The results show that the values ​​on the main diagonal (number of correctly classified samples) are significantly higher than those on the off-diagonal, further validating the model's accuracy and robustness in multi-classification tasks.

[0146] It should be noted that, Figure 7 A and Figure 8 The shaded area around the ROC curve shown in Figure A represents the 95% confidence band of the corresponding model's ROC curve, reflecting the statistical uncertainty of the model's predictive performance. Figure 7 Taking A as an example, the gray curve is the ROC curve of the pure ultrasound baseline model (US), with an AUROC value of 0.861 and a 95% confidence interval of 0.807-0.915. This confidence interval has a large range, and the corresponding 95% confidence band of the ROC curve is also large, resulting in a wide shaded area, indicating relatively weak stability in model performance estimation. The green curve represents the US-GPS model (…). The ROC curve of the model has an AUROC value of 0.966 and a 95% confidence interval of 0.961-0.991. The confidence interval is relatively narrow, so the 95% confidence band of the ROC curve almost coincides with the curve, indicating that the model performance estimation is more stable.

[0147] Furthermore, across multiple external test sets, the AUROC values ​​of the US-GPS model remained between 0.915 and 0.963. Subgroup analyses also demonstrated that the AUROC values ​​remained between 0.788 and 1.000 across patient populations of different ages, tumor sizes, and BI-RADS grades. These results indicate that the US-GPS model of this invention possesses beneficial stability and clinical adaptability, and can serve as a non-invasive and reliable clinical decision support tool.

[0148] Example 3

[0149] This embodiment provides an ultrasound-guided cross-modal deep learning-based molecular subtyping device for breast cancer, comprising:

[0150] The image acquisition module is used to acquire preoperative ultrasound images of breast tumors and postoperative immunohistochemical stained whole-section images corresponding to the tumor.

[0151] The feature extraction module includes an ultrasound feature extraction unit and a pathological feature extraction unit. The ultrasound feature extraction unit is used to extract ultrasound features from the ultrasound image, and the pathological feature extraction unit is used to extract multi-scale pathological features from the whole slice image. The multi-scale pathological features include block-level pathological features and region-level pathological features.

[0152] A cross-modal guided learning module is used to construct a cross-modal learning model, which includes an ultrasound branch that processes the ultrasound features and a pathology branch that processes the multi-scale pathological features. During the training phase, guided representation learning is performed on the ultrasound features using the multi-scale pathological features. The guided representation learning includes cross-modal attention interaction, knowledge distillation, contrastive learning, and bond consistency constraints.

[0153] The inference prediction module is used to input only the preoperative ultrasound image of the breast tumor to be classified during the inference stage based on the cross-modal learning model that has been trained, and output the molecular classification prediction result of breast cancer by the ultrasound branch.

[0154] For detailed descriptions of the functions of each module, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0155] Example 4

[0156] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the ultrasound-guided cross-modal deep learning-based molecular subtyping method for breast cancer described in Embodiment 1.

Claims

1. An ultrasound-guided cross-modality deep learning breast cancer molecular subtyping method, characterized in that, Includes the following steps: Obtain preoperative ultrasound images of breast tumors and postoperative immunohistochemical stained whole slide images corresponding to the tumors; Ultrasound features from the ultrasound images and multi-scale pathological features from the whole slice images are extracted. The multi-scale pathological features include block-level pathological features and region-level pathological features. Construct a cross-modal learning model, the model including an ultrasound branch that processes the ultrasound features and a pathology branch that processes the multi-scale pathological features; During the training phase, the ultrasound features are subjected to guided representation learning using the multi-scale pathological features. The guided representation learning includes cross-modal attention interaction, knowledge distillation, contrastive learning, and bond consistency constraints. Based on the trained cross-modal learning model, only the preoperative ultrasound image of the breast tumor to be classified is input during the inference stage, and the molecular classification prediction result of breast cancer is output by the ultrasound branch.

2. The method of claim 1, wherein, Extracting ultrasound features from the ultrasound image, including: The ultrasound images are preprocessed; The tumor region in the pre-processed image is automatically located using a pre-trained segmentation or detection model to obtain the region of interest of the tumor. Based on the tumor region of interest, manual features and depth features are extracted respectively, and the two are spliced ​​together to form ultrasound features. The spliced ​​feature vector is then linearly projected onto a feature space that matches the multi-scale pathological features.

3. The method of claim 1, wherein, Extracting multi-scale pathological features from the whole-slice image, including: The whole-slice image is preprocessed to retain only the tumor tissue region; The preprocessed full-slice image is divided into multiple image blocks to obtain an image block set; Extracting block-level pathological features: A pre-trained block-level pathological feature extractor is used to extract the corresponding block-level embedding features for each image block in the image block set, forming a block-level feature matrix; Extracting regional pathological features: A pre-trained regional pathological feature extractor is used to extract regional embedding features from the pre-processed whole-slice image to form a regional feature matrix; or, the regional pathological features are obtained by aggregating the block-level pathological features.

4. The method of claim 1, wherein, The cross-modal attention interaction adopts a two-stage co-attention structure, including: In the block-level co-attention stage, the ultrasound features are used as query vectors and the block-level pathological features are used as key and value vectors to calculate the ultrasound features after block-level interaction. In the regional-level co-attention stage, the ultrasound features after block-level interaction are used as query vectors, and the regional-level pathological features are used as key and value vectors to calculate the fused cross-modal features.

5. The method of claim 1, wherein, The knowledge distillation employs the Kullback-Leibler divergence loss function to make the predicted probability distribution of the ultrasound branch output approximate the predicted probability distribution of the pathology branch output.

6. The method of claim 5, wherein, The contrastive learning adopts bag-level contrastive learning, which forms positive sample pairs by pairing the global representation of ultrasound features of the same patient with the global representation of multi-scale pathological features, and forms negative sample pairs by pairing the global representation of ultrasound features of different patients with the global representation of multi-scale pathological features, and optimizes the learning using the InfoNCE loss function.

7. The method of claim 6, wherein, The key consistency constraint employs an L2 regularized loss function, ensuring that during cross-modal attention interaction, the key vector generated by the ultrasound branch under the self-attention mechanism approximates the key vector generated by the pathology branch.

8. The method according to claim 7, characterized in that, During the training phase, a weighted cross-entropy loss function is used to supervise the predicted outputs of both the ultrasound branch and the pathology branch.

9. The method according to claim 8, characterized in that, During the training phase, a joint loss function is used to optimize the cross-modal learning model. The joint loss function is a weighted sum of weighted cross-entropy loss, knowledge distillation loss, contrastive learning loss, and key consistency constraint loss. Specifically, the weighted cross-entropy loss is calculated using the weighted cross-entropy loss function, the knowledge distillation loss is calculated using the Kullback-Leibler divergence loss function, the contrastive learning loss is calculated using the InfoNCE loss function, and the key consistency constraint loss is calculated using the L2 regularization loss function.

10. The method according to claim 1, characterized in that, During the inference phase, the cross-modal learning model uses only the ultrasound branch for inference and prediction, and the ultrasound branch uses a self-attention mechanism based on ultrasound features to generate key vectors and calculate attention weights.

11. A cross-modal deep learning-based molecular subtyping device for breast cancer guided by ultrasound, characterized in that, include: The image acquisition module is used to acquire preoperative ultrasound images of breast tumors and postoperative immunohistochemical stained whole-section images corresponding to the tumor. The feature extraction module includes an ultrasound feature extraction unit and a pathological feature extraction unit. The ultrasound feature extraction unit is used to extract ultrasound features from the ultrasound image, and the pathological feature extraction unit is used to extract multi-scale pathological features from the whole slice image. The multi-scale pathological features include block-level pathological features and region-level pathological features. A cross-modal guided learning module is used to construct a cross-modal learning model, which includes an ultrasound branch that processes the ultrasound features and a pathology branch that processes the multi-scale pathological features. During the training phase, guided representation learning is performed on the ultrasound features using the multi-scale pathological features. The guided representation learning includes cross-modal attention interaction, knowledge distillation, contrastive learning, and bond consistency constraints. The inference prediction module is used to input only the preoperative ultrasound image of the breast tumor to be classified during the inference stage based on the cross-modal learning model that has been trained, and output the molecular classification prediction result of breast cancer by the ultrasound branch.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the ultrasound-guided cross-modal deep learning molecular subtyping method for breast cancer as described in any one of claims 1-10.

Images