Artificial intelligence system for identifying breast cancer molecular subtypes based on multi-modal images

Abstract

The application discloses an AI system for identifying breast cancer molecular typing based on multi-modal images and belongs to the technical field of medicine. The application comprises an image acquisition module, a dual-channel heterogeneous neural network module, a cross-modal self-attention fusion module, a molecular typing identification module and an explainable AI component. Multi-modal images are integrated by cooperating with mammography, dynamic contrast-enhanced MRI and ultrasound images, breaking through the limitation of single modal information, comprehensively capturing the anatomical, functional and tissue characteristics of the lesion, effectively reducing the risk of misdiagnosis and missed diagnosis, the dual-channel heterogeneous neural network and the cross-modal self-attention fusion module ensure the accurate capture of the fine-grained features of the mammography, the spatiotemporal features of the MRI and the shape and hardness features of the ultrasound, mine the cross-modal collaborative information through nonlinear alignment, generate high-discriminative cross-modal feature representation, improve the model generalization ability and the identification effect of rare typing, and the explainable AI component enables doctors to clearly master the model decision basis and assist doctors in deeply understanding the nature of the lesion.

Description

Technical Field

[0001] This invention relates to the field of medical technology, and in particular to an AI system for molecular subtyping identification of breast cancer based on multimodal imaging. Background Technology

[0002] Currently, in clinical practice, molecular subtyping of breast cancer mainly relies on pathological biopsy and immunohistochemical testing. This method has limitations such as being invasive, having a long testing cycle, and depending on sample quality. Furthermore, it is difficult to perform biopsy on patients with inaccessible lesions or those unsuitable for biopsy, thus failing to meet the needs for early diagnosis and rapid decision-making.

[0003] Traditional imaging diagnosis relies on physicians' subjective experience, resulting in poor diagnostic consistency and insufficient ability to differentiate early / rare subtypes. Existing AI-assisted diagnostic systems mostly focus on single-modal image analysis, failing to fully explore the complementary information of multimodal images, and the model decision-making process lacks interpretability, making it difficult to gain the trust of clinicians. At the same time, the scarcity of rare molecular subtype samples in clinical practice limits the generalization ability of existing models, making it difficult to adapt to complex clinical scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide an AI system for molecular subtyping identification of breast cancer based on multimodal imaging, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an AI system for molecular subtyping identification of breast cancer based on multimodal imaging, comprising: The image acquisition module is used to acquire mammograms, dynamic contrast-enhanced MRI images, and ultrasound images of the same patient, and to preprocess the acquired multimodal images to form a standardized image sequence. The dual-channel heterogeneous neural network module includes a first feature extraction submodule and a second feature extraction submodule. The first feature extraction submodule is used to extract a first image feature from mammography images, and the second feature extraction submodule is used to extract a second image feature from dynamic contrast-enhanced MRI images and ultrasound images. A cross-modal self-attention fusion module, connected to the dual-channel heterogeneous neural network module, is used to receive the first image features and the second image features, and to perform nonlinear alignment and interaction on the first image features and the second image features through a self-attention mechanism to generate a fused cross-modal feature representation; The molecular typing identification module, connected to the cross-modal self-attention fusion module, is used to receive the cross-modal feature representation and output the molecular typing identification result of breast cancer based on the cross-modal feature representation; An interpretable AI component, connected to the molecular typing identification module, is used to generate and output pixel-level heatmaps and key image feature reports based on the decision-making process of the molecular typing identification module. The key image feature reports are used to display image regions and biological features associated with molecular typing.

[0006] Furthermore, in the dual-channel heterogeneous neural network module, the first feature extraction submodule adopts a Transformer-based multi-scale feature extraction architecture, including four encoding blocks and two scale fusion layers: Each coding block contains a 64-channel convolutional layer, a LayerNorm layer, a GELU activation function, and a residual connection, used to capture fine-grained features such as calcifications and structural distortions in molybdenum images. The scale fusion layer maps the feature maps output by different coding blocks to the same channel dimension through 1×1 convolution, and uses attention pooling to weight and aggregate the multi-scale features to generate the first image feature with a dimension of 256×64×64. The second feature extraction submodule adopts an architecture that combines 3D convolution and attention mechanisms. For dynamically enhanced MRI images, it extracts spatiotemporal features through three 3D convolutional blocks. For ultrasound images, it extracts morphological and hardness features through two 2D convolutional blocks. Then, an adaptive feature alignment layer converts the MRI spatiotemporal features and ultrasound features into a unified resolution and stitches them together to generate a second image feature with a dimension of 384×64×64.

[0007] Furthermore, the dual-channel heterogeneous neural network module also includes: The transfer learning unit is used to pre-train the first feature extraction submodule and the second feature extraction submodule using a labeled multimodal image dataset from a medical center to initialize the network parameters; A few-sample incremental learning unit, connected to the transfer learning unit, is used to fine-tune the parameters of the pre-trained first feature extraction submodule and the second feature extraction submodule based on a small number of newly added image samples with molecular subtyping annotations from the new input system after system deployment.

[0008] Furthermore, the cross-modal self-attention fusion module includes: A feature mapping unit is used to map the first image feature and the second image feature to a shared feature space, respectively. A multi-head self-attention computing unit is used to calculate the interaction attention weights between the first image feature and the second image feature in the shared feature space; The feature weighting and stitching unit is used to weight the first image features and the second image features according to the interactive attention weights, and stitch the weighted features together to generate the cross-modal feature representation.

[0009] Furthermore, the feature mapping unit employs a dual-branch fully connected layer to achieve modality alignment: For the first image feature, after being flattened into a one-dimensional vector, it is input into a fully connected network with two hidden layers and outputs a modality adaptation vector with a dimension of 1024. For the second image feature, after being flattened, it is input into a fully connected network with the same structure and also outputs a 1024-dimensional modality adaptation vector. Together, they constitute the feature representation in the shared feature space. The multi-head self-attention calculation unit is configured with 8 attention heads, each with a dimension of 128. The basic attention weight is obtained by calculating the similarity between the query vector and the key vector. The query vector is a first image feature adaptation vector, and the key vector is a second image feature adaptation vector. Positional encoding is introduced to correct the spatial correlation bias of cross-modal features, and the final interaction attention weights are obtained by normalization through the Softmax function; The feature weighting stitching unit multiplies the first image feature adaptation vector and the second image feature adaptation vector by the corresponding interactive attention weights to obtain the weighted modal features. By performing element-wise addition and channel concatenation operations, a cross-modal feature representation with a dimension of 2048 is generated.

[0010] Furthermore, the explainable AI component includes: The heatmap generation unit is used to calculate and generate a pixel-level heatmap superimposed on the original input image by using a class activation mapping method, based on the feature map and output weights of the last convolutional layer in the molecular subtyping identification module, so as to visualize the image region on which the model decision is based. The feature association analysis unit is used to extract key image feature descriptors that are highly correlated with molecular subtyping from the cross-modal feature representation, and to map the key image feature descriptors to corresponding biological feature descriptions according to the medical knowledge base; The report generation unit is used to integrate the pixel-level heatmap and the biological feature description to generate a structured key image feature report; The interactive interface is used to display the pixel-level heatmap and the key image feature report to doctors, and allows users to click on a specific area on the heatmap to view the detailed feature description of that area and its correlation strength with molecular subtyping.

[0011] Furthermore, it also includes a result confidence assessment module, connected to the molecular subtyping identification module, used to calculate and output the confidence score of the molecular subtyping identification result based on the clarity of the cross-modal feature representation, the consistency between different modal features, and the probability distribution of the output layer of the molecular subtyping identification module.

[0012] Furthermore, the system is deployed as a software service on a cloud server or a local medical imaging workstation, and receives image input and returns identification results and interpretability reports through an application programming interface or graphical user interface.

[0013] Furthermore, the result confidence assessment module has a pre-built multi-dimensional confidence analysis model, which is configured to calculate the confidence score and perform a graded response through the following steps: The predicted probability distribution vector of the target subtype is obtained from the molecular subtyping identification module, and the normalized first image feature adaptation vector and second image feature adaptation vector are obtained from the cross-modal self-attention fusion module. The information entropy index representing decision determinism is calculated based on the predicted probability distribution vector; the feature distance index representing semantic consistency between modalities is calculated based on the first image feature adaptation vector and the second image feature adaptation vector; and the energy aggregation index representing feature saliency is calculated based on the cross-modal feature representation. The confidence score is obtained using a preset nonlinear weighted calculation formula, which is as follows: in: This represents the confidence score, which has a dimension of 1 and a value range of 0 to 1; , , The preset weight allocation coefficients are used, and they satisfy... ; This refers to the information entropy index. , For the predicted probability distribution vector, the first... The probability value of each element. This represents the total number of molecular subtypes. This term is used to transform uncertainty into a normalized measure of certainty; The feature distance index represents the first image feature adaptation vector. Adaptation vector with the second image feature The Euclidean distance between them; As a distance sensitivity adjustment factor, it has the same dimensions as the reciprocal of the eigenvector norm, such that... The term is a dimensionless numerical value; The energy accumulation index represents the cross-modal feature representation. The L2 norm is used to reflect the activation strength of the fusion feature; It is a feature intensity scaling factor, with the same dimensions as the reciprocal of the eigenvector norm; It is a hyperbolic tangent function used to map feature energy to a nonlinear interval between 0 and 1; The confidence score The score is compared with a preset clinical admission threshold. If the score is lower than the threshold, a manual review prompt is triggered, and the result is marked as a low confidence state in the interactive interface to prompt the doctor to pay attention to the inconsistency of features between modalities.

[0014] Furthermore, the interpretability AI component also includes a visual attribution counterfactual verification unit, used to verify the causal accuracy of the pixel-level heatmap, specifically executing the following steps: Based on the pixel-level heatmap output by the heatmap generation unit, a high-response region is extracted using an adaptive threshold segmentation algorithm to generate a key region mask covering the main lesion features. The original input standardized image sequence is subjected to counterfactual perturbation processing using the key region mask. Specifically, Gaussian blur or mean fill operation is performed on the image region covered by the key region mask, while keeping the pixel values ​​of the region outside the mask unchanged, to generate a counterfactual verification sequence. The counterfactual verification sequence is input into the dual-channel heterogeneous neural network module and subsequent modules to obtain the molecular subtyping prediction probability under the counterfactual state. The probability decrease is calculated between the original predicted probability and the molecular subtyping predicted probability under the counterfactual state. If the probability decrease exceeds a preset causal significance threshold, the region indicated by the pixel-level heatmap is determined to have high causal explanatory power and is marked as confirmatory evidence in the key image feature report. If the probability decrease does not exceed the causal significance threshold, the pixel-level heatmap is determined to have a risk of spurious correlation, and the system automatically triggers a feature relocation command.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. The multimodal image integration and standardized preprocessing architecture of this invention, by coordinating mammography, dynamic contrast-enhanced MRI and ultrasound images, and combining preprocessing operations such as unified resolution, grayscale correction and noise reduction, achieves complementary enhancement of diagnostic information and improved data compatibility. It breaks through the limitations of single-modal information, comprehensively captures the anatomical, functional and tissue characteristics of lesions, effectively reduces the risk of misdiagnosis and missed diagnosis, eliminates cross-device and cross-center data heterogeneity, adapts to cloud and local dual deployment scenarios, and seamlessly connects to existing medical systems through API / GUI interfaces, laying the foundation for large-scale clinical application.

[0016] 2. The dual-channel heterogeneous neural network and cross-modal self-attention fusion module of the present invention designs exclusive feature extraction paths for different modal characteristics, combines and extracts accurate features, and then achieves deep fusion through modal alignment and multi-head attention interaction. This ensures the accurate capture of fine-grained features of molybdenum targets, spatiotemporal features of MRI, and morphological and hard features of ultrasound. At the same time, it mines cross-modal collaborative information through nonlinear alignment to generate highly discriminative cross-modal feature representations. Combined with transfer learning and few-sample incremental learning, it greatly improves the model's generalization ability and rare classification identification effect.

[0017] 3. The interpretable AI component and confidence assessment module of this invention, through pixel-level heatmap visualization, image-biological feature mapping, structured report generation, and confidence quantification, enable doctors to clearly understand the basis of model decision-making, enhance the trust in results, and provide functions for interpreting the biological characteristics of lesions and interactive exploration to help doctors deeply understand the nature of lesions. The confidence score provides an objective reference for the reliability of results, avoids blindly relying on AI, and realizes a positive interaction between AI and clinical diagnosis. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the AI ​​system for identifying molecular subtyping of breast cancer according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 The present invention provides the following technical solutions: A multimodal imaging-based AI system for molecular subtyping of breast cancer is deployed as a software service on a cloud server or local medical imaging workstation. It receives image input via an application programming interface (API) or a graphical user interface (GUI) and returns identification results and interpretability reports, including: The image acquisition module is used to acquire mammograms, dynamic contrast-enhanced MRI images, and ultrasound images of the same patient, and to preprocess the acquired multimodal images to form a standardized image sequence. The dual-channel heterogeneous neural network module includes a first feature extraction submodule and a second feature extraction submodule. The first feature extraction submodule is used to extract a first image feature from mammography images, and the second feature extraction submodule is used to extract a second image feature from dynamic contrast-enhanced MRI images and ultrasound images. The cross-modal self-attention fusion module is connected to the dual-channel heterogeneous neural network module. It is used to receive the first image features and the second image features, and to perform non-linear alignment and interaction on the first image features and the second image features through a self-attention mechanism to generate a fused cross-modal feature representation. The molecular subtyping identification module, connected to the cross-modal self-attention fusion module, is used to receive cross-modal feature representations and output the molecular subtyping identification results of breast cancer based on the cross-modal feature representations; An interpretable AI component, connected to the molecular subtyping identification module, is used to generate and output pixel-level heatmaps and key image feature reports based on the decision-making process of the molecular subtyping identification module. The key image feature reports are used to display the image regions and biological features associated with the molecular subtyping. The confidence assessment module, connected to the molecular typing identification module, is used to calculate and output the confidence score of the molecular typing identification results based on the clarity of cross-modal feature representation, the consistency between different modal features, and the probability distribution of the output layer of the molecular typing identification module.

[0021] In the above embodiments, this system integrates three core imaging modalities—mammography, dynamic contrast-enhanced MRI, and ultrasound—and constructs a fully intelligent processing architecture, significantly overcoming the limitations of traditional single-modality imaging diagnosis. The collaborative application of multimodal imaging achieves complementary and enhanced diagnostic information. The high sensitivity of mammography in detecting calcifications, the accurate presentation of tumor hemodynamics and extent by dynamic contrast-enhanced MRI, and the real-time assessment of lesion morphology and hardness by ultrasound, combined with these three features, enable the system to comprehensively capture the anatomical structure, physiological function, and tissue characteristics of breast cancer. This effectively avoids misdiagnosis and missed diagnosis caused by information gaps in single-modality imaging, and significantly improves the ability to differentiate early breast cancer or atypical lesions. The system adopts a software service deployment model, supporting dual-scenario adaptation between cloud and local workstations. This meets the needs of large medical centers for centralized data management and efficient computing, while also adapting to the demands of primary healthcare institutions for localized, low-latency diagnosis. Furthermore, through a dual-interface design of API and GUI, it achieves seamless integration with existing medical imaging systems, lowering the threshold for clinical application and facilitating the rapid implementation and promotion of the technology.

[0022] In the above embodiments, the introduction of the result confidence assessment module provides an objective credibility reference for the identification results by quantifying feature clarity, modal consistency, and probability distribution rationality. This helps doctors judge the reliability of the results, avoid blindly relying on AI decision-making, and further improve the rigor and safety of clinical diagnosis. The entire chain of intelligent processing from image acquisition, processing, analysis to result output is realized, which not only improves the accuracy and efficiency of molecular subtyping identification of breast cancer, but also takes into account clinical practicality and compatibility, providing strong technical support for the application of precision medicine in the field of breast diseases.

[0023] In the dual-channel heterogeneous neural network module, the first feature extraction submodule adopts a Transformer-based multi-scale feature extraction architecture, including 4 encoding blocks and 2 scale fusion layers: Each coding block contains a 64-channel convolutional layer, a LayerNorm layer, a GELU activation function, and a residual connection, used to capture fine-grained features such as calcifications and structural distortions in molybdenum images. The scale fusion layer maps the feature maps output by different coding blocks to the same channel dimension through 1×1 convolution, and uses attention pooling to weight and aggregate the multi-scale features to generate the first image feature with a dimension of 256×64×64. The second feature extraction submodule adopts an architecture that combines 3D convolution and attention mechanisms. For dynamically enhanced MRI images, it extracts spatiotemporal features through three 3D convolutional blocks. For ultrasound images, it extracts morphological and hardness features through two 2D convolutional blocks. Then, an adaptive feature alignment layer converts the MRI spatiotemporal features and ultrasound features into a unified resolution and stitches them together to generate a second image feature with a dimension of 384×64×64. The dual-channel heterogeneous neural network module also includes: The transfer learning unit is used to pre-train the first feature extraction submodule and the second feature extraction submodule using a labeled multimodal image dataset from a medical center to initialize the network parameters; The few-sample incremental learning unit, connected to the transfer learning unit, is used to fine-tune the parameters of the pre-trained first and second feature extraction submodules based on a small number of newly added image samples with molecular subtyping annotations after system deployment, in order to improve the system's ability to identify rare breast cancer molecular subtypes.

[0024] In the above embodiments, the refined design of the dual-channel heterogeneous neural network module constructs dedicated feature extraction paths for the characteristics of different modalities of images. At the same time, it incorporates transfer learning and few-shot incremental learning mechanisms. For the first feature extraction submodule, the Transformer-based multi-scale feature extraction architecture uses a combination of convolutional layers, LayerNorm layers, GELU activation functions, and residual connections to ensure accurate capture of fine-grained features such as calcification points and structural distortions in molybdenum images. The residual connections alleviate the gradient vanishing problem in deep network training and improve the stability of feature extraction. The two scale fusion layers achieve channel unification of multi-resolution features through 1×1 convolutions. Then, attention pooling is used to weight and aggregate key scale features, effectively strengthening the representation of important diagnostic information. The final generated 256×64×64-dimensional first image features ensure feature richness, control data dimensionality, and reduce the burden on subsequent fusion processing.

[0025] In the above embodiments, for the second feature extraction submodule, 3D convolutional blocks are used to extract temporal and spatial joint features for the spatiotemporal characteristics of dynamically enhanced MRI, and 2D convolutional blocks are used to accurately capture the morphological and hardness information of ultrasound. Then, an adaptive feature alignment layer is used to unify the resolution and stitch together the two heterogeneous features, generating a 384×64×64-dimensional second image feature that fully preserves the modality-specific information of MRI and ultrasound, achieving effective integration of multi-source features. The transfer learning unit uses a large-scale labeled medical dataset for pre-training, enabling the network parameters to have good feature extraction capabilities in the initial stage, significantly shortening the model training cycle, and improving the model's generalization performance in small sample scenarios. The few-sample incremental learning unit addresses the pain point of insufficient rare molecular subtype samples in clinical practice. By fine-tuning the parameters of a small number of newly added labeled samples after deployment, the model can continuously learn new feature patterns, continuously improve its ability to identify rare subtypes, achieve dynamic optimization and long-term iteration of model performance, take into account the characteristic differences of different modalities, and improve the practicality and adaptability of the model through advanced learning mechanisms.

[0026] The cross-modal self-attention fusion module includes: The feature mapping unit is used to map the first image features and the second image features to a shared feature space, respectively. A multi-head self-attention computation unit is used to compute the interaction attention weights between the first image feature and the second image feature in a shared feature space. The feature weighting and stitching unit is used to weight the first image features and the second image features according to the interactive attention weights, and then stitch the weighted features together to generate a cross-modal feature representation; The feature mapping unit uses a dual-branch fully connected layer to achieve modality alignment. For the first image feature, after being flattened into a one-dimensional vector, it is input into a fully connected network with two hidden layers and outputs a modality adaptation vector with a dimension of 1024. For the second image feature, after being flattened, it is input into a fully connected network with the same structure and also outputs a 1024-dimensional modality adaptation vector. Together, they constitute the feature representation in the shared feature space. The multi-head self-attention computation unit is set with 8 attention heads, each with a dimension of 128. The basic attention weight is obtained by calculating the similarity between the query vector and the key vector. The query vector is the first image feature adaptation vector, and the key vector is the second image feature adaptation vector. Positional encoding is introduced to correct the spatial correlation bias of cross-modal features, and the final interaction attention weights are obtained by normalization through the Softmax function; The feature weighting stitching unit multiplies the first image feature adaptation vector and the second image feature adaptation vector by the corresponding interactive attention weights to obtain the weighted modal features. By performing element-wise addition and channel concatenation operations, a cross-modal feature representation with a dimension of 2048 is generated.

[0027] In the above embodiments, the cross-modal self-attention fusion module successfully solves the core problems of heterogeneity and alignment difficulties of multimodal image features through a phased and refined feature processing flow, achieving deep fusion and efficient utilization of different modal features. The feature mapping unit adopts a dual-branch fully connected layer design, providing an efficient solution for cross-modal feature alignment. Addressing the dimensional differences between the first and second image features, after flattening them into one-dimensional vectors, they are input into a fully connected network with the same structure, ultimately outputting 1024-dimensional modality adaptation vectors, achieving dimensional unification of different modal features. Through the nonlinear transformation of the fully connected layer, heterogeneous features are mapped to a shared feature space, eliminating modality differences. The distribution differences between modalities are addressed by the use of eight attention heads in the multi-head self-attention computing unit to achieve multi-dimensional and multi-perspective feature interaction. Each 128-dimensional attention head can focus on different feature association patterns. By calculating the similarity between the first image feature adaptation vector and the second image feature adaptation vector, the potential association between modalities is accurately captured. The introduction of position encoding effectively corrects the spatial association bias of cross-modal features, solves the problem of unclear correspondence between lesion locations in different modal images, and makes the calculation of attention weights more in line with clinical diagnostic logic. The final interactive attention weights, after being normalized by the Softmax function, can objectively reflect the importance of different modal features for molecular subtyping identification.

[0028] In the above embodiments, the feature weighted concatenation unit enhances important features and suppresses secondary features by multiplying the two modality adaptation vectors with their corresponding interactive attention weights. Through element-level addition and channel concatenation operations, it generates a 2048-dimensional cross-modal feature representation, retaining the core information of each modality. Feature interaction fusion produces more discriminative joint features, significantly improving the feature representation capability. Compared with traditional simple concatenation and weighted summation methods, this approach achieves nonlinear deep interaction of cross-modal features through modality alignment, correlation calculation, and precise fusion. This effectively mines complementary information and synergistic effects between different modalities, enabling the generated cross-modal feature representation to more comprehensively and accurately reflect the molecular subtyping characteristics of breast cancer. Simultaneously, the 2048-dimensional feature dimension ensures information integrity, avoids computational efficiency degradation caused by dimensionality explosion, and balances model performance and computational speed.

[0029] Explainable AI components include: The heatmap generation unit is used to calculate and generate a pixel-level heatmap superimposed on the original input image by using the class activation mapping method, based on the feature map and output weights of the last convolutional layer in the molecular subtyping identification module, so as to visualize the image region on which the model decision is based. The feature association analysis unit is used to extract key image feature descriptors that are highly correlated with molecular subtyping from cross-modal feature representations, and to map the key image feature descriptors to corresponding biological feature descriptions based on the medical knowledge base; The report generation unit integrates pixel-level heatmaps and biological feature descriptions to generate structured reports on key image features. The interactive interface is used to display pixel-level heatmaps and key image feature reports to doctors, and allows users to click on specific areas on the heatmap to view detailed feature descriptions of that area and the strength of its correlation with molecular subtyping.

[0030] In the above embodiments, the refined design of the interpretable AI component breaks down the barriers of black-box decision-making in traditional AI models. Through visualization and structured reporting, it achieves transparency and traceability in the AI ​​decision-making process, providing strong support for clinicians. This component not only solves the interpretability problem of AI models, but also achieves deep integration of AI technology and clinical diagnostic processes through multi-dimensional result presentation and interactive design. This improves the accuracy and efficiency of diagnosis, ensures the rigor of the diagnostic process, and promotes the standardized application of AI in the medical field.

[0031] In a preferred embodiment of the present invention, to address the potential blind risks during the molecular subtyping of breast cancer, a multi-dimensional confidence analysis model is pre-built into the result confidence assessment module. This model does not rely solely on the probability distribution of the neural network output layer, but rather quantifies the reliability of the diagnostic results from three orthogonal physical dimensions—decision certainty, modality consistency, and feature saliency—by tracing back the entire process of feature extraction and fusion.

[0032] Specifically, the multidimensional confidence analysis model is configured to calculate the confidence score and perform a graded response through the following steps: Synchronous acquisition and standardized preprocessing of multi-source heterogeneous decision feature data.

[0033] After the molecular subtyping identification module completes the forward inference operation on the current patient's mammogram, dynamic contrast-enhanced MRI and ultrasound images, the result confidence assessment module first extracts key intermediate feature data from different network levels through the system's internal high-speed data bus.

[0034] Specifically, the system first obtains the predicted probability distribution vector of the target subtype from the output layer of the molecular subtyping identification module, denoted as... It is understandable that this vector It is a length of A one-dimensional tensor, in which Corresponding to the total number of breast cancer molecular subtypes preset by the system (in this embodiment, (corresponding to Luminal A, Luminal B, HER2 overexpression, and triple-negative types, respectively), each element in the vector... The precise representation model considers the posterior probability value of the current case belonging to the corresponding subtype, and satisfies the normalization condition. .

[0035] Simultaneously, the system traces back to the output of the feature mapping unit of the cross-modal self-attention fusion module to obtain the original feature vector before entering the attention mechanism, i.e., the first image feature adaptation vector after normalization. and the second image feature adaptation vector .in, It is a high-dimensional vector (dimension 1024) originating from the mammography branch and mapped to the shared feature space, mainly representing anatomical features such as microcalcifications, structural distortions, and edge spiculations; It is a high-dimensional vector (dimension 1024) derived from MRI and ultrasound branches, mainly characterizing functional features such as hemodynamic curves, tissue stiffness, and echo attenuation.

[0036] In addition, the system also needs to obtain the cross-modal feature representation generated by the final fusion. (With a dimension of 2048), this vector carries all the effective semantic information for the model to make the final classification decision.

[0037] To eliminate the inconsistency in feature vector magnitude caused by differences in the original signal intensity of different imaging devices (e.g., MRI signal values ​​are usually much higher than mammogram target gray values), the system acquires... , and Then, L2 norm normalization is immediately performed on it. Specifically, for any vector... Its normalized form This preprocessing step projects all feature vectors onto the same unit hypersphere, ensuring that subsequent Euclidean distance-based difference metrics purely reflect the differences in the semantic direction of the features, while eliminating interference from signal strength.

[0038] The information entropy index, which represents the determinism of decision-making, is calculated based on the principle of information entropy.

[0039] After acquiring the basic data, the system first assesses the model's "hesitation level" from the perspective of the probability distribution's morphology. According to Shannon's information theory, entropy is the core physical quantity for measuring the disorder of a system.

[0040] Specifically, the system uses formulas Calculate the information entropy of the predicted probability distribution vector. In actual computer numerical calculations, to prevent [the information entropy] due to a certain [factor], [this process is repeated]. Setting the value to 0 will cause logarithmic overflow, and the system will add a very small smoothing constant to each probability value. (For example ).

[0041] This indicator has clear clinical significance: if the model has extremely high confidence in a certain subtype (such as "triple negative"), its probability distribution will show a steep single peak, and the calculated information entropy will be... The probability distribution approaches 0; conversely, if the model fluctuates between multiple subtypes (for example, the probabilities of "Luminal B" and "HER2" are extremely close), the probability distribution will tend to flatten, at which point the information entropy will decrease. It will increase significantly.

[0042] To transform this measure of uncertainty into a deterministic indicator that positively contributes to confidence, the system constructs a normalization term. .in, It represents The maximum entropy value for classification problems (i.e., the entropy when guessing randomly). Through this normalization process, the value of this indicator is strictly limited to between 0 and 1. When the model makes the most decisive decision, this value is 1; when the model makes the most chaotic decision, this value is 0. This design provides a benchmark value that reflects the model's own state for the final confidence score.

[0043] The feature distance index, which represents the semantic consistency between modalities, is calculated based on the geometric distance in the feature space.

[0044] This step aims to address common issues of conflicting diagnostic evidence in multimodal data. In clinical practice, challenging cases frequently arise where mammography shows malignant features while ultrasound shows benign features.

[0045] Specifically, the system calculates the normalized first image feature adaptation vector. Adaptation vector with second image features The Euclidean distance between them is denoted as . This distance value directly reflects the degree of difference between different modalities in the shared semantic space. The larger the distance, the more severe the feature conflict between modalities, and the less reliable the diagnostic results.

[0046] To map this physical distance into a dimensionless confidence penalty term, the system introduces a distance sensitivity adjustment factor. This factor This is a key hyperparameter, whose dimension is the reciprocal of the eigenvector norm, used to eliminate the influence of dimensions. In specific implementations, The value is set based on the reciprocal of the average feature distance between positive and negative samples in a large-scale historical case training set.

[0047] The system constructs an inverse proportional function term. Understandably, the monotonically decreasing nature of this function means that when modal features are highly consistent (distance approaches 0), this term approaches 1, having no impact on confidence; however, when modal features are severely conflicting (distance increases), the denominator increases rapidly, and the value of this term rapidly decays towards 0, thus significantly lowering the final confidence score. This nonlinear inhibition mechanism effectively simulates the cautious attitude of human doctors when faced with inconsistent examination results.

[0048] Energy aggregation indexes that characterize the significance of features are calculated based on vector norms and saturation functions.

[0049] Besides probability distribution and modality consistency, the amount of effective information in the input image itself is also a crucial factor determining diagnostic reliability. In deep neural networks, the L2 norm of the feature vector often corresponds to the activation intensity of neurons.

[0050] Specifically, the system computes cross-modal feature representations. L2 norm And introduce the hyperbolic tangent function. and feature intensity scaling factor Through calculation The system maps unbounded feature energy to a nonlinear interval between 0 and 1.

[0051] Here, the feature intensity scaling factor The setting principle is to ensure that the characteristic norm of typical confirmed cases is within the range of The saturation region of a function. This is understandable. The S-shaped curve characteristic of the function makes this index highly sensitive to low-energy features (in the linear region) and exhibits a saturation effect on high-energy features (in the flat region). This means that as long as the lesion features are sufficiently significant (such as typical spiculations or strong enhancement), regardless of their specific intensity, they can achieve a full confidence score contribution; however, if the features are weak (such as blurry shadows), the confidence score will be severely penalized. This mechanism effectively prevents the model from generating false high confidence scores for empty or low-quality images.

[0052] Nonlinear weighted fusion and confidence score generation.

[0053] After independently calculating the normalized indices for the three dimensions mentioned above, the system uses a preset non-linear weighted calculation formula to derive the final confidence score. The formula is:

[0054] in, , , The preset weight allocation coefficients are used, and they satisfy... .

[0055] In a preferred embodiment, considering that the final classification probability is the most direct basis for clinical decision-making, it is usually assigned... (Decision certainty) has the highest weight (e.g., 0.4); considering that modal conflict is a key risk point leading to serious misdiagnosis, it is assigned... (Modal consistency) Medium weight (e.g., 0.4); assigned (Feature significance) Base weight (e.g., 0.2). It is worth noting that this system supports an online learning mechanism to adaptively fine-tune the above weight coefficients based on the data distribution characteristics of different medical centers, thereby continuously optimizing the accuracy of the scoring.

[0056] Tiered response and manual review triggering.

[0057] The system will calculate the confidence score. It compares the results in real time with a preset clinical admission threshold (e.g., 0.75) and executes a tiered response strategy.

[0058] If rating If the score is below the threshold, the system determines that the diagnostic result is high-risk and immediately triggers a manual review prompt. Specifically, in the doctor's interface, the result will be marked as low-confidence (e.g., a yellow warning box will be displayed). Furthermore, the system will generate specific explanatory prompts based on the sub-item with the most deducted points. For example, if the modality consistency score is low, the system will prompt: "Note: There is a significant semantic conflict between mammographic calcification features and MRI enhancement features; a thorough review is recommended." To endow the system with true interpretability and verify the genuine causal relationships of the regions indicated by the heatmap, this invention integrates a visual attribution counterfactual verification unit into the interpretability AI component. This unit works by utilizing the counterfactual concept in causal inference, dynamically constructing counterfactual samples during the inference phase to verify whether heatmap regions truly dominate the model's decisions.

[0059] Specifically, the visual attribution counterfactual verification unit is configured to verify the causal accuracy of the pixel-level heatmap through the following steps: High-response region extraction and key region mask generation based on heatmap.

[0060] First, the system receives a pixel-level heatmap output by the heatmap generation unit (usually using Grad-CAM++ or the Attention Rollout algorithm). This heatmap is a two-dimensional matrix with the same resolution as the original image, and the numerical value represents the level of attention the model pays to the corresponding pixel.

[0061] To accurately pinpoint the core regions of interest to the model, the system uses an adaptive threshold segmentation algorithm (such as Otsu's method) to process the heatmap. Specifically, the system automatically calculates the optimal segmentation threshold, marking pixels in the heatmap above the threshold as 1 (foreground) and those below the threshold as 0 (background), thereby generating a mask of key regions that cover the main lesion features.

[0062] To ensure the integrity of the mask and prevent lesion edges from being truncated, the system further performs a morphological dilation operation on the binarized mask. This is typically done using... The structural elements are iterated one to two times to ensure that the mask can completely cover the suspected lesion area and its surrounding microenvironment, which is especially important for malignant tumors containing invasive information.

[0063] Construction and generation of counterfactual perturbation sequences.

[0064] Subsequently, the system uses the key region mask to perform counterfactual perturbation processing on the original input standardized image sequence. Counterfactual means constructing a virtual image state with missing lesion features.

[0065] Specifically, the system performs Gaussian blur or mean fill operations on the image area covered by the mask (i.e., the key lesion area). For example, using a large kernel size (such as... A Gaussian filter is used to strongly smooth the area, causing it to lose its texture features but retaining the basic brightness information; at the same time, the pixel values ​​of the area outside the mask remain completely unchanged.

[0066] Through this operation, the system generates a counterfactual verification sequence. This sequence represents the following: what would the image look like if the patient's breast tissue did not contain this specific lesion feature (such as spiculation or calcification)? Compared to simply blacking out the image, this process avoids introducing artificially created strong edge artifacts and is more consistent with medical logic.

[0067] Step S1003: Re-inference under counterfactual conditions and calculation of probability drop.

[0068] The system then inputs the generated counterfactual verification sequence back into the dual-channel heterogeneous neural network module and subsequent modules to perform complete forward inference.

[0069] It is important to note that the network's weight parameters remain frozen at this point. The system obtains the molecular subtyping prediction probability under the counterfactual state and compares it with the original prediction probability, calculating the probability decrease magnitude.

[0070] Understandably, if the area indicated by the heatmap is indeed the decisive factor that leads the model to make a judgment (i.e., it has causal power), then when that area is blurred, the model's prediction probability of the original target classification should drop sharply; conversely, if the probability remains almost unchanged, it means that the model is making decisions mainly based on background information or image artifacts.

[0071] Causal significance determination and closed-loop feedback.

[0072] The system compares the calculated probability decrease magnitude with a preset causal significance threshold (e.g., 0.4).

[0073] If the probability decrease exceeds the threshold, the system determines that the region indicated by the pixel-level heatmap has high causal explanatory power. At this time, in the generated key image feature report, the region is marked as confirmatory evidence and highlighted to confirm to the doctor: "The model made its judgment based on the features here, and the basis is reliable." If the probability decrease does not exceed the threshold, the system determines that the pixel-level heatmap has a risk of spurious correlation. This usually means that the model is incorrectly focusing on non-lesion areas (such as text or artifacts at the edges of film). At this time, the system automatically triggers a feature relocation instruction, starts a secondary attention mining mechanism, suppresses the gradient response of the current region, searches for the region in the image that contributes the second largest to classification, and executes the verification process again until a region with true causal power is found or an anomaly is alerted in the report.

[0074] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An AI system for molecular subtyping identification of breast cancer based on multimodal imaging, characterized in that, include: The image acquisition module is used to acquire mammograms, dynamic contrast-enhanced MRI images, and ultrasound images of the same patient, and to preprocess the acquired multimodal images to form a standardized image sequence. The dual-channel heterogeneous neural network module includes a first feature extraction submodule and a second feature extraction submodule. The first feature extraction submodule is used to extract a first image feature from mammography images, and the second feature extraction submodule is used to extract a second image feature from dynamic contrast-enhanced MRI images and ultrasound images. A cross-modal self-attention fusion module, connected to the dual-channel heterogeneous neural network module, is used to receive the first image features and the second image features, and to perform nonlinear alignment and interaction on the first image features and the second image features through a self-attention mechanism to generate a fused cross-modal feature representation; The cross-modal self-attention fusion module includes: A feature mapping unit is used to map the first image feature and the second image feature to a shared feature space, respectively. A multi-head self-attention computing unit is used to calculate the interaction attention weights between the first image feature and the second image feature in the shared feature space; The feature weighting and stitching unit is used to weight the first image features and the second image features according to the interactive attention weights, and stitch the weighted features together to generate the cross-modal feature representation; The feature mapping unit uses a dual-branch fully connected layer to achieve modality alignment: For the first image feature, after being flattened into a one-dimensional vector, it is input into a fully connected network with two hidden layers and outputs a modality adaptation vector with a dimension of 1024. For the second image feature, after being flattened, it is input into a fully connected network with the same structure and also outputs a 1024-dimensional modality adaptation vector. Together, they constitute the feature representation in the shared feature space. The multi-head self-attention calculation unit is configured with 8 attention heads, each with a dimension of 128. The basic attention weight is obtained by calculating the similarity between the query vector and the key vector. The query vector is a first image feature adaptation vector, and the key vector is a second image feature adaptation vector. Positional encoding is introduced to correct the spatial correlation bias of cross-modal features, and the final interaction attention weights are obtained by normalization through the Softmax function; The feature weighting stitching unit multiplies the first image feature adaptation vector and the second image feature adaptation vector by the corresponding interactive attention weights to obtain the weighted modal features. By performing element-wise addition and channel concatenation operations, a cross-modal feature representation with a dimension of 2048 is generated; The molecular typing identification module, connected to the cross-modal self-attention fusion module, is used to receive the cross-modal feature representation and output the molecular typing identification result of breast cancer based on the cross-modal feature representation; An interpretable AI component, connected to the molecular typing identification module, is used to generate and output pixel-level heatmaps and key image feature reports based on the decision-making process of the molecular typing identification module. The key image feature reports are used to display image regions and biological features associated with molecular typing.

2. The AI ​​system for molecular subtyping of breast cancer based on multimodal imaging as described in claim 1, characterized in that, In the dual-channel heterogeneous neural network module, the first feature extraction submodule adopts a Transformer-based multi-scale feature extraction architecture, including 4 encoding blocks and 2 scale fusion layers: Each coding block contains a 64-channel convolutional layer, a LayerNorm layer, a GELU activation function, and a residual connection, used to capture fine-grained features such as calcifications and structural distortions in molybdenum images. The scale fusion layer maps the feature maps output by different coding blocks to the same channel dimension through 1×1 convolution, and uses attention pooling to weight and aggregate the multi-scale features to generate the first image feature with a dimension of 256×64×64. The second feature extraction submodule adopts an architecture that combines 3D convolution and attention mechanisms. For dynamically enhanced MRI images, it extracts spatiotemporal features through three 3D convolutional blocks. For ultrasound images, it extracts morphological and hardness features through two 2D convolutional blocks. Then, an adaptive feature alignment layer converts the MRI spatiotemporal features and ultrasound features into a unified resolution and stitches them together to generate a second image feature with a dimension of 384×64×64.

3. The AI ​​system for molecular subtyping of breast cancer based on multimodal imaging as described in claim 2, characterized in that, The dual-channel heterogeneous neural network module also includes: The transfer learning unit is used to pre-train the first feature extraction submodule and the second feature extraction submodule using a labeled multimodal image dataset from a medical center to initialize the network parameters; A few-sample incremental learning unit, connected to the transfer learning unit, is used to fine-tune the parameters of the pre-trained first feature extraction submodule and the second feature extraction submodule based on a small number of newly added image samples with molecular subtyping annotations from the new input system after system deployment.

4. The AI ​​system for molecular subtyping of breast cancer based on multimodal imaging as described in claim 1, characterized in that, The explainable AI components include: The heatmap generation unit is used to calculate and generate a pixel-level heatmap superimposed on the original input image by using a class activation mapping method, based on the feature map and output weights of the last convolutional layer in the molecular subtyping identification module, so as to visualize the image region on which the model decision is based. The feature association analysis unit is used to extract key image feature descriptors that are highly correlated with molecular subtyping from the cross-modal feature representation, and to map the key image feature descriptors to corresponding biological feature descriptions according to the medical knowledge base; The report generation unit is used to integrate the pixel-level heatmap and the biological feature description to generate a structured key image feature report; The interactive interface is used to display the pixel-level heatmap and the key image feature report to doctors, and allows users to click on a specific area on the heatmap to view the detailed feature description of that area and its correlation strength with molecular subtyping.

5. The AI ​​system for molecular subtyping of breast cancer based on multimodal imaging as described in claim 1, characterized in that, It also includes a result confidence assessment module, which is connected to the molecular typing identification module. This module is used to calculate and output the confidence score of the molecular typing identification result based on the clarity of the cross-modal feature representation, the consistency between different modal features, and the probability distribution of the output layer of the molecular typing identification module.

6. The AI ​​system for molecular subtyping of breast cancer based on multimodal imaging as described in claim 1, characterized in that, Deployed as a software service on cloud servers or local medical imaging workstations, it receives image input and returns identification results and interpretable reports via application programming interfaces or graphical user interfaces.

7. The AI ​​system for molecular subtyping of breast cancer based on multimodal imaging as described in claim 5, characterized in that, The result confidence assessment module has a pre-built multi-dimensional confidence analysis model, which is configured to calculate the confidence score and perform a graded response through the following steps: The predicted probability distribution vector of the target subtype is obtained from the molecular subtyping identification module, and the normalized first image feature adaptation vector and second image feature adaptation vector are obtained from the cross-modal self-attention fusion module. The information entropy index representing decision determinism is calculated based on the predicted probability distribution vector; the feature distance index representing semantic consistency between modalities is calculated based on the first image feature adaptation vector and the second image feature adaptation vector; and the energy aggregation index representing feature saliency is calculated based on the cross-modal feature representation. The confidence score is obtained using a preset nonlinear weighted calculation formula, which is as follows: in: This represents the confidence score, which has a dimension of 1 and a value range of 0 to 1; , , The preset weight allocation coefficients are used, and they satisfy... ; This refers to the information entropy index. , For the predicted probability distribution vector, the first... The probability value of each element. This represents the total number of molecular subtypes. This term is used to transform uncertainty into a normalized measure of certainty; The feature distance index represents the first image feature adaptation vector. Adaptation vector with the second image feature The Euclidean distance between them; As a distance sensitivity adjustment factor, it has the same dimensions as the reciprocal of the eigenvector norm, such that... The term is a dimensionless numerical value; The energy accumulation index represents the cross-modal feature representation. The L2 norm is used to reflect the activation strength of the fusion feature; It is a feature intensity scaling factor, with the same dimensions as the reciprocal of the eigenvector norm; It is a hyperbolic tangent function used to map feature energy to a nonlinear interval between 0 and 1; The confidence score The score is compared with a preset clinical admission threshold. If the score is lower than the threshold, a manual review prompt is triggered, and the result is marked as a low confidence state in the interactive interface to prompt the doctor to pay attention to the inconsistency of features between modalities.

8. The AI ​​system for molecular subtyping of breast cancer based on multimodal imaging as described in claim 4, characterized in that, The interpretable AI component also includes a visual attribution counterfactual verification unit, used to verify the causal accuracy of the pixel-level heatmap, specifically including the following steps: Based on the pixel-level heatmap output by the heatmap generation unit, a high-response region is extracted using an adaptive threshold segmentation algorithm to generate a key region mask covering the main lesion features. The original input standardized image sequence is subjected to counterfactual perturbation processing using the key region mask. Specifically, Gaussian blur or mean fill operation is performed on the image region covered by the key region mask, while keeping the pixel values ​​of the region outside the mask unchanged, to generate a counterfactual verification sequence. The counterfactual verification sequence is input into the dual-channel heterogeneous neural network module and subsequent modules to obtain the molecular subtyping prediction probability under the counterfactual state. The probability decrease is calculated between the original predicted probability and the molecular subtyping predicted probability under the counterfactual state. If the probability decrease exceeds a preset causal significance threshold, the region indicated by the pixel-level heatmap is determined to have high causal explanatory power and is marked as confirmatory evidence in the key image feature report. If the probability decrease does not exceed the causal significance threshold, the pixel-level heatmap is determined to have a risk of spurious correlation, and the system automatically triggers a feature relocation command.

Images