Breast cancer diagnosis method based on bidirectional semantic coupling and cross-modal attention fusion

Abstract

The application discloses a breast cancer diagnosis method based on bidirectional semantic coupling and cross-modal attention fusion, and comprises the following steps: collecting multi-modal data and performing data preprocessing; then, image coding is performed on breast ultrasound images by adopting a ViT architecture; for clinical text data, text semantic coding is performed by adopting a medical field pre-training MedBERT model; thirdly, double-strategy regularization calibration of dropout random inactivation and elastic network sparse constraint is performed on image and text features respectively; fourthly, a two-level hybrid attention mechanism of intra-modal self-attention purification and inter-modal bidirectional cross-attention coupling is adopted to realize fine-grained bidirectional alignment and deep semantic coupling of visual feature space and text semantic space, and multi-modal fusion features are generated; fifthly, an image single-modal, text single-modal and multi-modal coupling three-branch diagnosis structure is constructed; and finally, the classification probabilities output by the three branches are fused by adopting an adaptive weighted voting mechanism to generate a final diagnosis result.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis technology, and in particular to a breast cancer diagnosis method based on bidirectional semantic coupling and cross-modal attention fusion. Background Technology

[0002] Currently, intelligent assisted diagnostic technology for breast cancer has been gradually applied in clinical screening and diagnosis scenarios. However, existing solutions still have multi-dimensional application shortcomings in real-world diagnosis and treatment, making it difficult to adapt to the actual clinical needs of breast disease diagnosis. Its main shortcomings are: 1. Single-modal feature extraction lacks clinical adaptability, making it difficult to provide a reliable feature base for accurate diagnosis. Breast ultrasound image feature extraction largely relies on traditional convolutional neural networks, which, limited by local receptive field mechanisms, cannot effectively model the long-distance spatial dependence of lesion edges, internal echoes, and surrounding glandular tissue. Some solutions using the general Transformer architecture are not optimized for breast lesion characteristics, exhibiting weak capture capabilities for small lesions against a dense breast background. Clinical text feature extraction often employs general pre-trained language models, lacking deep adaptation to breast-specific medical corpora, resulting in high feature redundancy and low weighting of key tests and medical history information, failing to provide high-quality semantic support for multimodal fusion.

[0003] 2. The problem of ineffective alignment of heterogeneous medical features is prominent, easily introducing semantic noise that dilutes diagnostic value. Most existing cross-modal fusion schemes remain at the shallow fusion stage of feature splicing and simple weighting; a few schemes that introduce attention mechanisms adopt a general unidirectional or bidirectional architecture, without constructing specific alignment constraints for the medical semantics of breast diagnosis, which easily leads to meaningless cross-modal associations, diluting the weight of core diagnostic features, failing to establish a bidirectional closed-loop coupling between the image visual space and the text semantic space, and making it difficult to effectively resolve the semantic gap between modalities.

[0004] 3. Insufficient cross-institutional generalization and adaptability to complex scenarios, resulting in significant bottlenecks in clinical implementation. Existing models mostly employ a single regularization strategy, which can only address overfitting issues under similarly distributed data. They cannot adapt to data heterogeneity caused by different medical institutions' equipment parameters and scanning methods, leading to a significant decrease in diagnostic accuracy during cross-center testing. Furthermore, existing multimodal models generally require complete bimodal data input and lack compatibility mechanisms for common clinical scenarios such as missing single modalities or incomplete fields. Incomplete data significantly reduces diagnostic reliability, making it difficult to adapt to the complex and diverse data conditions in outpatient settings. In addition, existing fusion models are mostly black-box structures, lacking clinical interpretability, making it difficult to gain physician trust and truly integrate into the diagnostic and treatment process. Summary of the Invention

[0005] The purpose of this invention is to address the aforementioned problems by providing a breast cancer diagnosis method based on bidirectional semantic coupling and cross-modal attention fusion. This method establishes a complete technical process encompassing multimodal data construction, data preprocessing, image and text feature extraction, cross-modal feature enhancement, cross-modal attention fusion, joint optimization training, and intelligent diagnostic output.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A breast cancer diagnostic method based on bidirectional semantic coupling and cross-modal attention fusion includes the following steps: S1. Collect multimodal data to form a multimodal breast cancer dataset and perform data preprocessing on the multimodal breast cancer dataset; S2. Image encoding of breast ultrasound images is performed using the ViT architecture; for clinical text data, text semantic encoding is performed using a pre-trained MedBERT model in the medical field. S3. Perform dual-strategy regularization calibration on image and text features respectively using Dropout random deactivation and elastic network sparsity constraints; S4. A two-level hybrid attention mechanism of "intramodal self-attention purification + intermodal bidirectional cross-attention coupling" is adopted to achieve fine-grained bidirectional alignment and deep semantic coupling between visual feature space and text semantic space, and generate multimodal fusion features. S5. Construct a three-branch diagnostic structure that couples image monomodality, text monomodality, and multimodality; and generate the final diagnostic result by fusing the classification probabilities of the three branches through an adaptive weighted voting mechanism.

[0007] As an improvement to the above technical solution, in step S1, the multimodal data includes breast ultrasound imaging data and patient clinical text data; the patient clinical text data includes medical history, laboratory indicators, and demographic information; the data preprocessing is to perform uniform standardized preprocessing on the dataset.

[0008] As an improvement to the above technical solution, the specific steps of step S2 are as follows: The S201.ViT architecture encodes global features of breast ultrasound images. It divides preprocessed breast ultrasound images into fixed-size, non-overlapping image blocks, which are then linearly projected and embedded into learnable locations before being input into a multi-layer Transformer encoder. Through a self-attention mechanism, it models the long-distance spatial dependency between lesions and surrounding glandular tissue, accurately capturing core visual features such as lesion morphology, texture, and calcification distribution. The mathematical expression is: ; in, This is a classification label vector used to aggregate global feature information; For the first One ultrasound image block; The linear embedding weight matrix; This represents the total number of image patches; It is a learnable location encoding vector used to preserve the spatial location characteristics of the image; The S202.Transformer encoder comprehensively mines global semantic information of images by parallel computing feature associations across multiple subspaces. Its self-attention calculation formula is: ; The expression for multi-head self-attention is: ; in, These are the query, key, and value feature vectors, respectively. The dimension of the key vector; For the number of attention heads; This is used to output the projection weight matrix. After ViT encoding, the basic visual features of the breast ultrasound image are output. ; S203. A pre-trained MedBERT model in the medical field is used to perform deep semantic encoding of clinical texts. The pre-processed clinical texts undergo professional word segmentation and semantic normalization, constructing a fusion feature representation of word embeddings, positional embeddings, and segment embeddings. A multi-layer Transformer encoder is then used to deeply capture the core medical contextual semantic relationships in the clinical texts, such as medical history, symptoms, and laboratory indicators, generating globally relevant text features with professional discriminative capabilities. .

[0009] As an improvement to the above technical solution, the specific steps of step S3 are as follows: S301. Perform a dual-strategy combined calibration of Dropout random deactivation regularization and elastic network sparse constraint regularization on the features output by the bimodal encoding; the calculation formula for Dropout regularization is: ; in, Input features are either ultrasound or text-based single-modal features; Preserve the probability for the neuron; This is an element-wise multiplication operation; A binary mask generated for a Bernoulli distribution; The loss function for regularization of elastic networks is calculated using the following formula: ; in, For feature reconstruction loss; L1 regularization coefficient, The L2 regularization coefficient; L1 norm, It is an L2 norm.

[0010] S302. The features optimized by Dropout random deactivation regularization and elastic network regularization are concatenated dimensionally, and then mapped through a linear layer and activation function to obtain single-modality calibration features. ; in, , The weight matrix is ​​a learnable weight matrix; , For bias terms; This is a feature dimension concatenation operation; These are the image features after regularization enhancement. These are text features enhanced by regularization. As an improvement to the above technical solution, the specific steps of step S4 are as follows: S401. Perform multi-head self-attention computation on the regularized and calibrated breast ultrasound features and clinical text features respectively, focusing on the core discriminative features within each modality, strengthening the feature correlation between local and global features within the modality, suppressing irrelevant noise interference, improving the task relevance of single-modal features, providing high-purity feature input for subsequent cross-modal coupling, and outputting purified single-modal features. , ; S402. Construct a cross-modal semantic affinity matrix to quantify the fine-grained semantic association between text semantic tokens and ultrasound image patches, providing a semantic weight basis for subsequent bidirectional coupling and achieving precise alignment based on semantic similarity; the semantic metric calculation formula is as follows: ; in, For cross-modal semantic affinity matrix, The length of the text sequence. The length of the image sequence; , These are the dimensions of text and image features, respectively; S403. Based on the semantic affinity matrix, construct two sets of dedicated cross-modal projection matrices to realize bidirectional spatial mapping of text semantic guidance visual feature calibration and visual feature guidance text semantic enhancement, respectively, and complete fine-grained bidirectional alignment of the two heterogeneous feature spaces. The text-guided visual mapping path uses clinical semantics as a guide to strengthen the feature weights of corresponding lesion areas in the image, while the visual-guided text mapping path uses image signs as a basis to associate and match corresponding clinical semantic information. The two mapping paths complement each other, fully exploring the complementary diagnostic value of dual-modal data. The calculation formula is as follows: ; in, The cross-modal projection matrix customized for this invention is different from the intramodal self-attention projection matrix and is used to achieve dimensionality adaptation and semantic mapping of heterogeneous feature spaces. The dimension of the cross-modal attention key vector; Cross-fusion features of image features guided by text semantics are used to achieve mapping alignment and feature calibration from semantic space to visual space; Cross-fusion features that guide textual semantics based on image features enable mapping and alignment from visual space to semantic space and semantic enhancement; S404. Perform bidirectional semantic alignment consistency constraints; the loss calculation formula is as follows: ;in, This is a global average pooling operation along the sequence dimension, used to extract a global semantic vector of bidirectional mapping features; The squared L2 norm is used to measure the difference between two global semantic vectors. S405. Perform adaptive semantic coupling gated fusion, dynamically adjusting the fusion weights of the two mapped features based on the quality of the current input bimodal features to achieve data-driven dynamic semantic coupling; the calculation formula for the gated unit is: ; in, The Sigmoid activation function has an output range of [0,1]. , The learnable weights and biases of the gating unit; An adaptive gating coefficient is used to dynamically balance the contribution ratios of text-guided visual and visual-guided text features. This is an element-wise multiplication operation; The initial coupling features after gating fusion; S406. Perform coupled feature residual optimization, fusing and optimizing the purified original single-modal features with the coupled features. This preserves cross-modal complementary information while maintaining the discriminative power of the core single-modal features, ultimately outputting the optimized multimodal deep semantic coupled features. ; in, , The learnable weight matrix and bias term of the residual mapping are used to achieve dimensional alignment between single-modal features and coupled features; The output is a multimodal deep semantic coupling feature.

[0011] As an improvement to the above technical solution, the specific steps of step S5 are as follows: S501. Perform multi-loss joint optimization; use the classification loss of the multimodal coupling branch as the main loss, and the classification losses of the two single-modal branches (breast ultrasound and clinical text) as auxiliary losses. On this basis, superimpose a bidirectional semantic alignment constraint loss to guide semantic consistency in the bidirectional cross-attention mapping process, avoid semantic bias in bidirectional mapping, and ensure the stability and interpretability of cross-modal coupling; the total loss calculation formula is as follows:

[0012] ; in, For multimodal coupled branch cross-entropy loss; This represents single-modal branching loss in breast ultrasound. For single-modal branching loss of clinical text; These are the classification loss weight coefficients, used to optimize the weights that balance the primary and secondary classification losses; These are the semantic alignment loss weight coefficients, used to control the constraint strength of bidirectional semantic alignment; for Defined bidirectional semantic alignment consistency loss; S502. Each classification branch uses the cross-entropy loss function, calculated as follows: ; in, This represents the number of training samples; Number of breast cancer classification categories; The true labels for the samples; Predict probability values ​​for the model; S503. An adaptive weighted decision mechanism is used to fuse the classification probabilities of the three-branch outputs; The multimodal semantic coupling branch has the highest weight to ensure diagnostic accuracy when both modalities are fully input. The weight allocation is automatically adjusted based on the completeness of the input data. When clinical text has missing fields or poor quality, the weight of the image monomodal branch is increased. When image information is limited, the diagnostic contribution of the text branch is strengthened accordingly to achieve data-driven dynamic decision-making. The final formula for calculating the diagnostic probability is as follows: ; in, , , These represent the predicted probabilities for breast images, clinical text, and multimodal coupling branches, respectively. , , For dynamic adaptive weights, satisfying .

[0013] Compared with the prior art, the advantages and positive effects of this invention are: This invention addresses the shortcomings of existing multimodal diagnostic protocols for breast cancer in clinical application by providing targeted optimizations, achieving significant improvements in diagnostic accuracy, generalization adaptability, and clinical usability. Specifically: (1) The diagnostic accuracy is substantially improved, effectively reducing the risk of missed or misdiagnosed diagnoses in clinical practice. Through a medically-oriented bidirectional semantic coupling mechanism, the problems of ineffective alignment of heterogeneous features and dilution of feature value by semantic noise in general attention schemes are solved. Fine-grained bidirectional alignment and deep complementarity between image visual features and clinical text semantics are achieved, effectively bridging the cross-modal semantic gap. Compared with shallow fusion and general attention schemes, this invention significantly improves the core indicators for classifying benign and malignant breast cancers. In particular, for the scenario of small lesions in the background of dense breast tissue, the lesion detection recall rate is greatly improved, and the missed diagnosis rate is reduced by orders of magnitude, which can provide a more reliable reference for clinical diagnosis.

[0014] (2) The robustness and generalization ability of the model are significantly enhanced, making it suitable for heterogeneous clinical data scenarios. A dual-strategy regularization calibration scheme designed for the characteristics of medical data is adopted to simultaneously suppress the risks of overfitting in small samples and cross-center feature domain shift. Compared with a single regularization scheme, the model's diagnostic accuracy decay on heterogeneous data across medical institutions is significantly reduced, and the generalization performance is qualitatively improved. At the same time, the three-branch joint optimization architecture has natural single-modal compatibility. Even in scenarios with incomplete data such as missing clinical texts and incomplete fields, it can still maintain stable diagnostic performance, and the performance degradation is significantly reduced, which can fully adapt to the complex and diverse diagnosis and treatment conditions in outpatient clinics.

[0015] (3) The diagnostic process aligns with clinical evidence-based logic and has stronger clinical interpretability. The bidirectional semantic coupling architecture can intuitively correspond the semantic relationship between ultrasound lesion signs and clinical history and test indicators, providing traceable bimodal evidence for diagnostic conclusions. It breaks through the black-box limitations of traditional fusion models, conforms to the diagnostic thinking of clinicians for bidirectional verification, and is more likely to be accepted by clinicians. It can effectively assist primary care physicians in improving their diagnostic level and promote the downward flow of high-quality medical resources.

[0016] (4) The technical architecture is highly scalable and has broad clinical extension potential. The overall solution adopts a modular design, which can be flexibly expanded to include more imaging modalities such as mammography and MRI, as well as multi-dimensional clinical data such as pathology and omics, to support more clinical tasks such as tumor molecular subtyping and prognostic assessment, and has the technical potential for continuous iteration and optimization. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the overall system flow of the present invention; Figure 2 This is a schematic diagram of the overall architecture of the model of the present invention; Figure 3 This is a schematic diagram of the ViT architecture of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art to all other embodiments obtained without creative effort should be included within the protection scope of the present invention.

[0020] This invention provides a breast cancer diagnosis method based on bidirectional semantic coupling and cross-modal attention fusion, the overall process of which is as follows: Figure 1 As shown, the overall architecture of the model is as follows: Figure 2 As shown, the method of the present invention forms a complete technical process of "multimodal data construction, data preprocessing, image and text feature extraction, cross-modal feature enhancement, cross-modal attention fusion, joint optimization training, and intelligent diagnostic output". The specific steps are as follows: (1) Multimodal data collection: Multimodal breast cancer data were collected from the First Affiliated Hospital of Zhengzhou University, covering breast ultrasound imaging data and patient clinical text data (including medical history, test indicators, demographic information, etc.). Professional physicians completed the labeling of imaging lesions and the standardization of clinical data according to the BI-RADS classification standard, and constructed a multimodal breast cancer dataset, which was divided into training set, validation set and test set.

[0021] (2) Data preprocessing: The dataset is subjected to uniform standardized preprocessing. On the clinical text side, data cleaning, terminology normalization, entity extraction and missing value filling are completed; on the image side, format conversion, grayscale normalization, image enhancement and geometric correction are completed to provide high-quality standardized input for subsequent feature encoding.

[0022] (3) ViT Image Encoding: For breast ultrasound images, the Vision Transformer (ViT) architecture is used for image encoding. Through image segmentation, linear projection, position embedding, and multi-layer Transformer encoders, global visual features such as lesion morphology, edges, and internal echoes are extracted, overcoming the limitations of local perception in traditional convolutional networks. The ViT architecture is as follows: Figure 3 As shown.

[0023] (4) MedBERT text encoding: For clinical text data, the medical domain pre-trained MedBERT model is used to perform text semantic encoding, construct the fusion feature representation of word embedding, position embedding and segment embedding, capture the medical context semantic association of clinical text through multi-layer Transformer encoding, and output highly discriminative global text features.

[0024] (5) Dual-strategy regularization feature calibration: Dual-strategy regularization calibration of dropout random deactivation and elastic network sparsity constraint is performed on image and text features respectively to suppress the risk of overfitting in medical small sample scenarios, purify core diagnostic features, and provide a stable and high-quality feature base for cross-modal semantic coupling.

[0025] (6) Bidirectional semantic coupling cross-modal attention fusion: A two-level hybrid attention mechanism of "self-attention purification within a modality + bidirectional cross-attention coupling between modalities" is adopted. First, the association of key features within a single modality is strengthened. Then, through bidirectional spatial mapping of semantic-guided visual calibration and visual-guided semantic enhancement, fine-grained bidirectional alignment and deep semantic coupling between the visual feature space and the text semantic space are achieved, generating multimodal fusion features.

[0026] (7) Multi-loss joint optimization: Construct a three-branch diagnostic structure of image single-modality, text single-modality, and multi-modality coupling. Using pathological diagnosis results as the gold standard, design a multi-loss joint optimization strategy with coupling branch loss as the main component and single-modality branch loss as the auxiliary component. Simultaneously constrain the single-modality feature quality and cross-modality coupling accuracy, and backpropagate to iteratively update the full-link parameters.

[0027] (8) Intelligent diagnostic results output: The classification probabilities of the three branches are integrated through an adaptive weighted voting mechanism to generate the final diagnostic results, outputting the conclusion of benign or malignant breast cancer and the pathological subtype classification results, providing auxiliary diagnostic reference for clinicians.

[0028] The core module of this invention and the method steps for its execution are as follows: (1) Multimodal coding feature extraction module: This module serves as the feature input basis for the model, constructing a global lesion perception ViT encoding branch for breast ultrasound and a medical semantic MedBERT encoding branch for clinical text, respectively, to complete customized deep feature extraction for the two types of heterogeneous data.

[0029] The ViT image encoding submodule employs the ViT architecture to perform global feature encoding of breast ultrasound images. It divides the preprocessed breast ultrasound images into fixed-size non-overlapping image blocks, which are then embedded using linear projection and learnable locations before being input into a multi-layer Transformer encoder. A self-attention mechanism models the long-distance spatial dependency between lesions and surrounding glandular tissue, accurately capturing core visual features such as lesion morphology, texture, and calcification distribution. The mathematical expression for the input embedding process is: ; in, This is a classification label vector used to aggregate global feature information; For the first One ultrasound image block; The linear embedding weight matrix; This represents the total number of image patches; It is a learnable location encoding vector used to preserve the spatial location characteristics of the image.

[0030] The core of the Transformer encoder is a multi-head self-attention mechanism, which comprehensively mines the global semantic information of the image by parallelly calculating the feature associations of multiple subspaces. The core self-attention calculation formula is: ; The expression for multi-head self-attention: ; in, These are the query, key, and value feature vectors, respectively. The dimension of the key vector; For the number of attention heads; This is used to output the projection weight matrix. After ViT encoding, the basic visual features of the breast ultrasound image are output.

[0031] A pre-trained MedBERT model in the medical field is used to perform deep semantic encoding of clinical texts. The pre-processed clinical texts undergo professional word segmentation and semantic normalization, constructing a fusion feature representation of word embeddings, positional embeddings, and segment embeddings. A multi-layer Transformer encoder deeply captures the core medical contextual semantic relationships in the clinical texts, such as medical history, symptoms, and laboratory indicators, generating globally relevant text features with professional discriminative capabilities. Based on this, multi-strategy feature selection and principal component analysis dimensionality reduction are integrated to eliminate redundant information, retain core diagnostic features, and improve the discrimination efficiency of text features.

[0032] (2) Dual-regularized feature calibration module for cross-modal coupling To address the issues of limited sample size and strong heterogeneity of medical data across institutions, which can easily lead to overfitting and distribution shift of single-modal features and consequently decrease the accuracy of cross-modal semantic coupling, this module performs a dual-strategy calibration on the features output by bimodal encoding, combining Dropout random deactivation regularization and elastic network sparse constraint regularization. This collaborative optimization addresses both generalization capability improvement and core feature purification, providing stable and highly discriminative feature inputs for subsequent cross-modal semantic coupling.

[0033] Dropout regularization randomly deactivates some neurons during model training, breaking redundant dependencies between neurons. This specifically addresses the overfitting problem in medical scenarios with small sample sizes and improves the generalization stability of single-modal features. The calculation formula is as follows: ; in, Input features are either ultrasound or text-based single-modal features; Preserve the probability for the neuron; This is an element-wise multiplication operation; A binary mask generated for the Bernoulli distribution.

[0034] Elastic network regularization combines the feature sparsity of L1 regularization with the feature stability of L2 regularization. On the one hand, it automatically weakens redundant feature weights through sparse constraints, strengthening the contribution of key diagnostic features such as lesion signs and core test indicators, ensuring feature discrimination purity during cross-modal coupling. On the other hand, it smooths feature distribution through L2 constraints, mitigating feature domain shifts between data from different medical institutions and improving the model's adaptability to heterogeneous data. Its loss function is calculated as follows: ]; in, For feature reconstruction loss; L1 regularization coefficient, The L2 regularization coefficient; L1 norm, It is an L2 norm.

[0035] The two types of regularized optimized features are concatenated dimensionally, and then mapped through a linear layer and activation function to obtain single-modality calibration features: ' in, , The weight matrix is ​​a learnable weight matrix; , For bias terms; This is a feature dimension concatenation operation; These are the image features after regularization enhancement. These are text features enhanced by regularization.

[0036] (3) Hybrid attention-driven cross-modal feature fusion module To address the issues of semantic misalignment in multimodal heterogeneous features, information bias in unidirectional attention mapping, and the inability to achieve deep complementarity, a full-process bidirectional semantic coupling architecture is constructed, which includes intramodal self-attention optimization, cross-modal semantic affinity measurement, bidirectional cross-attention mapping, semantic alignment, consistency constraints, adaptive gating coupling, and residual feature optimization. This architecture enables closed-loop mapping and fine-grained semantic alignment between the visual feature space and the text semantic space, effectively mitigating the cross-modal semantic gap.

[0037] 1) Intramodal self-attention feature optimization: Multi-head self-attention computation is performed on regularized and calibrated breast ultrasound features and clinical text features respectively, focusing on the core discriminative features within each modality, strengthening the feature correlation between local and global features within the modality, suppressing irrelevant noise interference, improving the task relevance of single-modal features, providing high-purity feature input for subsequent cross-modal coupling, and outputting purified single-modal features. , .

[0038] 2) Cross-modal semantic affinity measurement Before performing bidirectional cross-attention mapping, a cross-modal semantic affinity matrix is ​​constructed to quantify the fine-grained semantic association between text semantic tokens and ultrasound image patches. This provides a semantic weighting basis for subsequent bidirectional coupling, achieving precise alignment based on semantic similarity, rather than a general, undifferentiated attention mapping. The semantic metric calculation formula is as follows: ; in, For cross-modal semantic affinity matrix, The length of the text sequence. The length of the image sequence; , These represent the dimensions of text and image features, respectively. The higher the value of each element in this matrix, the stronger the semantic correlation between the corresponding text semantic unit and the image visual unit.

[0039] 3) Bidirectional cross-attention semantic mapping Based on the semantic affinity matrix, two sets of dedicated cross-modal projection matrices are constructed to achieve bidirectional spatial mapping of text semantic-guided visual feature calibration and visual feature-guided text semantic enhancement, respectively, completing fine-grained bidirectional alignment of the two heterogeneous feature spaces. The text-guided visual mapping path strengthens the feature weights of corresponding lesion regions in the image based on clinical semantics, while the visual-guided text mapping path associates and matches corresponding clinical semantic information based on image signs. The two mapping paths complement each other, fully exploring the complementary diagnostic value of dual-modal data. The calculation formula is as follows: ; in, The cross-modal projection matrix customized for this invention is different from the intramodal self-attention projection matrix and is used to achieve dimensionality adaptation and semantic mapping of heterogeneous feature spaces. The dimension of the cross-modal attention key vector; Cross-fusion features of image features guided by text semantics are used to achieve mapping alignment and feature calibration from semantic space to visual space; This approach guides the cross-fusion of image features with textual semantics, enabling the mapping and alignment from visual space to semantic space and semantic enhancement.

[0040] 4) Bidirectional semantic alignment consistency constraints To ensure semantic consistency of bidirectional mapping and avoid semantic deviation in the coupling results of the two directions, a bidirectional semantic alignment loss is introduced. This loss constrains the distance between the global features mapped in the two directions in the common semantic space, strengthening the closed-loop consistency of bidirectional coupling. The loss calculation formula is as follows: ; in, This is a global average pooling operation along the sequence dimension, used to extract a global semantic vector of bidirectional mapping features; The L2 norm squared is used to measure the difference between two global semantic vectors. This loss reduces the semantic bias of the bidirectional mapping through training, ensuring the semantic consistency of the coupling process.

[0041] 5) Adaptive semantic coupling gated fusion Abandoning the simple summation method with fixed weights, an adaptive semantic coupling gating unit is designed. This unit dynamically adjusts the fusion weights of the two mapped features based on the quality of the current input bimodal features, achieving data-driven dynamic semantic coupling. When the data quality of a certain modality is poor or information is missing, the gating unit can automatically reduce the contribution ratio of the corresponding path, ensuring the reliability of the fusion result and adapting to real-world scenarios with uneven clinical data quality. The gating unit calculation formula is as follows: ; in, The Sigmoid activation function has an output range of [0,1]. , The learnable weights and biases of the gating unit; An adaptive gating coefficient is used to dynamically balance the contribution ratios of text-guided visual and visual-guided text features. This is an element-wise multiplication operation; This represents the initial coupling characteristics after gating fusion.

[0042] 6) Coupling Feature Residual Optimization To avoid losing the original core diagnostic information of a single modality during the cross-modal coupling process, a residual connection mechanism is introduced to fuse and optimize the purified original single-modal features with the coupled features. This retains the complementary information across modalities while maintaining the discriminative power of the core single-modal features, ultimately outputting optimized multimodal deep semantic coupling features. ; in, , The learnable weight matrix and bias term of the residual mapping are used to achieve dimensional alignment between single-modal features and coupled features; The output is a multimodal deep semantic coupling feature.

[0043] (4) Multi-loss joint optimization and adaptive decision-making module To address the issues of model bias caused by single-loss optimization, diagnostic bias due to missing single-modal data, and insufficient diagnostic reliability in scenarios with incomplete clinical data, this module constructs a multi-dimensional joint loss function with a three-branch classification loss and bidirectional semantic alignment constraints. It is paired with a validation set-driven adaptive weighted decision mechanism to simultaneously constrain the quality of single-modal feature encoding, the accuracy of cross-modal semantic coupling, and the diagnostic robustness in complex scenarios.

[0044] The classification loss of the multimodal coupling branch is used as the primary loss to ensure the core accuracy of the fusion diagnosis, while the classification losses of the two single-modal branches (breast ultrasound and clinical text) are used as auxiliary losses. On the one hand, this inversely constrains the feature discrimination ability of the dual-modal coding branch, preventing excessive dilution of single-modal features and resulting in modality collapse during cross-modal coupling. On the other hand, it preserves independent diagnostic capabilities for single-modal input scenarios, supporting downgraded diagnoses when clinical data is incomplete. Based on this, a bidirectional semantic alignment constraint loss is superimposed to guide semantic consistency during the bidirectional cross-attention mapping process, avoiding semantic bias in bidirectional mapping and ensuring the stability and interpretability of cross-modal coupling. The total loss calculation formula is as follows: ; in, For multimodal coupled branch cross-entropy loss; This represents single-modal branching loss in breast ultrasound. For single-modal branching loss of clinical text; These are the classification loss weight coefficients, used to optimize the weights that balance the primary and secondary classification losses; These are the semantic alignment loss weight coefficients, used to control the constraint strength of bidirectional semantic alignment; The bidirectional semantic alignment consistency loss is defined for Equation (9). Each classification branch uses the cross-entropy loss function, calculated as follows: ; in, This represents the number of training samples; Number of breast cancer classification categories; The true labels for the samples; Predict probability values ​​for the model.

[0045] During the inference phase, an adaptive weighted decision mechanism is employed to fuse the classification probabilities of the three-branch outputs. The weight coefficients are dynamically allocated based on the model's AUC performance on the validation set, with the multimodal semantic coupling branch having the highest weight to ensure diagnostic accuracy when both modalities are fully input. This mechanism also automatically adjusts weight allocation based on the completeness of the input data. When clinical text contains missing fields or is of poor quality, the weight of the single-modal imaging branch is automatically increased; when imaging information is limited, the diagnostic contribution of the text branch is strengthened accordingly, achieving data-driven dynamic decision-making and fully adapting to the complex and diverse data conditions in outpatient settings. The final diagnostic probability calculation formula is as follows: ; in, , , These represent the predicted probabilities for breast images, clinical text, and multimodal coupling branches, respectively. , , For dynamic adaptive weights, satisfying .

Claims

1. A breast cancer diagnostic method based on bidirectional semantic coupling and cross-modal attention fusion, characterized in that: Includes the following steps: S1. Collect multimodal data to form a multimodal breast cancer dataset and perform data preprocessing on the multimodal breast cancer dataset; S2. Image encoding of breast ultrasound images is performed using the ViT architecture; for clinical text data, text semantic encoding is performed using a pre-trained MedBERT model in the medical field. S3. Perform dual-strategy regularization calibration on image and text features, respectively, using Dropout random deactivation and elastic network sparsity constraints. S4. A two-level hybrid attention mechanism of "intramodal self-attention purification + intermodal bidirectional cross-attention coupling" is adopted to achieve fine-grained bidirectional alignment and deep semantic coupling between visual feature space and text semantic space, and generate multimodal fusion features. S5. Construct a three-branch diagnostic structure that couples image monomodality, text monomodality, and multimodality; and generate the final diagnostic result by fusing the classification probabilities of the three branches through an adaptive weighted voting mechanism.

2. The breast cancer diagnosis method based on bidirectional semantic coupling and cross-modal attention fusion as described in claim 1, characterized in that: In step S1, the multimodal data includes breast ultrasound imaging data and patient clinical text data; the patient clinical text data includes medical history, laboratory indicators, and demographic information; the data preprocessing is to perform uniform standardized preprocessing on the dataset.

3. The breast cancer diagnosis method based on bidirectional semantic coupling and cross-modal attention fusion as described in claim 1, characterized in that: The specific steps of step S2 are as follows: The S201.ViT architecture encodes global features of breast ultrasound images. Preprocessed breast ultrasound images are divided into fixed-size, non-overlapping image blocks. After linear projection and learnable location embedding, these blocks are input into a multi-layer Transformer encoder. A self-attention mechanism models the long-distance spatial dependency between lesions and surrounding glandular tissue, accurately capturing core visual features such as lesion morphology, texture, and calcification distribution. The mathematical expression is: ； in, This is a classification label vector used to aggregate global feature information; For the first One ultrasound image block; The linear embedding weight matrix; This represents the total number of image patches; It is a learnable location encoding vector used to preserve the spatial location characteristics of the image; The S202.Transformer encoder comprehensively mines global semantic information of images by parallel computing feature associations across multiple subspaces. The self-attention calculation formula is: ； The expression for multi-head self-attention is: ; in, These are the query, key, and value feature vectors, respectively. The dimension of the key vector; For the number of attention heads; The output projection weight matrix is ​​used to encode the basic visual features of breast ultrasound images. ; S203. A pre-trained MedBERT model in the medical field is used to perform deep semantic encoding of clinical texts. The pre-processed clinical texts undergo professional word segmentation and semantic normalization, constructing a fusion feature representation of word embeddings, positional embeddings, and segment embeddings. A multi-layer Transformer encoder is then used to deeply capture the core medical contextual semantic relationships in the clinical texts, such as medical history, symptoms, and laboratory indicators, generating globally relevant text features with professional discriminative capabilities. .

4. The breast cancer diagnosis method based on bidirectional semantic coupling and cross-modal attention fusion as described in claim 1, characterized in that: The specific steps of step S3 are as follows: S301. Perform a dual-strategy combination calibration of Dropout random deactivation regularization and elastic network sparse constraint regularization on the features output by the dual-modal coding; The formula for calculating Dropout regularization is: ； in, Input features are either ultrasound or text-based single-modal features; Preserve the probability of the neuron; This is an element-wise multiplication operation; A binary mask generated for a Bernoulli distribution; The loss function for regularization of elastic networks is calculated using the following formula: ； in, For feature reconstruction loss; L1 regularization coefficient, The L2 regularization coefficient; L1 norm, It is an L2 norm; S302. The features optimized by Dropout random deactivation regularization and elastic network regularization are concatenated dimensionally, and then mapped through a linear layer and activation function to obtain the single-modality calibration features: ； in, , The weight matrix is ​​a learnable weight matrix; , For bias terms; This is a feature dimension concatenation operation; These are the image features after regularization enhancement. These are text features enhanced by regularization.

5. The breast cancer diagnosis method based on bidirectional semantic coupling and cross-modal attention fusion as described in claim 1, characterized in that: The specific steps of step S4 are as follows: S401. Perform multi-head self-attention computation on the regularized and calibrated breast ultrasound features and clinical text features respectively, focusing on the core discriminative features within each modality, strengthening the feature correlation between local and global features within the modality, suppressing irrelevant noise interference, improving the task relevance of single-modal features, providing high-purity feature input for subsequent cross-modal coupling, and outputting purified single-modal features. , ; S402. Construct a cross-modal semantic affinity matrix to quantify the fine-grained semantic association between text semantic tokens and ultrasound image patches, providing a semantic weight basis for subsequent bidirectional coupling and achieving accurate alignment based on semantic similarity; the semantic metric calculation formula is as follows: ； in, For cross-modal semantic affinity matrix, The length of the text sequence. The length of the image sequence; , These are the dimensions of text and image features, respectively; S403. Based on the semantic affinity matrix, construct two sets of dedicated cross-modal projection matrices to realize bidirectional spatial mapping of text semantic guidance visual feature calibration and visual feature guidance text semantic enhancement, respectively, and complete fine-grained bidirectional alignment of the two heterogeneous feature spaces. The text-guided visual mapping path uses clinical semantics as a guide to strengthen the feature weights of corresponding lesion areas in the image, while the visual-guided text mapping path uses image signs as a basis to associate and match corresponding clinical semantic information. The two mapping paths complement each other, fully exploring the complementary diagnostic value of dual-modal data. The calculation formula is as follows: ； in, The cross-modal projection matrix customized for this invention is different from the intramodal self-attention projection matrix and is used to achieve dimensionality adaptation and semantic mapping of heterogeneous feature spaces. The dimension of the cross-modal attention key vector; Cross-fusion features of image features guided by text semantics are used to achieve mapping alignment and feature calibration from semantic space to visual space; Cross-fusion features that guide textual semantics based on image features enable mapping and alignment from visual space to semantic space and semantic enhancement; S404. Perform bidirectional semantic alignment consistency constraints; the loss calculation formula is as follows: ;in, This is a global average pooling operation along the sequence dimension, used to extract a global semantic vector of bidirectional mapping features; The squared L2 norm is used to measure the difference between two global semantic vectors. S405. Perform adaptive semantic coupling gated fusion, dynamically adjusting the fusion weights of the two mapped features based on the quality of the current input bimodal features to achieve data-driven dynamic semantic coupling; the calculation formula for the gated unit is: ； in, The Sigmoid activation function has an output range of [0,1]. , The learnable weights and biases of the gating unit; An adaptive gating coefficient is used to dynamically balance the contribution ratios of text-guided visual and visual-guided text features. This is an element-wise multiplication operation; Initial coupling features after gating fusion S406. Perform coupled feature residual optimization, fusing and optimizing the purified original single-modal features with the coupled features. This preserves cross-modal complementary information while maintaining the discriminative power of the core single-modal features, ultimately outputting the optimized multimodal deep semantic coupled features. ; in, , The learnable weight matrix and bias term of the residual mapping are used to achieve dimensional alignment between single-modal features and coupled features; The output is a multimodal deep semantic coupling feature.

6. The breast cancer diagnosis method based on bidirectional semantic coupling and cross-modal attention fusion as described in claim 1, characterized in that: The specific steps of step S5 are as follows: S501. Perform multi-loss joint optimization; use the classification loss of the multimodal coupling branch as the main loss, and the classification losses of the two single-modal branches of breast ultrasound and clinical text as auxiliary losses. On this basis, superimpose a bidirectional semantic alignment constraint loss to guide semantic consistency in the bidirectional cross-attention mapping process, avoid semantic bias in bidirectional mapping, and ensure the stability and interpretability of cross-modal coupling; the total loss calculation formula is: ； in, For multimodal coupled branch cross-entropy loss; This represents single-modal branching loss in breast ultrasound. For single-modal branching loss of clinical text; These are the classification loss weight coefficients, used to optimize the weights that balance the primary and secondary classification losses; These are the semantic alignment loss weight coefficients, used to control the constraint strength of bidirectional semantic alignment; for Defined bidirectional semantic alignment consistency loss; S502. Each classification branch uses the cross-entropy loss function, calculated as follows: ; in, This represents the number of training samples; Number of breast cancer classification categories; The true labels for the samples; Predict probability values ​​for the model; S503. An adaptive weighted decision mechanism is used to fuse the classification probabilities of the three-branch outputs; The multimodal semantic coupling branch has the highest weight to ensure diagnostic accuracy when both modalities are fully input. The weight allocation is automatically adjusted based on the completeness of the input data. When clinical text has missing fields or poor quality, the weight of the image monomodal branch is increased. When image information is limited, the diagnostic contribution of the text branch is strengthened accordingly to achieve data-driven dynamic decision-making. The final formula for calculating the diagnostic probability is as follows: ; in, , , These represent the predicted probabilities for breast images, clinical text, and multimodal coupling branches, respectively. , , For dynamic adaptive weights, satisfying .

Images