A breast image classification method fusing dual-view information and clinical metadata

Abstract

This invention relates to a breast image classification method that integrates dual-view information and clinical metadata, belonging to the field of deep learning-based image recognition technology. The method includes: establishing a classification dataset and preprocessing it; constructing a dual-view feature extraction module to acquire features from both views; achieving deep information interaction between views through a local cross-view interaction module to generate local representation features, and combining this with a global representation module to obtain overall features; enriching feature representation using a multi-scale feature enhancement module; extracting age and density features through a clinical metadata encoding module; designing a multimodal fusion attention mechanism and an adaptive weight fusion module to integrate local image features, global features, and metadata features; and finally constructing a cross-modal dual-view interactive convolutional neural network, using a combined loss function including focus loss, contrast learning loss, and consistency loss for training and evaluation. This invention can improve the accuracy and robustness of breast image classification.

Description

Technical Field

[0001] This invention belongs to the field of deep learning-based image recognition technology, specifically relating to a breast image classification method that integrates dual-view information and clinical metadata. Background Technology

[0002] Breast cancer is one of the most common malignant tumors among women worldwide. According to the latest statistics from the World Health Organization in 2024, breast cancer ranks first in the incidence of cancer among women, seriously threatening women's health. Early detection and accurate diagnosis are of great significance for improving the survival rate and quality of life of breast cancer patients. Mammography, as the gold standard for breast cancer screening, is widely used in clinical practice.

[0003] With the rapid development of deep learning technology, AI-based medical image analysis methods have shown great potential in the field of breast cancer detection. However, most existing deep learning methods only utilize image information from a single view, failing to fully explore the complementary information of multi-view images, and neglecting the guiding role of important clinical metadata such as patient age and breast density in diagnosis. The Mini-DDSM dataset, as a standard dataset in the field of breast cancer detection, contains mammograms of two standard views, CC and MLO, as well as important clinical information such as patient age and breast density. Effectively fusing this information is expected to improve the accuracy of breast image classification. Summary of the Invention

[0004] The purpose of this invention is to address the problems in the prior art, such as insufficient utilization of single-view image information, insufficient interaction between views, insufficient fusion of clinical metadata, and limited model generalization ability, and to provide a breast image classification method that integrates dual-view information and clinical metadata.

[0005] To achieve the above objectives, the technical solution of the present invention is: a breast image classification method that integrates dual-view information and clinical metadata, comprising the following steps:

[0006] Step S1: Establish a breast image classification dataset;

[0007] Step S2: Preprocess the mammogram images and clinical metadata in the dataset;

[0008] Step S3: Construct a dual-view feature extraction module;

[0009] Step S4: Construct a local cross-view interaction module;

[0010] Step S5: Construct the global representation module;

[0011] Step S6: Construct a multi-scale feature enhancement module;

[0012] Step S7: Construct the clinical metadata encoding module;

[0013] Step S8: Construct a multimodal fusion attention mechanism;

[0014] Step S9: Construct an adaptive weight fusion module;

[0015] Step S10: Construct a cross-modal dual-view interactive convolutional neural network;

[0016] Step S11: Design the combined loss function;

[0017] Step S12: Train a cross-modal dual-view interactive convolutional neural network and evaluate its performance.

[0018] Furthermore, in step S1, mammograms of CC view and MLO view, along with corresponding clinical metadata such as patient age and breast density, are extracted from the Mini-DDSM dataset to establish a breast image classification dataset, which is then divided into training set, validation set, and test set in an 8:1:1 ratio.

[0019] Furthermore, in step S2, the mammogram images are subjected to CLAHE contrast enhancement, Gaussian noise reduction filtering, and ROI region extraction, and the image size is uniformly adjusted to 512×512 pixels; the patient's age is standardized, and the breast density is encoded using one-hot encoding; and data augmentation techniques such as random rotation, flipping, and scaling are used to expand the training set.

[0020] Furthermore, step S3 is specifically implemented as follows:

[0021] Step S31: Construct CC view feature extraction branch: The improved EfficientNetV2-S is used as the backbone network. The input layer receives a 512×512 CC view image. After a series of convolution, batch normalization, activation function and pooling operations, a 512-dimensional feature vector is finally output.

[0022] Step S32: Construct the MLO view feature extraction branch: Using the same network structure as the CC view, the input layer receives a 512×512 MLO view image and outputs a 512-dimensional feature vector.

[0023] Furthermore, step S4 is specifically implemented as follows:

[0024] Step S41: Construct a cross-shape window attention mechanism: Divide the features of the CC view and MLO view into different horizontal and vertical stripes. The height of the horizontal strip is sw, and the width of the vertical strip is sw, where sw is a dynamic strip width parameter and is set to 1. Divide the attention heads into two groups. The first group processes the horizontal strips, and the second group processes the vertical strips to expand the receptive field of the features.

[0025] Step S42: Construct a cross-view attention module: For horizontal stripes, perform attention calculations on the Q matrix generated by the MLO view and the K and V matrices generated by the CC view. The specific expression is as follows:

[0026]

[0027] in The query matrix for MLO view features. The key matrix of CC view features. The value matrix of the CC view features. The transpose of the key matrix of the CC view features. The dimension of the key vector.

[0028] Similarly, the same cross-view attention calculation is applied to vertical stripes to achieve depth information interaction and feature enhancement between the two views.

[0029] Step S43: Generate local feature representations: The features after cross-view attention processing are processed by global average pooling, and the results are concatenated to obtain local representation features.

[0030] Furthermore, step S5 is specifically implemented as follows:

[0031] Step S51: Extract global features: Concatenate the CC view features and MLO view features output by the dual-view feature extraction module in the channel dimension;

[0032] Step S52: Apply global average pooling: Perform global average pooling on the concatenated features to obtain global representation features, so as to capture the overall breast structure information and global semantic relationships.

[0033] Furthermore, step S6 is specifically implemented as follows:

[0034] Step S61: Construct multi-scale convolutional branches: Design four parallel convolutional branches. The first branch uses a convolutional layer with a kernel size of 1×1, 64 kernels, and a stride of 1 to capture detailed features. The second branch uses a convolutional layer with a kernel size of 3×3, 64 kernels, a stride of 1, and padding of 1 to capture local features. The third branch uses a convolutional layer with a kernel size of 5×5, 64 kernels, a stride of 1, and padding of 2 to capture medium-range features. The fourth branch uses a convolutional layer with a kernel size of 7×7, 64 kernels, a stride of 1, and padding of 3 to capture global features. Each branch is followed by a batch normalization layer and a ReLU activation function.

[0035] Step S62: Construct the dilated convolution module: Design three dilated convolution layers. The first layer uses a dilation rate of 2, a kernel size of 3×3, and 128 kernels. The second layer uses a dilation rate of 4, a kernel size of 3×3, and 128 kernels. The third layer uses a dilation rate of 8, a kernel size of 3×3, and 128 kernels. Each layer is followed by a batch normalization layer and a ReLU activation function to expand the receptive field and capture contextual information at different scales.

[0036] Step S63: Construct a feature fusion layer: The outputs of the multi-scale convolutional branches and dilated convolutional modules are concatenated in the channel dimension to obtain 640-dimensional features; feature fusion and dimension adjustment are performed through a convolutional layer with a kernel size of 1×1, 256 kernels, and a stride of 1, followed by a batch normalization layer, a ReLU activation function, and a Dropout layer.

[0037] Furthermore, step S7 is specifically implemented as follows:

[0038] Step S71: Construct an age encoding submodule: Encode continuous age values ​​through a three-layer fully connected network. The first layer contains 16 neurons and uses the ReLU activation function and Dropout; the second layer contains 32 neurons and uses the ReLU activation function and batch normalization; the third layer contains 64 neurons and uses the ReLU activation function, outputting a 64-dimensional age feature vector.

[0039] Step S72: Construct a breast density encoding submodule: Map the breast density category to a 64-dimensional dense vector through an embedding layer; then further encode it through a two-layer fully connected network. The first layer contains 128 neurons and uses the ReLU activation function and Dropout; the second layer contains 64 neurons and uses the ReLU activation function to output a 64-dimensional density feature vector.

[0040] Step S73: Construct a metadata fusion layer: Concatenate the age feature vector and the density feature vector to obtain a 128-dimensional metadata feature vector; perform feature fusion through a convolutional layer with a kernel size of 1×1 and 128 kernels, followed by a batch normalization layer and a ReLU activation function to output the final metadata encoded features.

[0041] Furthermore, step S8 is specifically implemented as follows:

[0042] Step S81: Construct an image-metadata cross-attention module: Use image features as Q and metadata features as K and V, and calculate cross-attention weights through a multi-head attention mechanism; set the number of attention heads to 8, and the dimension of each head to 64.

[0043] Step S82: Construct a dual-view self-attention module: Calculate self-attention for the features of the CC view and MLO view respectively. Each view uses 16 attention heads, each head has a dimension of 64, and the hidden layer has a dimension of 768. The self-attention calculation adopts a Transformer encoder structure, which includes position encoding, multi-head self-attention layers, feedforward network layers, and residual connections. The feedforward network contains two linear transformation layers with an intermediate dimension of 2048 and uses the GELU activation function.

[0044] Step S83: Construct a global context fusion module: Perform global average pooling and global max pooling on the fused feature map to obtain two global feature vectors; perform feature transformation through two layers of MLP, with the first layer having 256 neurons and using the ReLU activation function; and the second layer having 512 neurons and using the Sigmoid activation function; multiply the transformed feature vectors element-wise with the original feature map to achieve the fusion of global context information.

[0045] Furthermore, step S9 is specifically implemented as follows:

[0046] Step S91: Construct a feature importance evaluation module: Use a channel attention mechanism to evaluate the importance of different feature channels; calculate the importance score of each channel through a convolutional layer with a kernel size of 1×1 and 512 kernels; normalize the score to the [0,1] interval using the Sigmoid activation function; multiply the importance score element-wise with the corresponding feature channel to achieve feature weighting;

[0047] Step S92: Construct a dynamic weight generation module: Design a three-layer fully connected network to adaptively generate fused weights; the input is the concatenated multimodal features. The first layer contains 256 neurons and uses the ReLU activation function and Dropout; the second layer contains 128 neurons and uses the ReLU activation function and batch normalization; the third layer contains 3 neurons and uses the Softmax activation function to ensure that the weight sum is 1.

[0048] Step S93: Construct a weighted feature fusion module: Perform weighted fusion of local representation features, global representation features, and metadata features based on dynamically generated weights.

[0049]

[0050] in, For local representation features, For global representation features, Metadata characteristics, , , These are the corresponding adaptive weights.

[0051] Furthermore, step S10 is specifically implemented as follows:

[0052] Step S101: Construct the network backbone structure: The dual-view feature extraction module receives CC and MLO view images and outputs two 512-dimensional feature vectors;

[0053] Step S102: Construct parallel processing branches: The local cross-view interaction module and the global representation module process the dual-view features in parallel, generating local and global representations respectively; the multi-scale feature enhancement module enhances the local representation; the clinical metadata encoding module processes patient age and breast density, outputting 128-dimensional metadata features;

[0054] Step S103: Integrate the multimodal fusion module: The multimodal fusion attention mechanism and adaptive weight fusion module integrate local representation, global representation and metadata features to output the final feature representation;

[0055] Step S104: Construct a classification decision module: It contains a two-layer fully connected network. The first layer has 256 neurons and uses the ReLU activation function and Dropout. The second layer has 2 neurons and uses the Softmax activation function to output the class probability distribution.

[0056] Furthermore, in step S11, a combined loss function is designed to improve the model's learning performance and generalization ability. The specific expression is as follows:

[0057]

[0058] in, This is the focus loss function, used to address the class imbalance problem. To learn a better feature representation, a contrastive learning loss function is used. The consistency loss function ensures the consistency of predictions between the two views, where α, β, and δ are the weighting coefficients, respectively.

[0059] Furthermore, in step S12, the Adam optimizer is used for network training, with an initial learning rate of 0.0001 and a weight decay coefficient of 0.01; a cosine annealing learning rate scheduling strategy is adopted, with a minimum learning rate of 1e-6; the batch size is set to 32 and the number of training rounds is set to 60; an early stopping strategy is adopted, and training is stopped when the validation set AUC does not improve for 10 consecutive rounds; and 5-fold cross-validation is used to evaluate the model performance.

[0060] Compared to existing technologies, this invention offers the following advantages: Firstly, it achieves deep interaction and fusion of CC and MLO view features through a local cross-view interaction module, enabling the model to automatically learn dependencies between different views, fully exploit complementary information, and significantly improve classification accuracy. Secondly, this invention not only utilizes image information but also integrates important clinical metadata such as patient age and breast density, achieving effective combination of image and clinical information through a specially designed encoding module and fusion mechanism. Thirdly, through a multimodal fusion attention mechanism and a cross-view attention module, it automatically learns the importance weights of different modalities and views, achieving more accurate feature representation and better model interpretability. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating the overall process of the breast image classification method of the present invention, which integrates dual-view information and clinical metadata.

[0062] Figure 2 This is a schematic diagram of the dual-view feature extraction module of the present invention.

[0063] Figure 3 This is a schematic diagram of the structure of the local cross-view interaction module of the present invention.

[0064] Figure 4 This is a schematic diagram of the structure of the multi-scale feature enhancement module of the present invention.

[0065] Figure 5 This is a schematic diagram of the adaptive weight fusion module of the present invention.

[0066] Figure 6 This is a schematic diagram of the overall neural network structure based on cross-view interaction and multimodal fusion of the present invention.

[0067] Figure 7 This is a comparison chart of the experimental results. Detailed Implementation

[0068] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0069] This invention provides a breast image classification method that integrates dual-view information and clinical metadata, comprising the following steps:

[0070] Step S1: Establish a breast image classification dataset;

[0071] Step S2: Preprocess the mammogram images and clinical metadata in the dataset;

[0072] Step S3: Construct a dual-view feature extraction module;

[0073] Step S4: Construct a local cross-view interaction module;

[0074] Step S5: Construct the global representation module;

[0075] Step S6: Construct a multi-scale feature enhancement module;

[0076] Step S7: Construct the clinical metadata encoding module;

[0077] Step S8: Construct a multimodal fusion attention mechanism;

[0078] Step S9: Construct an adaptive weight fusion module;

[0079] Step S10: Construct a cross-modal dual-view interactive convolutional neural network;

[0080] Step S11: Design the combined loss function;

[0081] Step S12: Train a cross-modal dual-view interactive convolutional neural network and evaluate its performance.

[0082] The following is a detailed implementation process of the present invention.

[0083] like Figure 1 As shown, this invention provides an overall design flow for a breast image classification method that integrates dual-view information and clinical metadata, including the following steps:

[0084] Step S1: Establish a breast image classification dataset: Extract mammograms of CC view and MLO view from the Mini-DDSM dataset, along with corresponding clinical metadata such as patient age and breast density, to establish a breast image classification dataset, and divide it into training set, validation set and test set in an 8:1:1 ratio.

[0085] Step S2, Data Preprocessing: Perform CLAHE contrast enhancement, Gaussian filtering for noise reduction, and region of interest extraction on mammogram images; uniformly adjust the image size to 512×512 pixels; standardize patient age and perform one-hot encoding on breast density; expand the training set using data augmentation techniques such as random rotation, flipping, and scaling.

[0086] Step S3, as follows Figure 2 As shown, a dual-view feature extraction module is constructed:

[0087] Step S31: Constructing the CC view feature extraction branch: An improved EfficientNetV2-S is used as the backbone network, with the input layer receiving a 512×512 CC view image. EfficientNetV2-S consists of multiple MBConv blocks and FusedMBConv blocks, including convolutional layers, batch normalization layers, Swish activation functions, and depthwise separable convolutions. The network outputs a feature map through a series of feature extraction and downsampling operations. Finally, global average pooling is used to convert the feature map into a 1280-dimensional feature vector, and then a 1×1 convolutional layer is used to reduce the dimension to 512, obtaining the final CC view feature representation.

[0088] Step S32: Construct the MLO view feature extraction branch: Use the same EfficientNetV2-S network structure as the CC view. The input layer receives a 512×512 MLO view image and outputs a 512-dimensional feature vector.

[0089] Step S4, as follows Figure 3 As shown, construct a local cross-view interaction module:

[0090] Step S41: Construct a cross-shape window attention mechanism: Reshape the 512-dimensional features of the CC view and MLO view into 20×20×128 feature maps, and then evenly divide them into horizontal and vertical stripes. For horizontal strip segmentation, each strip has a height of sw=1, resulting in 20 strips, each with a dimension of 1×20×128; for vertical strip segmentation, each strip has a width of sw=1, resulting in 20 strips, each with a dimension of 20×1×128. Set the number of attention heads K=4, and divide these heads into two groups: the first group of 2 heads processes the horizontal strips, and the second group of 2 heads processes the vertical strips.

[0091] Step S42: Construct a cross-view attention module: For the first set of attention heads processing horizontal stripes, use the features of the MLO view as Q and the features of the CC view as K and V, and calculate the cross-view attention:

[0092]

[0093] in The query matrix for MLO view features. The key matrix of CC view features. The value matrix of the CC view features. The transpose of the key matrix of the CC view features. The dimension of the key vector.

[0094] Step S43: Generate Local Representation: Perform global average pooling on the CC view and MLO view features after cross-view attention processing to obtain two 512-dimensional feature vectors. Concatenate these two vectors to form a 1024-dimensional local representation feature for subsequent multimodal fusion.

[0095] Step S5: Construct the global representation module:

[0096] Step S51: Extract global features: The 512-dimensional features of the CC view and the 512-dimensional features of the MLO view output by the dual-view feature extraction module are concatenated in the channel dimension to obtain 1024-dimensional concatenated features.

[0097] Step S52: Apply global average pooling: Perform global average pooling on the 1024-dimensional concatenated features to obtain 512-dimensional global representation features. These global representation features are primarily used to capture overall breast structure information, global morphology, and density distribution features.

[0098] Step S6, as follows Figure 4 As shown, a multi-scale feature enhancement module is constructed:

[0099] Step S61: Construct multi-scale convolutional branches: Design four parallel branches for multi-scale feature extraction. The first branch uses a convolutional layer with a kernel size of 1×1, 64 kernels, and a stride of 1 to preserve the original feature information; the second branch uses a convolutional layer with a kernel size of 3×3, 64 kernels, a stride of 1, and padding of 1, preceded by a 1×1 convolution for dimensionality reduction; the third branch uses two cascaded 3×3 convolutional layers, each with 64 kernels, a stride of 1, and padding of 1, preceded by a 1×1 dimensionality-reducing convolution; the fourth branch uses three cascaded 3×3 convolutional layers with the same configuration as the third branch. Each branch is followed by a batch normalization layer, a ReLU activation function, and a Dropout layer.

[0100] Step S62: Construct the dilated convolution module: An ASPP structure is adopted, containing four dilated convolution branches. The first branch is a regular 1×1 convolution with 128 kernels; the second, third, and fourth branches use 3×3 dilated convolutions with dilation rates of 6, 12, and 18 respectively, each with 128 kernels. An additional global average pooling branch is added, followed by a 1×1 convolution and bilinear upsampling to restore the original size. All branch outputs are concatenated along the channel dimension and then fused into 256-dimensional features via a 1×1 convolution.

[0101] Step S63: Construct a feature fusion layer: The outputs of the multi-scale convolutional branches and dilated convolution modules are concatenated into 512-dimensional features. Feature fusion and dimensionality adjustment are performed through a convolutional layer with a kernel size of 1×1 and 256 kernels. This is followed by a batch normalization layer, a ReLU activation function, a spatial dropout layer, and a SENet attention module, ultimately outputting 256-dimensional enhanced features.

[0102] Step S7: Construct the clinical metadata encoding module:

[0103] Step S71: Construct an age encoding submodule: Encode continuous age values ​​through a three-layer fully connected network. The first layer contains 16 neurons and uses the ReLU activation function and Dropout; the second layer contains 32 neurons and uses the ReLU activation function and batch normalization; the third layer contains 64 neurons and uses the ReLU activation function, outputting a 64-dimensional age feature vector.

[0104] Step S72: Construct a breast density encoding submodule: Map the breast density category to a 64-dimensional dense vector through an embedding layer; then further encode it through a two-layer fully connected network. The first layer contains 128 neurons and uses the ReLU activation function and Dropout; the second layer contains 64 neurons and uses the ReLU activation function to output a 64-dimensional density feature vector.

[0105] Step S73: Construct a metadata fusion layer: Concatenate the age feature vector and the density feature vector to obtain a 128-dimensional metadata feature vector; perform feature fusion through a convolutional layer with a kernel size of 1×1 and 128 kernels, followed by a batch normalization layer and a ReLU activation function to output the final metadata encoded features.

[0106] Step S8: Construct a multimodal fusion attention mechanism:

[0107] Step S81: Construct the image-metadata cross-attention module: Using image features as Q and metadata features as K and V, calculate the cross-attention weights through a multi-head attention mechanism; set the number of attention heads to 8, and the dimension of each head... The value is 64; the formula for calculating attention weight is...

[0108]

[0109] Step S82: Construct a dual-view self-attention module: Calculate self-attention for both the 512-dimensional CC feature and the 512-dimensional MLO feature. Each view uses a 12-head self-attention mechanism, with each head having a dimension of 64, resulting in a total hidden layer dimension of 768. The self-attention layer includes multi-head attention calculation, residual connections, layer normalization, a feedforward network, and further residual connections. The feedforward network uses the GELU activation function, and the dropout rate is set to 0.1.

[0110] Step S83: Construct a global context fusion module: Perform global information aggregation on the fused feature map. Global average pooling and global max pooling are used to extract global features, each generating a 512-dimensional vector. The two vectors are concatenated to a 1024-dimensional matrix, which is then transformed using a two-layer MLP: the first layer reduces the dimension to 256 using the ReLU activation function; the second layer restores the dimension to 512 and uses the Sigmoid activation function to generate attention weights. The weights are then multiplied element-wise with the original 512-dimensional features to achieve global context enhancement.

[0111] Step S9, as follows Figure 5 As shown, an adaptive weight fusion module is constructed:

[0112] Step S91: Construct a feature importance evaluation module: Evaluate channel attention for the input multimodal features. A 1664-dimensional vector is obtained through global average pooling. This vector is then processed through a two-layer fully connected network: the first layer reduces the dimension to 208 and uses ReLU activation; the second layer restores the dimension to 1664 and uses Sigmoid activation to generate channel attention weights. These weights are then multiplied channel-by-channel with the original features to achieve importance weighting.

[0113] Step S92: Construct a dynamic weight generation module: Design an adaptive weight network to learn the optimal fusion weights. The input is the weighted 1664-dimensional multimodal features, processed through a three-layer fully connected network: the first layer has 256 neurons, using ReLU activation function, batch normalization, and Dropout; the second layer has 128 neurons, using ReLU activation function and batch normalization; the third layer has 3 neurons, using Softmax activation function to ensure the weight sum is 1.

[0114] Step S93: Construct a weighted feature fusion module: Based on the learned weights , , The three modal features are weighted and fused. The fusion formula is as follows:

[0115]

[0116] in For local representation features, For global representation features, These are metadata features. To maintain dimensional consistency, first... and The linear layers are used to map to 1024 dimensions. The final output is a 1024-dimensional fused feature, which is then reduced to 512 dimensions through a final 1×1 convolution.

[0117] Step S10, as follows Figure 6 As shown, a cross-modal dual-view interactive convolutional neural network is constructed:

[0118] Step S101: Construct the network backbone structure: The dual-view feature extraction module receives the CC view image and the MLO view image, and outputs two 512-dimensional feature vectors;

[0119] Step S102: Construct parallel processing branches: The local cross-view interaction module and the global representation module process the dual-view features in parallel, generating 1024-dimensional local representation features and 512-dimensional global representation features respectively; the multi-scale feature enhancement module enhances the local representation features to 256 dimensions; the clinical metadata encoding module processes the patient's age and breast density, outputting 128-dimensional metadata features.

[0120] Step S103: Integrating the multimodal fusion module: The multimodal fusion attention mechanism receives local representation features, global representation features, and metadata features, and calculates and outputs 896-dimensional fused features through cross-attention and self-attention. The adaptive weight fusion module further integrates multimodal information and outputs the final 512-dimensional feature representation;

[0121] Step S104: Construct a classification decision module: This module contains a classifier consisting of two fully connected layers. The first layer maps 512-dimensional features to 256-dimensional features, using ReLU activation, batch normalization, and Dropout. The second layer outputs a 2-dimensional vector corresponding to the two categories of benign and malignant, using the Softmax activation function to generate a probability distribution.

[0122] Step S11: Design the combined loss function:

[0123]

[0124] in, The focus loss function addresses the data imbalance problem, and its specific expression is:

[0125]

[0126] in To balance the weights for each category, γ is the predicted probability of the true class, and γ is the focusing parameter set to 2.0, with greater weight given to difficult samples.

[0127] To compare the learning loss, a discriminative feature representation is learned, the specific expression of which is:

[0128]

[0129] in Let i be the feature representation of sample i. This represents the features of positive samples. Let P(i) be the feature representation of all samples, A(i) be the set of positive samples, sim be the cosine similarity, and τ be the temperature parameter.

[0130] The specific expression for the consistency loss in dual-view prediction, ensuring consistency between CC and MLO view prediction results, is as follows:

[0131]

[0132] in and These are the predicted probabilities for the two views, respectively. The mean of the two values ​​is MSE, where MSE is the mean squared error. Let KL divergence be denoted as KL divergence.

[0133] The loss weights were determined by grid search: α=0.6, β=0.3, δ=0.1.

[0134] Step S12, Network Training and Performance Evaluation: The Adam optimizer is used, with cosine annealing as the learning rate scheduling strategy. The initial learning rate is 0.0001, the batch size is 32, and the total number of training epochs is 60. The model performance is evaluated on the validation set every 10 epochs, and the best model is saved.

[0135] The effectiveness and advantages of this method are illustrated below through experiments (see [link]). Figure 7 The dataset used in this invention contains 9684 mammogram images, each labeled as healthy or diseased. To systematically verify the effectiveness of the model proposed in this invention, four comparative experiments were conducted: (1) Classification using only mammogram CC view images, with a classification accuracy of 85.2%; (2) Classification using only mammogram MLO view images, with a classification accuracy of 84.7%; (3) Classification using both mammogram CC and MLO view images, with a classification accuracy of 89.5%; (4) Classification using both mammogram CC and MLO view images combined with metadata, with a classification accuracy of 92.8%.

[0136] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0137] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of breast image classification methods that integrate dual-view information and clinical metadata under the guidance of this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A breast image classification method integrating dual-view information and clinical metadata, characterized in that, Includes the following steps: Step S1: Establish a breast image classification dataset; Step S2: Preprocess the mammogram images and clinical metadata in the dataset; Step S3: Construct a dual-view feature extraction module; Step S4: Construct a local cross-view interaction module; Step S5: Construct the global representation module; Step S6: Construct a multi-scale feature enhancement module; Step S7: Construct the clinical metadata encoding module; Step S8: Construct a multimodal fusion attention mechanism; Step S9: Construct an adaptive weight fusion module; Step S10: Construct a cross-modal dual-view interactive convolutional neural network; Step S11: Design the combined loss function; Step S12: Train a cross-modal dual-view interactive convolutional neural network and evaluate its performance.

2. The breast image classification method integrating dual-view information and clinical metadata according to claim 1, characterized in that, In step S1, mammograms of CC view and MLO view, along with corresponding clinical metadata such as patient age and breast density, are extracted from the Mini-DDSM dataset to establish a breast image classification dataset, which is then divided into training set, validation set, and test set in an 8:1:1 ratio. In step S2, the mammogram images are subjected to CLAHE contrast enhancement, noise reduction filtering, and ROI region extraction. Clinical metadata is standardized and normalized, and data augmentation techniques are used to expand the dataset.

3. The breast image classification method integrating dual-view information and clinical metadata according to claim 1, characterized in that, Step S3 is implemented as follows: Step S31: Construct the CC view feature extraction branch: Use the improved EfficientNetV2-S as the backbone network to generate a 512-dimensional feature representation; Step S32: Construct the MLO view feature extraction branch: Use the same network structure as the CC view to generate a 512-dimensional feature representation.

4. The breast image classification method integrating dual-view information and clinical metadata according to claim 1, characterized in that, Step S4 is implemented as follows: Step S41: Construct a cross-shape window attention mechanism: Divide the input features evenly into horizontal and vertical stripes, and apply the self-attention mechanism to each. Step S42: Construct a cross-view attention module: Achieve information interaction between views by exchanging the query matrices of the CC view and the MLO view; Step S43: Generate local feature representations: The features processed by cross-view attention are subjected to global average pooling to obtain local representation features.

5. The breast image classification method integrating dual-view information and clinical metadata according to claim 1, characterized in that, Step S5 is implemented as follows: Step S51: Extract global features: Concatenate the features of the CC view and MLO view along the channel dimension; Step S52: Apply global average pooling: Perform global average pooling on the concatenated features to obtain global representation features.

6. The breast image classification method integrating dual-view information and clinical metadata according to claim 1, characterized in that, Step S6 is implemented as follows: Step S61: Construct multi-scale convolutional branches: including four convolutional branches with kernel sizes of 1×1, 3×3, 5×5, and 7×7 respectively; Step S62: Construct a dilated convolutional module: Use three dilated convolutional layers with dilation rates of 2, 4, and 8 respectively; Step S63: Construct a feature fusion layer: Fuse multi-scale features through a convolutional layer with a kernel size of 1×1.

7. The breast image classification method according to claim 1, characterized in that, Step S7 is implemented as follows: Step S71: Construct an age encoding submodule: Use a three-layer fully connected network to encode age, with 16, 32, and 64 nodes respectively, and ReLU activation function; Step S72: Construct a breast density encoding submodule: Use an embedding layer to map the density category to a 64-dimensional vector, and then encode it through two fully connected network layers with 128 and 64 nodes respectively. Step S73: Construct a metadata fusion layer: fuse age and density features through a convolutional layer with a kernel size of 1×1 and 128 channels.

8. The breast image classification method according to claim 1, characterized in that, Step S8 is implemented as follows: Step S81: Construct the image-metadata cross-attention module: Calculate the attention weights of image features and metadata features using the Query, Key, and Value matrices, with 8 attention heads; Step S82: Construct a dual-view self-attention module: Calculate self-attention for the features of CC and MLO views respectively, with 16 attention heads and 768 hidden layer dimensions; Step S83: Construct a global context fusion module: global average pooling and global max pooling are used to extract global features, and feature transformation is performed through a multilayer perceptron.

9. The breast image classification method integrating dual-view information and clinical metadata according to claim 1, characterized in that, Step S9 is implemented as follows: Step S91: Construct a feature importance evaluation module: Calculate feature importance scores using a convolutional layer with a kernel size of 1×1 and 512 channels; Step S92: Construct a dynamic weight generation module: Generate adaptive weights through a three-layer fully connected network with 256, 128, and 3 network nodes respectively; Step S93: Construct a weighted feature fusion module: Perform weighted fusion of local representation features, global representation features, and metadata features according to their weights. The fusion formula is as follows: in, For local representation features, For global representation features, Metadata characteristics, , , These are the corresponding adaptive weights.

10. A breast image classification method integrating dual-view information and clinical metadata according to claim 1, characterized in that, In step S11, the specific expression of the combined loss function is as follows: in, For focus loss function, To compare the learning loss function, Let be the consistency loss function, and α, β, and δ be the weighting coefficients, respectively.

Images