Breast cancer focus benign and malignant discrimination method based on gated multi-expert mechanism

Abstract

The invention belongs to the technical field of medical image intelligent diagnosis, and provides a breast cancer focus benign and malignant discrimination method based on a gated multi-expert mechanism. The method comprises the following steps: firstly, carrying out standardization and semantic preprocessing on a mammary gland X-ray image, a BI-RADS imaging report and structured clinical data, embedding age, mammary gland density and focus position information into a text template in a natural language form, and realizing unified expression of multi-modal input; secondly, extracting image features by utilizing a ResNet network and a simplified CLIP model, obtaining a text semantic vector by adopting a Bio-ClinicalBERT model, and establishing two sub-paths of a lump expert and a calcification expert in a Transform structure; further, an expert weight is dynamically generated through a gating routing mechanism, and soft routing fusion is executed; and finally, outputting benign and malignant results of the breast cancer focus by the binary classification module. According to the method, deep fusion and dynamic collaboration of the mammary gland X-ray image, the BI-RADS text and the clinical information are realized, and the accuracy and interpretability of breast cancer discrimination can be remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of medical image analysis and artificial intelligence diagnostic technology, and provides a method for distinguishing between benign and malignant breast cancer lesions based on a gated multi-expert mechanism. Background Technology

[0002] Breast cancer is one of the most common malignant tumors among women worldwide, and its early detection and accurate diagnosis are of great significance for improving cure rates and reducing mortality. In recent years, multimodal deep learning methods have become a research hotspot. By fusing image features and textual information, these methods can not only improve the accuracy of breast cancer lesion identification but also enhance the clinical interpretability of the model to a certain extent.

[0003] In the field of multimodal breast cancer lesion discrimination, although approaches based on joint image and text modeling have been gradually explored, several shortcomings still need to be addressed. First, existing methods generally lack the ability to differentiate features across different modalities in their model structure, mostly employing a unified network or simple fusion strategy, failing to fully leverage the complementarity of images and text in representing lesion information. Second, current multimodal models often over-rely on a single expert or a single fusion method when processing complex clinical data, leading to insufficient adaptability to cases, especially when faced with missing modal information, imbalanced distribution, or anomalous samples, where model performance significantly deteriorates. Third, although cross-modal representation methods demonstrate advantages in image-text alignment, their application in medical scenarios remains limited, such as insufficient fine-grained semantic understanding of long text reports and inaccurate capture of local lesion features in images. Furthermore, existing research remains simplistic in its multimodal fusion mechanisms, lacking dynamic expert selection strategies and failing to achieve flexible discrimination processes tailored to different case characteristics.

[0004] To address the problems of limited multimodal fusion methods, insufficient cross-modal alignment, and lack of dynamic expert mechanisms in existing technologies, this invention proposes a method and system for distinguishing benign and malignant breast cancer lesions based on a gated multi-expert mechanism. Summary of the Invention

[0005] The purpose of this invention is to provide a method for distinguishing between benign and malignant breast cancer lesions based on a gated multi-expert mechanism. This method integrates a multi-layer cross-attention mechanism with a gated multi-expert dynamic fusion strategy, enhancing the model's feature representation and discrimination capabilities in the task of determining the benignity and malignancy of breast cancer lesions. It achieves deep interaction between different modalities and adaptive integration of type-specific knowledge. Leveraging its dynamic weight control and semantic alignment characteristics, this method exhibits higher adaptability and robustness when facing multiple types of lesions and multi-source clinical information.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for determining the benign or malignant nature of breast cancer lesions based on a gated multi-expert mechanism includes the following steps:

[0008] S1. Download the breast image dataset;

[0009] S2. Preprocess the dataset to generate two types of image samples and three types of text samples. Calculate the centroid coordinates of breast tissue in the image samples, and embed them into the text samples after intervalization.

[0010] S3. Vectorize the image using a pre-trained image encoder to generate global feature vectors and feature maps. The unprocessed original image is encoded by different image encoders, and the resulting vector is used as the query vector. The three types of text samples are encoded using a pre-trained text encoder.

[0011] S4. The text vector describing the breast cancer mass and the text vector describing calcification are fused with the image feature map vector respectively. The resulting fused vector is used as two sets of memory vectors of the expert subsystem. It is input into the expert subsystem based on the Transformer architecture along with the query vector in step 3. This drives the image-driven multi-layer cross-attention mechanism to achieve semantic alignment with image as the main focus and text as the guide, thereby enhancing the semantic sensitivity and localization accuracy of local lesions.

[0012] S5. A gated router is constructed using the embedded representations of global image features and global text features as input. Gating factors corresponding to the tumor expert and calcification expert are generated through nonlinear transformation and activation function. After normalization, weight parameters that satisfy complementary constraints are obtained to dynamically adjust the importance of each expert's output. The tumor and calcification decoder outputs are weighted and synthesized into a unified feature representation.

[0013] S6. Input the weighted and fused multimodal fusion feature vector into the binary classification prediction module. This module and the gated network both adopt the form of a multilayer perceptron in terms of structure. It is used to map the high-dimensional fusion features to the binary classification probability space. The last layer is activated by the Sigmoid function and outputs the probability value of whether the breast cancer lesion is benign or malignant.

[0014] As a further optimization, in step S1, the downloaded image dataset specifically refers to the TOMPEI-CMMD dataset;

[0015] The TOMPEI-CMMD dataset includes mammogram data and BI-RADS descriptors for the images, as well as patients' clinical data, including age and breast density.

[0016] As a further optimization, in step S2, the preprocessing includes breast region alignment, tissue edge detection and cropping, size normalization, original image scaling, BI-RADS descriptor templated, clinical data intervalization, and template embedding.

[0017] The breast region alignment refers to flipping all images marked "R" horizontally along the vertical axis so that the left and right breasts maintain a consistent orientation in visual presentation.

[0018] The tissue edge detection and cropping refers to using a threshold-growth-based edge detection algorithm to extract the breast tissue region in the image and retain only the part relevant to diagnosis.

[0019] The size normalization refers to filling the edges of images that are not wide enough after cropping to make them reach a consistent width standard. After filling, all images are adjusted proportionally to a size of 1400×1024 to match the aspect ratio of the breast region after edge detection.

[0020] The scaling operation of the original image refers to scaling the unprocessed original image to 224×224, thereby forming standardized data specifically for input to the image encoder;

[0021] The BI-RADS descriptor template specifically refers to filling the BI-RADS descriptor keywords in the dataset into the corresponding template, thereby generating a structured text representation related to the lesion type;

[0022] The clinical data intervalization and template embedding specifically refer to discretizing age into several intervals through percentile binning and embedding them into templates in the form of phrases to ensure that numerical features can participate in subsequent encoding in a textual way, while breast density is expressed in a standardized manner according to the density category grading system.

[0023] As a further optimization, the two types of image samples and the three types of text samples are specifically preprocessed image samples, original image samples scaled to 224×224, descriptive text containing only calcified lesions, descriptive text containing only mass lesions, and descriptive text containing both types of lesions.

[0024] The centroid coordinates of the breast tissue specifically refer to the centroid calculation of the preprocessed breast X-ray image sample, taking the average of the coordinates of all pixels in the tissue region as the centroid coordinates of the image sample. Let Ω be the effective pixel set of the breast region, then the center coordinates can be expressed as:

[0025]

[0026] The centroid coordinates are (x c ,y c Then normalize:

[0027]

[0028] Where W and H are the image width and height respectively, the obtained x', y'∈[0,1], then the normalization result is processed into intervals, and the interval labels are converted into natural language phrases (such as "located around 20–40% in the horizontal axis") and embedded into the text template;

[0029] As a further optimization, in step S3, the preprocessed image encoder is a ResNet with the fully connected layers removed, combined with a hollow spatial pyramid pooling structure, to obtain a feature representation that combines global information and spatial distribution.

[0030] Both the original image encoder and the text encoder adopt a simplified version of the SimpleCLIP architecture. The pre-trained weights are initialized based on the ViT-B / 32 model to ensure the consistency of cross-modal feature representation and the transferability of the pre-trained semantic space.

[0031] As a further optimization, in step S4, the embedding dimensions of calcification and tumor text are both 512, and they are mapped to a 256-dimensional latent feature space via a linear projection layer; at the same time, the image feature map is flattened into a one-dimensional vector with a feature dimension of 256, and then the text features and image features are summed element by element to achieve a fusion representation of cross-modal features in a unified semantic space.

[0032] The expert subsystem based on the Transformer architecture consists of three parts: a multi-layer cross-attention module, a normalization layer, and a feedforward network. Finally, stable feature reconstruction and aggregation are achieved through residual connections and layer normalization.

[0033] Let the spatial characteristics of the input be:

[0034]

[0035] Where B is the batch size, N represents the number of spatial features, and D is the feature dimension;

[0036] Each expert corresponds to a learnable query vector:

[0037]

[0038] The module aims to obtain the expert's aggregated feature representation by interacting q with spatial features X through a cross-attention mechanism.

[0039]

[0040] The cross-attention submodule consists of a multi-layer cross-attention structure, with each layer employing a multi-head attention mechanism. The input dimension is D = 256, and the number of heads is 8. Within each layer, the expert query vector (Query) and spatial features (Key and Value) engage in semantic interaction to calculate the relevance of different spatial locations to the expert's semantics. The calculation form is as follows:

[0041]

[0042] in As a learnable projection matrix, the attention output o obtained from this operation represents the weighted aggregation result of experts in the spatial dimension. The updated expert features are obtained through residual connections and layer normalization:

[0043] q' = LayerNorm(q+o)

[0044] This process is repeated at each layer, stacked in two layers, refining the expert's attention focus and feature representation layer by layer. The update rule for the l-th layer is:

[0045] q (l) =LayerNorm(q (l-1) +Attention(q (l-1) ,X))

[0046] After multi-layer cross-attention, a feed-forward network (FFN) is used to enhance the non-linear expressive power of the features:

[0047] z'=FFN(q (L) ) = W2·GELU(W1q (L) +b1)+b2

[0048] in And the activation function is GELU;

[0049] The final output is in residual normalized form: z = LayerNorm(q) (L) +z'), where z is the final aggregation feature of the experts;

[0050] As a further optimization, in step S5, the gated router consists of an input splicing layer, a gated mapping layer, and a Softmax normalization layer;

[0051] The input concatenation layer concatenates the global image features with the global text features to obtain a joint feature vector:

[0052]

[0053] The fused features are first mapped to a 128-dimensional latent space through a linear transformation, and then nonlinearly transformed using layer normalization and the ReLU activation function to enhance the stability and expressive power of the feature distribution.

[0054] h1 = ReLU(LayerNorm(W1f) comb +b1))

[0055] To prevent overfitting and improve the model's generalization ability, a Dropout operation is introduced after this layer. Subsequently, the network further compresses the dimension to 64 through a second linear mapping layer, and then undergoes ReLU activation.

[0056] h2 = ReLU(W2h1 + b2)

[0057] The final linear transformation outputs two unnormalized gating scores (logits):

[0058]

[0059] Two components correspond to the weighted predictions of the mass expert and the calcification expert, respectively. Then, the softmax function is used to normalize the score into a probabilistic form.

[0060] w = softmax(z) = [w mass ,w calc ],w mass +w calc =1

[0061] Among them, w mass and w calc These two weights represent the gating network's level of trust in the tumor and calcification experts given the current input. They will be used in the subsequent expert output fusion stage for weighted summation to form the final prediction result.

[0062] y = w mass ·y mass +w calc ·y calc

[0063] As a further optimization, in step S6, both the binary classification prediction module head and the gated network adopt a multilayer perceptron structure, and layer normalization, ReLU activation and Dropout regularization are introduced between each layer. This consistency in structural design aims to maintain the consistency of feature transformation and training stability. However, there are differences in the implementation details. The classification head adopts three linear layers (the dimension changes from D→128→64→1), and layer normalization and a high proportion of Dropout are introduced in the first two layers to enhance the stability and generalization performance of the model.

[0064] The beneficial effects of this invention are:

[0065] This invention introduces a gated multi-expert mechanism in the multimodal feature fusion process, establishing independent expert representations for two types of lesions: masses and calcifications. Soft-routing fusion is achieved through dynamic weighting of gated units, enabling the model to highlight the role of the primary expert in single-lesion cases and integrate the advantages of both types of experts in mixed-lesion cases. This not only improves the accuracy and robustness of the classification results but also enhances the rationality of expert allocation and the interpretability of model predictions, thus better meeting the actual clinical diagnostic needs. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of a breast X-ray image containing only a lump in an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of a mammary X-ray image containing only calcification in an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of a breast X-ray image that includes both a mass and calcification in an embodiment of the present invention;

[0069] Figure 4 This is an operation flowchart of an embodiment of the present invention;

[0070] Figure 5 This is a schematic diagram of the modules of the present invention; Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0072] This embodiment provides a method for distinguishing between benign and malignant breast cancer lesions based on a gated multi-expert mechanism, including the following steps:

[0073] S1. Download the breast image dataset;

[0074] S2. Preprocess the dataset to generate two types of image samples and three types of text samples. Calculate the centroid coordinates of breast tissue in the image samples, and embed them into the text samples after intervalization.

[0075] S3. Vectorize the image using a pre-trained image encoder to generate global feature vectors and feature maps. The unprocessed original image is encoded by different image encoders, and the resulting vector is used as the query vector. The three types of text samples are encoded using a pre-trained text encoder.

[0076] S4. The text vector describing the breast cancer mass and the text vector describing calcification are fused with the image feature map vector respectively. The resulting fused vector is used as two sets of memory vectors of the expert subsystem. It is input into the expert subsystem based on the Transformer architecture along with the query vector in step 3. This drives the image-driven multi-layer cross-attention mechanism to achieve semantic alignment with image as the main focus and text as the guide, thereby enhancing the semantic sensitivity and localization accuracy of local lesions.

[0077] S5. A gated router is constructed using the embedded representations of global image features and global text features as input. Gating factors corresponding to the tumor expert and calcification expert are generated through nonlinear transformation and activation function. After normalization, weight parameters that satisfy complementary constraints are obtained to dynamically adjust the importance of each expert's output. The tumor and calcification decoder outputs are weighted and synthesized into a unified feature representation.

[0078] S6. Input the weighted and fused multimodal fusion feature vector into the binary classification prediction module. This module and the gated network both adopt the form of a multilayer perceptron in terms of structure. It is used to map the high-dimensional fusion features to the binary classification probability space. The last layer is activated by the Sigmoid function and outputs the probability value of whether the breast cancer lesion is benign or malignant.

[0079] The specific implementation of step S1 is as follows:

[0080] The downloaded dataset is the publicly available TOMPEI-CMMD breast cancer dataset; this dataset contains clinically confirmed mammogram images and corresponding BI-RADS descriptors and clinical information; the breast images in the dataset are all stored in DCOM format; for each case, this embodiment only selects breast images from the MLO view; the BI-RADS descriptors include six categories: Location, Shape, Margin, Density, Morphology, and Distribution; the clinical information includes the patient's age and overall breast density;

[0081] The specific implementation of step S2 is as follows:

[0082] The preprocessing of the dataset specifically involves horizontally flipping the image labeled as the right breast along the vertical axis; then, a threshold-growing edge detection algorithm is used to extract the breast tissue region, retaining only the diagnostically relevant tissue portions, and eliminating background and non-tissue noise regions through edge cropping. For images with inconsistent cropped sizes, edge padding is used to unify the width, and all images are scaled proportionally to 1400×1024 pixels to maintain a consistent aspect ratio in the breast region; furthermore, to meet the input requirements of the subsequent encoder, the original images are scaled to 224×224 copies to form standardized image samples for feature extraction.

[0083] For text and clinical data, BI-RADS descriptors are templated by filling keywords in the descriptions into predefined semantic templates, thus forming structured phrases that can be directly input into the text encoder. At the same time, numerical attributes in clinical data are intervalized and textualized. Age is discretized into several intervals using percentile binning and embedded into templates in natural language form. Breast density is converted into text phrases according to a standard classification system, enabling clinical data to participate in subsequent feature fusion in the form of a linguistic modality.

[0084] To improve the spatial semantic consistency between text and images, the centroid coordinates of the preprocessed breast region are calculated. The average coordinates of the effective tissue pixels are taken as the center position of the sample, and the obtained coordinates are normalized according to the image width and height. The normalized results are further intervalized and transformed into natural language phrases (e.g., "located around 20–40% in the horizontal axis"), which are then embedded into the BI-RADS template text, thereby explicitly introducing spatial location information in the language modality.

[0085] After completing the semantic embedding of BI-RADS descriptors, clinical data, and centroid coordinates, the text samples are divided into three types: those containing only calcifications, those containing only masses, and those containing both. In this embodiment, the ratio of the training set to the validation set is 8:2.

[0086] The specific implementation of step S3 is as follows:

[0087] Furthermore, to achieve unified representation and deep semantic interaction between image and text modalities, feature encoding and cross-modal mapping are performed on the preprocessed mammogram image and the templated BI-RADS description text, respectively. The image encoder adopts a ResNet34 structure with the fully connected layers removed and is combined with a Spatial Pyramid Pooling (ASPP) module. The ASPP template has 512 input channels, 256 output channels, and a dilation rate list [6, 12, 18]. Convolution, normalization, and activation operations are performed on each dilation rate. The convolution size is 3×3 to enhance the perception of spatial details while maintaining global semantic information. After processing by this encoder, the global feature vector and spatial feature map of the image can be obtained simultaneously.

[0088] For unprocessed raw image samples, this embodiment uses a simplified CLIP architecture encoder to extract features and generate query vectors corresponding to text features; this architecture is based on ViT-B / 32 pre-trained weight initialization; for text samples, this embodiment uses the Bio-ClinicalBERT encoder.

[0089] To ensure the fusionability of cross-modal features in a unified latent space, this embodiment performs a linear mapping on the encoding results, projecting both the mass description text vector and the calcification description text vector onto a 256-dimensional latent feature space. After the image feature map is flattened into a one-dimensional vector of the same dimension, it is summed element-wise with the mass and calcification text features to generate two sets of memory vectors.

[0090] The specific implementation of step S4 is as follows:

[0091] Furthermore, a multi-expert subsystem based on the Transformer architecture is constructed. This subsystem consists of two parts: a mass expert and a calcification expert. Each part receives fused features as input to perform feature modeling and semantic enhancement for different types of breast lesions. The model achieves division of labor learning and complementary representation of features of different lesion types through the multi-expert structure, thereby improving the overall accuracy and robustness of the discrimination.

[0092] In this embodiment, the expert module training uses a feature embedding dimension of 256, 8 attention heads, 2 attention layers, and a learning rate of 1×10⁻⁶. -4 The Adam optimization algorithm is used, with a weight decay coefficient of 0.05.

[0093] During training, the model incorporates expert contrast loss during the optimization phase to constrain the tumor expert output F. mass With calcification experts output F calc To mitigate the differences in semantic space, this loss function prevents two types of experts from exhibiting excessively high similarity at the feature level. The loss is measured by cosine similarity, assuming the expert outputs, after L2 normalization, are respectively... and The expert comparison loss is defined as:

[0094]

[0095] Where B is the batch size and D is the feature dimension (256 in this embodiment); the goal of minimizing the loss is to reduce the cosine similarity between the outputs of the two experts, thereby enabling the model to achieve semantic decoupling during the feature learning process, so that the mass expert focuses on the image morphology and edge features, and the calcification expert focuses on the dot-like high-density areas and distribution patterns, and the two complement each other.

[0096] The specific implementation of step S5 is as follows:

[0097] In this embodiment, a router structure based on a gating mechanism is constructed to achieve adaptive integration and dynamic balancing of different expert output features. The main function of the gating router is to adaptively adjust the weights of the tumor expert and calcification expert outputs in the final fused features according to the semantic features of the input samples, thereby achieving dynamic selection and complementary fusion of information in multi-type lesion discrimination tasks.

[0098] The gated router consists of an input concatenation layer, a gated mapping layer, and a Softmax normalization layer. The input concatenation layer concatenates the global feature vector of the image and the global feature vector of the text along the feature dimension to form a joint feature representation, denoted as:

[0099]

[0100] Where f vis For global features of the image, f text Text feature embedding is performed. The concatenated joint feature dimension is 512. It is first projected to a 128-dimensional latent space through a linear mapping layer, and then nonlinearly transformed by layer normalization and ReLU activation function. To prevent overfitting and improve the model's generalization performance, Dropout (P=0.2) is introduced after this layer. Subsequently, the features are further compressed to 64 dimensions through a second linear mapping layer and then activated again by ReLU to obtain a high-order representation of the gating score. Finally, two unnormalized gating scores (logits) are output through a linear layer, corresponding to the weight predictions of the tumor expert and calcification expert, respectively. Softmax normalization is performed on the score vector to obtain the gating weights. The entire gating soft selection mechanism can be represented as follows:

[0101] w=softmax(FC3(RELU(FC2(Dropout(ReLU(LayerNorm(FC1(f comb )))))))

[0102] During training, the model introduces an expert equilibrium loss constraint during the optimization phase to regularize the expected distribution of expert weights, encouraging a relatively balanced selection tendency across the global sample distribution. Let the average weights of the gating network for the tumor expert and the calcification expert in the current batch be respectively... and The expert equilibrium loss is defined as:

[0103]

[0104] The loss is set to a target value of 0.5 to encourage the network to maintain the overall weight symmetry between the two experts, prevent the model from over-relying on the output of a single expert, and thus maintain a balance in the ability to distinguish different lesion types.

[0105] Meanwhile, the gating layer and the expert subsystem share a unified learning rate setting, with the gating network learning rate being 1×10⁻⁶. -4 The optimization algorithm uses Adam with a weight decay coefficient of 0.05.

[0106] The specific implementation of step S6 is as follows:

[0107] In this embodiment, the classification head adopts a multilayer perceptron structure, which is consistent with the gating network in form, so as to ensure the structural consistency and training stability of the feature transformation method.

[0108] The classification head consists of three fully connected layers, with the feature dimensions changing sequentially from 256 to 128 to 64 to 1. The first two linear layers undergo non-linear transformation using layer normalization and the ReLU activation function, and a Dropout mechanism is introduced after each layer to prevent overfitting and enhance the model's generalization performance. The final layer outputs a single-dimensional predicted value, which is mapped to the interval [0,1] using the Sigmoid function to obtain the predicted probability that the breast cancer lesion is a malignant sample. Based on this probability value, the model achieves binary classification output by setting a threshold. The default threshold is 0.5. When the predicted value is greater than this threshold, it is judged as malignant; otherwise, it is judged as benign.

[0109] During training, the classification module, expert subsystem, and gating network together form an end-to-end optimization structure. To address the imbalance in the distribution of benign and malignant samples in breast cancer imaging data, this embodiment uses FocalLoss as the primary supervised loss function. Let the predicted probability be P and the true label be y, then the loss is defined as:

[0110] L focal =-α t (1-p t ) γ log(p t )

[0111] Where p t =p, when y=1, p t =1-p; when y=0, α t γ is the class weight (set to 0.25), and γ is the adjustment factor (set to 2.0);

[0112] In addition to the primary supervised loss, the model also employs a combination of expert equilibrium loss and expert contrast loss for multi-objective optimization. The equilibrium loss takes the average gating weights of the tumor and calcification experts as input, constraining their distribution to near 0.5 to encourage balanced cooperation among experts during long-term training. The contrast loss calculates the semantic differences between the outputs of the two types of experts based on cosine similarity, minimizing similarity to achieve feature decoupling and complementary learning. These three losses work together on the model's overall optimization objective, and the combined loss function is defined as:

[0113] L total =L focal +λ1L balance +λ2L contrast

[0114] The weighting parameters are λ1 = 0.1 and λ2 = 0.05.

[0115] For configuration optimization, this embodiment uses the Adam optimizer with a learning rate of 1×10⁻⁶. -4 The weight decay coefficient is 0.05; the batch size is set to 8 during training, the total number of training rounds is 50, and an early stopping strategy is adopted to prevent overfitting. When the validation set loss decreases by less than 0.001 for 15 consecutive rounds, the training process is automatically terminated; all training tasks are completed on GPU devices with CUDA support to ensure computational acceleration and numerical stability.

[0116] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various modifications and variations.

Claims

1. A method for determining the benign or malignant nature of breast cancer lesions based on a gated multi-expert mechanism, characterized in that, Follow these steps: S1. Download the breast image dataset; S2. Preprocess the dataset to generate two types of image samples and three types of text samples. Calculate the centroid coordinates of breast tissue in the image samples, and embed them into the text samples after intervalization. S3. Vectorize the image using a pre-trained image encoder to generate global feature vectors and feature maps. The unprocessed original image is encoded by different image encoders, and the resulting vector is used as the query vector. The three types of text samples are encoded using a pre-trained text encoder. S4. The text vector describing the breast cancer mass and the text vector describing calcification are fused with the image feature map vector respectively. The resulting fused vector is used as two sets of memory vectors of the expert subsystem. It is input into the expert subsystem based on the Transformer architecture along with the query vector in step 3. This drives the image-driven multi-layer cross-attention mechanism to achieve semantic alignment with image as the main focus and text as the guide, thereby enhancing the semantic sensitivity and localization accuracy of local lesions. S5. A gated router is constructed using the embedded representations of global image features and global text features as input. Gating factors corresponding to the tumor expert and calcification expert are generated through nonlinear transformation and activation function. After normalization, weight parameters that satisfy complementary constraints are obtained to dynamically adjust the importance of each expert's output. The tumor and calcification decoder outputs are weighted and synthesized into a unified feature representation. S6. Input the weighted and fused multimodal fusion feature vector into the binary classification prediction module. This module and the gated network both adopt the form of a multilayer perceptron in terms of structure. It is used to map the high-dimensional fusion features to the binary classification probability space. The last layer is activated by the Sigmoid function and outputs the probability value of whether the breast cancer lesion is benign or malignant.

2. The method according to claim 1, characterized in that... : The breast image dataset includes mammogram data and BI-RADS descriptors for the images, as well as the patient's clinical data, including age and breast density; the preprocessing methods include left and right alignment of breast regions, tissue edge detection and cropping, size normalization, scaling, BI-RADS descriptor templated, clinical data intervalization, and template embedding. The BI-RADS descriptor template refers to filling the BI-RADS descriptor into the corresponding keyword template to generate a structured text description related to the lesion type; the clinical data intervalization and template embedding refers to discretizing age into several intervals through percentile binning and embedding them into the template in the form of phrases; and breast density is expressed in BI-RADS standardization according to the density category grading system. The centroid coordinates are obtained by taking the average of the coordinates of all tissue region pixels as the centroid coordinates of the image sample. Then, the normalization result is processed into intervals, and the interval labels are converted into natural language phrases and embedded into the text template. After processing using the above method, two types of image samples and three types of text samples are finally generated; The two types of image samples are: preprocessed image samples and unprocessed image samples that only underwent scaling operations. The three types of text samples are: descriptive text containing only calcified lesions, descriptive text containing only mass lesions, and descriptive text containing both types of lesions; each type of text contains BI-RADS text description, age text description, breast density category description, and centroid coordinate description.

3. The method according to claim 1, characterized in that... : The pre-trained image encoder is a ResNet34 with pre-trained weights, and it uses a dilated spatial pyramid pooling (ASPP) structure to encode the image; the global feature vector and feature map are features of the pre-processed image samples. The different image encoders are simplified versions of the SimpleCLIP architecture, and the pre-trained weights are initialized based on the ViT-B / 32 model; the pre-trained text encoder is a Bio-ClinicalBERT encoder; and the query vector is a vector generated by the SimpleCLIP architecture.

4. The method according to claim 1, characterized in that... : The memory vector is generated by projecting the mass description text vector and the calcification description text vector to a low dimension; after the image feature map is flattened into a one-dimensional vector of the same dimension, it is summed element-wise with the mass and calcification text features respectively to generate two sets of memory vectors. The two sets of memory vectors are respectively input into two different expert subsystems, one for calcification and the other for mass; the two expert systems share the query vector and finally generate two different fusion features for calcification and mass. The expert subsystem based on the Transformer architecture consists of three parts: a multi-layer cross-attention module, a normalization layer, and a feedforward network. Finally, stable feature reconstruction and aggregation are achieved through residual connections and layer normalization. The multi-layer cross-attention module has the following form for each layer: Q l '=MHA(Q l-1 ,X,X) Q l =LayerNorm(Q l-1 +Q l ') Q is the query vector, X is the memory vector, and MHA stands for Multi-Head Attention, defined as: MHA(Q,K,V)=Concat(h1,…,h H )W O H represents the number of attention heads. W is a linear mapping matrix. O To output the mapping matrix; Attention weights for each layer's output: After stacking the cross-attention layers, the final output Q is obtained. L Nonlinear mapping is performed through a feedforward network: Z=FFN(Q L )=W2,GELU(W1Q L +b1)+b2 And perform residual connection and normalization: Y=LayerNorm(Q L +Z) The entire module can be summarized as follows: in:

5. The method according to claim 1, characterized in that... : The gated router consists of an input splicing layer, a gated mapping layer, and a Softmax normalization layer. The input concatenation layer concatenates the global image features with the global text features to obtain a joint feature vector: The fused features are first mapped to a low-dimensional space through a linear transformation, and then nonlinearly transformed using layer normalization and the ReLU activation function to enhance the stability and expressive power of the feature distribution. h1=ReLU(LayerNorm(W1f comb +b1)) To prevent overfitting and improve the model's generalization ability, a Dropout operation is introduced after this layer; subsequently, the network further compresses the dimensionality through a second linear mapping layer, and then undergoes ReLU activation. h2 = ReLU(W2h1 + b2) The final linear transformation outputs two unnormalized gating scores (logits): Two components correspond to the weighted predictions of the mass expert and the calcification expert, respectively. Then, the softmax function is used to normalize the score into a probabilistic form. w=softmax(z)=[w mass ,In calc ],In mass +in calc =1 Among them, w mass and w calc These two weights represent the gating network's level of trust in the tumor and calcification experts given the current input; they will be used for weighted summation in the subsequent expert output fusion stage to form the final prediction result. y=w mass ·y mass +w calc ·y calc 6. The method according to claim 1, characterized in that... : The binary classification prediction module adopts a multilayer perceptron structure, which is consistent with the gating network in form. Layer normalization and a high proportion of Dropout are introduced in the first two layers to enhance the stability and generalization performance of the model. This module contains three fully connected layers, with the feature dimensions changing sequentially from 256 to 128 to 64 to 1. The first two linear layers undergo non-linear transformation through layer normalization and the ReLU activation function, and a Dropout mechanism is introduced after each layer to prevent overfitting and enhance the model's generalization performance. The final layer outputs a single-dimensional predicted value, which is mapped to the interval [0,1] through the Sigmoid function to obtain the predicted probability that the breast cancer lesion is a malignant sample. Based on this probability value, the model achieves binary classification output by setting a threshold.

Images