Multi-modal information collaborative breast cancer auxiliary identification method

Abstract

The invention discloses a multi-modal information collaborative breast cancer auxiliary identification method. The method comprises the following steps: S1, acquiring a breast ultrasonic image, a breast X-ray image and clinical text data; s2, carrying out pretreatment operation; s3, segmenting the breast ultrasonic image to obtain a breast ultrasonic segmentation mask graph; s4, performing feature extraction operation on the breast ultrasonic image and the breast ultrasonic segmentation mask graph; s5, acquiring enhanced ultrasonic joint features according to feature extraction results of the breast ultrasonic image and the breast ultrasonic segmentation mask graph; s6, performing feature extraction on the mammary gland X-ray image to obtain X-ray image features; the image fusion feature is fused with the enhanced ultrasonic joint feature to obtain an image fusion feature; s7, performing feature extraction on the clinical text data to obtain clinical text features; the multi-modal features are fused with the image fusion features to obtain multi-modal features; and S8, inputting the multi-modal features into an MLP network model for cancer identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical imaging, in particular to a breast cancer auxiliary identification method based on multi-modal information cooperation. BACKGROUND

[0002] At present, in the field of intelligent auxiliary identification technology for breast cancer, the existing technology has obvious shortcomings. In the aspect of single-modal feature extraction, for breast ultrasound images, the existing basic network cannot fully capture the spatial correlation between local details such as lesion edge burr degree and internal echo texture and the surrounding tissue, and does not construct a cooperative feature mining mechanism between segmentation mask and original image, so the detection capability for small lesions in dense breast tissue is insufficient; when processing mammography, the fixed convolution kernel size is used, which is difficult to balance the local details of small calcification clusters and the global feature extraction of glandular structure, and an effective attention mechanism is not designed for high-density glandular background interference, so the calcification cluster features are easily covered and the recognition accuracy is low; single screening or evaluation method is usually used to extract clinical text features, a multi-dimensional feature evaluation system is not constructed, and semantic normalization is not combined with medical terminology dictionary, resulting in feature redundancy and insufficient key information weight proportion. In the aspect of cross-modal feature fusion, the existing schemes are mostly limited to feature splicing, simple weighting and other shallow fusion stages, and the deep semantic association between different modalities is not established, so the cooperative advantages of multi-modal data cannot be fully utilized, and it is difficult to meet the requirements of clinical precise diagnosis on technology. SUMMARY

[0003] The purpose of the present application is to provide a breast cancer auxiliary identification method based on multi-modal information cooperation, which can assist doctors in carrying out breast cancer diagnosis work.

[0004] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0005] A breast cancer auxiliary identification method based on multi-modal information cooperation, comprising the following steps:

[0006] S1, collecting breast ultrasound images, breast X-ray images and clinical text data of a patient;

[0007] S2, performing pretreatment operation on the clinical text data, breast ultrasound images and breast X-ray images;

[0008] S3, using a U-Net model with fusion attention mechanism to segment the pretreated breast ultrasound images to obtain a breast ultrasound segmentation mask graph;

[0009] S4, using a ResNet50 network model to perform feature extraction operation on the breast ultrasound images to obtain original ultrasound features and bottom-level visual features; using the ResNet50 network model to perform feature extraction operation on the breast ultrasound segmentation mask graph to obtain segmentation graph features and lesion morphology features;

[0010] S5, preliminarily fuse the bottom layer visual features with the lesion morphology features, and input the fusion result, original ultrasound features and segmentation map features into a Vision Transformer model to mine intra-modal correlation information and obtain enhanced ultrasound joint features;

[0011] S6, using a DenseNet-121 network model to extract features from the preprocessed breast X-ray images to obtain X-ray image features; inputting the enhanced ultrasound joint features and the X-ray image features into a cross-modal fusion module to establish an inter-image anatomical structure correlation and obtain image fusion features;

[0012] S7, using a BERT model to extract features from the preprocessed clinical text data to obtain clinical text features; designing a cross-modal attention distillation module to fuse the image fusion features and the clinical text features to realize deep collaboration of semantic and visual information and obtain deeply fused multi-modal features;

[0013] S8, inputting the deeply fused multi-modal features into an MLP network model to realize normal, attention, Luminal A type, Luminal B type, HER2 positive type and triple negative breast cancer subtype identification and outputting corresponding probability values.

[0014] Further, in step S1, a plurality of sets of breast ultrasound images, breast X-ray images and clinical text data are collected and divided into a training set and a test set for operation.

[0015] Further, step S2 specifically includes the following steps:

[0016] S21, sequentially performing medical terminology normalization, dirty data cleaning, natural language processing and missing value filling on the clinical text data to convert it into structured semantic data;

[0017] S22, parsing the breast ultrasound images and the breast X-ray images into pixel matrices through a Pydicom library and performing gray scale normalization to eliminate device gray scale deviation, and the normalization calculation formula is:

[0018] ;

[0019] wherein, is the original pixel value, , are the minimum and maximum values of the image gray scale respectively.

[0020] Further, step S3 specifically includes the following steps:

[0021] S31, embedding a hybrid attention module after each convolution block in the contraction path of the U-Net model, the channel attention module compresses the feature map of the breast ultrasound image into a channel description vector through global average pooling ; the calculation formula is:

[0022] ;

[0023] wherein H represents the height of the feature map, W represents the width of the feature map, represents the feature value of the feature map of the cth channel at the coordinate ;

[0024] S32, generating channel weights through 2 fully connected layers and a Sigmoid activation function , the spatial attention module generates a spatial weight map through channel dimension compression and Sigmoid activation ; the hybrid attention module fuses the original feature map through element multiplication to improve the lesion feature representation ability

[0025] S33, using a compound loss function composed of cross-entropy loss function and Dice coefficient weighted combination; the calculation formula is:

[0026] ;

[0027] wherein, is a weight coefficient, is a cross-entropy loss function, is a real mask value, is a model prediction value, and N is the total number of pixels

[0028] The calculation formula of the Dice coefficient is:

[0029] ;

[0030] wherein, is a real mask value, is a model prediction value, and N is the total number of pixels

[0031] S34, minimizing the compound loss through the Adam optimizer, and outputting the breast ultrasound segmentation mask after iterative training until the Dice coefficient of the validation set is stable.

[0032] Further, in step S4, a ResNet50 network model is used to extract features from the breast ultrasound image, and the calculation formula is:

[0033] ;

[0034] wherein x is the input feature tensor of the residual block is the residual fusion feature tensor, and F(x) is the feature tensor after convolution.

[0035] Further, in the dense connection layer of the DenseNet-121 network model in step S6, a dilated convolution with a variable expansion rate is introduced, and the calculation formula of the dilated convolution is as follows:

[0036] ;

[0037] wherein, is the convolution kernel weight, is the input feature map, is the expansion rate of the i-th layer; n represents the number of convolution kernel elements participating in the dilated convolution calculation;

[0038] and a channel attention mechanism is embedded in the DenseNet-121 network model, and the importance weight of each channel feature is calculated through global average pooling and a fully connected layer to realize the enhancement of the key feature channel, and the calculation formula is as follows:

[0039] ;

[0040] wherein, is the feature map output by the DenseNet, is the channel attention weight vector.

[0041] Further, in step S7, the BERT model contains 12 layers of Transformer encoders, and the hidden state update formula of the first layer of the Transformer encoder is as follows:

[0042] ;

[0043] wherein, is the hidden state of the l-1 layer, MultiHead(·) is a multi-head attention module, and LayerNorm(·) is a layer normalization operation.

[0044] Further, in step S7, a cross-modal attention distillation module is designed to fuse the image fusion feature and the clinical text feature, which specifically includes the following steps:

[0045] S71, the image fusion feature and the clinical text feature are unified in dimension, the clinical text feature is mapped to 512 dimensions through linear projection, and ;

[0046] ;

[0047] wherein, (512x384), (512x1) are projection parameters,

[0048] S72, the image fusion features are normalized by LayerNorm to eliminate the differences in modal feature distribution, and the calculation formula is as follows:

[0049] ;

[0050] wherein, , are the mean and variance of the image fusion features , respectively, is a constant to prevent the denominator from being zero;

[0051] S73, according to the multi-head attention mechanism of the Transformer, taking as the semantic reference, Key, as the feature to be optimized Query, the correlation weight between the dimension of each image fusion feature and the clinical text feature is calculated, and the calculation formula is as follows:

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] wherein, , , are attention parameters, is the feature dimension, is the cross-modal attention weight matrix;

[0057] S74, the cross-modal attention weight matrix is applied to the image fusion features to generate the multi-modal features after deep fusion, and the calculation formula is as follows:

[0058] ;

[0059] wherein, λ=0.3 is the distillation intensity coefficient.

[0060] Compared with the prior art, the application has the advantages and positive effects that:

[0061] ​The breast ultrasound segmentation map is introduced as a new modality, the complementary values of different image modalities can be fully mined and utilized, the recognition effect of related features is further improved, the precision of intelligent auxiliary identification of breast cancer is effectively improved, meanwhile, the semantic association fusion of clinical text and image features can reduce the diagnosis deviation caused by insufficient single modality information, provide more comprehensive and reliable diagnosis reference for doctors, better assist doctors in carrying out breast cancer diagnosis work, on the other hand, the present application has strong generalization ability, can be applied to intelligent auxiliary identification of various cancers, and further improves the use effect of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows, obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0063] Figure 1 The framework flowchart of the present application is shown in the figure;

[0064] Figure 2 The logic architecture diagram of the present application is shown in the figure;

[0065] Figure 3 The architecture schematic diagram of the U-Net model is shown in the figure;

[0066] Figure 4 The architecture schematic diagram of the mixed attention module is shown in the figure. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application, obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, any modification, equivalent replacement, improvement, etc. obtained by those skilled in the art without creative labor should be included in the protection scope of the present application.

[0068] The present application discloses a kind of multi-modal information collaborative breast cancer auxiliary identification method, the method forms "multi-modal data preprocessing, single modality feature extraction and ultrasound modality internal fusion, ultrasound-X ray multi-feature fusion, image-text cross-modal feature fusion, auxiliary identification classification" complete technical process, its overall process is as shown in Figure 1 、 Figure 2 Specific steps are as follows:

[0069] (1) Collect breast cancer ultrasound images, breast X-ray images, and clinical text data from hospital historical data, and divide the data into training set and test set;

[0070] (2) Preprocess the clinical text data, breast ultrasound images, and breast X-ray images respectively, and simultaneously generate breast ultrasound segmentation mask graphs through the segmentation network. Introduce breast ultrasound segmentation graphs as a new modality to construct a "text + original ultrasound + ultrasound segmentation graph + X-ray" four-modality input system;

[0071] (3) Single-modality feature extraction stage: breast ultrasound images are preliminarily extracted through ResNet50 network to obtain bottom visual features, and corresponding segmentation graphs are obtained through U-Net model with fusion attention mechanism. The ultrasound segmentation mask graph is extracted through ResNet50 to obtain lesion shape features. After preliminary fusion, the enhanced ultrasound joint features are obtained by inputting the original ultrasound features, segmentation graph features, and VisionTransformer (ViT) together to mine intra-modality correlation information. At the same time, X-ray image feature extraction (using DenseNet-121) and text feature extraction (using BERT) are completed;

[0072] (4) Cross-modality fusion stage: first, input the enhanced ultrasound joint features and X-ray image features into the cross-modality fusion module to establish the correlation between the anatomical structures of the images to generate image fusion features; then design a cross-modality attention distillation module, and finally pass the image fusion features and clinical text features through the cross-modality attention distillation module for secondary fusion to realize the deep cooperation of semantic and visual information;

[0073] (5) Finally, input the deeply fused multi-modality features into a multi-layer perceptron (MLP) to realize normal, attention, Luminal A type, Luminal B type, HER2 positive type, and triple negative breast cancer subtype identification, and output corresponding probability values to assist doctors in judgment.

[0074] The main operations in the above steps are described in detail as follows:

[0075] (1) Data preprocessing and ultrasound segmentation graph generation;

[0076] Multi-modality data preprocessing: clinical text is normalized by medical terminology, dirty data cleaning, natural language processing (word segmentation, entity recognition), and missing value filling, and is converted into structured semantic data; breast ultrasound and X-ray images in DICOM format are parsed into pixel matrices through the Pydicom library, and gray scale normalization is performed to eliminate device gray scale deviation. The min-max normalization formula is used:

[0077] ;

[0078] wherein, is the original pixel value, , are the minimum and maximum values of the image gray scale, respectively, and the normalized gray scale range is mapped to [0, 1]. The size is standardized to 512x512 pixels, and the bilinear interpolation algorithm is used to retain the image details; for the problem of low contrast of the lesion area, adaptive histogram equalization (CLAHE) is applied, and the image is divided into 8x8 sub-blocks, and the cumulative distribution function (CDF) is calculated for each sub-block:

[0079] ;

[0080] wherein, H and W are the height and width of the sub-block, is the number of pixels of the gray scale , and the CDF is used to uniformly map the gray scale of the sub-block to [0, 255] to enhance the gray scale difference between the lesion and the surrounding tissue.

[0081] Ultrasound segmentation map generation: to obtain accurate spatial information of the lesion, a U-Net segmentation network with a hybrid attention mechanism is designed to generate an ultrasound segmentation mask as a new modality, and the specific scheme is as follows:

[0082] Network structure design: after each convolution block in the U-Net shrinkage path, a hybrid attention module (channel attention + spatial attention) is embedded, as shown in Figure 3 , Figure 4 , the channel attention compresses the feature map into a channel description vector :

[0083] ;

[0084] wherein, H represents the height of the feature map, W represents the width of the feature map, represents the feature value of the feature map of the cth channel at coordinates .

[0085] After passing through 2 fully connected layers and a Sigmoid activation function, the channel weight is generated to strengthen the key channels such as "lesion edge" and "internal echo"; the spatial attention generates a spatial weight map through channel dimension compression (1x1 convolution) and Sigmoid activation to focus on the spatial position of the lesion. The hybrid attention output is fused with the original feature map through element multiplication to improve the lesion feature representation ability.

[0086] Loss function optimization: To address the imbalance between positive and negative samples in medical image segmentation, a composite loss function combining cross-entropy loss and Dice coefficient weighting is adopted.

[0087] ;

[0088] in, Here are the weighting coefficients, and the cross-entropy loss function is... , The actual mask value, Here, N is the model's predicted value, and N is the total number of pixels. The Dice coefficient is calculated as follows:

[0089] ;

[0090] in the formula Parameters and above The parameters in the formula have the same meaning;

[0091] The composite loss is minimized by using the Adam optimizer (with an initial learning rate of 1e-4, decreasing by 10% every 50 epochs). The training is iteratively completed until the Dice coefficients on the validation set are stable (fluctuation <0.01). Finally, a binarized ultrasound segmentation mask is output (lesion area is 1, background is 0), providing accurate spatial features for subsequent intramodal fusion.

[0092] (2) Single-modal feature extraction and fusion;

[0093] Ultrasonic Feature Extraction and Intramodal Fusion: The preprocessed raw ultrasound images are input into the ResNet50 network to solve the gradient vanishing problem in deep network training and ensure the effective transfer of low-level features such as tissue texture and gray-level distribution patterns.

[0094] ;

[0095] Where x is the input feature tensor of the residual block, with dimension 1. (C represents the number of input channels, and H and W represent the feature map height and width, respectively). This is the feature output after processing by "convolutional layer + ReLU activation function"; This is the feature tensor after residual fusion. The final output of the ResNet50 network is the original ultrasound feature with a dimension of 2048×16×16, which contains global structural and gray-level distribution information of breast tissue. The feature extraction process of the ultrasound segmentation mask image is the same as that of the original ultrasound image, extracting morphological features such as lesion edges and aspect ratios separately. After the two types of features are initially fused, they are input into ViT along with the original ultrasound features and segmentation image features. Through a self-attention mechanism, the correlation between local lesion details and global tissue is captured, and the enhanced ultrasound joint feature is output.

[0096] X-ray Image Feature Extraction: Based on the DenseNet-121 network, depth feature extraction from X-ray images is achieved through structural optimization and multi-stage feature enhancement strategies. First, to address the complexity of X-ray images, the DenseNet-121 network structure is specifically improved. In the densely connected layers of the traditional DenseNet, dilated convolutions with variable dilation rates are introduced to replace some standard convolution operations. Dilated convolutions expand the receptive field by inserting "holes" between convolution kernel elements; its calculation formula is as follows:

[0097] ;

[0098] in, For convolution kernel weights, For the input feature map, Let be the dilation rate of the i-th layer, and n represent the number of kernel elements involved in the dilated convolution calculation. A smaller dilation rate (e.g., ...) is set for shallower networks. To capture local details such as the edges of calcified clusters, and to increase the expansion rate in deeper networks (e.g.) This allows for coverage of a larger area, capturing global features such as the distribution of microcalcification clusters and structural distortions. Simultaneously, to suppress the interference of high-density glandular background on lesion features, a channel attention mechanism is embedded in the network. This mechanism calculates the importance weights of each channel feature through global average pooling and fully connected layers. This enhances key feature channels such as calcification clusters, as shown in the following formula:

[0099] ;

[0100] in, The feature map output by DenseNet. This is the channel attention weight vector.

[0101] Text feature extraction: A pre-trained BERT model was used to extract semantic features from clinical texts, and the association between key information such as body mass index (BMI) and past medical history and breast cancer was explored. The specific process is as follows:

[0102] The preprocessed text is input into a BERT model containing a 12-layer Transformer encoder, and the contextual semantic relationships of the text are captured through a bidirectional attention mechanism. The hidden state update formula for the layer Transformer encoder is:

[0103] ;

[0104] in, The hidden state represents the (l-1)th layer. MultiHead(⋅) is a multi-head attention module (set to 12 attention heads), and LayerNorm(⋅) is a layer normalization operation to ensure training stability. The hidden state corresponding to the feature (768 dimensions) is mapped to a 384-dimensional text feature through a fully connected layer. This feature contains semantic representations of key information such as BMI, history of breast disease, and family history of cancer, and can be directly used for subsequent cross-modal fusion.

[0105] (3) Design a cross-modal attention distillation module;

[0106] To address the issues of cross-modal feature alignment and insufficient feature fusion in image-text overall feature fusion, a cross-modal attention distillation module is proposed and added between the Vision Transformer and the Language Transformer.

[0107] The cross-modal attention distillation module mainly consists of three parts: a modal feature alignment submodule, an attention weight calculation submodule, and a feature distillation update submodule.

[0108] Modal feature alignment submodule: for image fusion features containing ultrasound combined features and X-ray features, with a dimension of 512×1. Text features To achieve dimensional uniformity, linear projection is used to... Mapping to 512 dimensions yields At the same time, LayerNorm normalization is performed to eliminate differences in modal feature distributions, using the following formula:

[0109] ;

[0110] ;

[0111] in, (512×384) (512×1) represents the projection parameters. , They are respectively The mean and variance, To prevent constants with zero denominators, this submodule addresses the issues of "cross-modal feature alignment and inconsistent distribution," laying the foundation for subsequent distillation.

[0112] Attention weight calculation submodule: Based on the Transformer multi-head attention mechanism, a "semantic-guided attention" is designed to... For semantic benchmark Key, For the feature Query to be optimized, calculate the association weight between each image feature dimension and the text semantics, using the following formula:

[0113] ;

[0114] ;

[0115] ;

[0116] ;

[0117] in, , , (All are 512×512) are attention parameters. For feature dimension, (512×512) is the cross-modal attention weight matrix. This weight matrix can quantify the matching degree between each dimension of image features and text semantics. For example, the key corresponding to "BRCA gene mutation" in the text will strengthen the weight of the "calcification cluster distribution" dimension in the image features. This design is different from the "no benchmark weight allocation" of traditional attention and realizes semantic orientation guidance.

[0118] Feature distillation update submodule: This module updates attention weights. By applying "teacher modality (text) knowledge distillation" to image features, semantically enhanced image features are generated. The formula is as follows:

[0119] ;

[0120] Here, λ=0.3 is the distillation intensity coefficient, used to balance the contributions of original image features and semantically guided features. This submodule preserves the original discriminative power of image features such as lesion edges and calcification morphology through residual connections, while incorporating textual semantic information such as medical records and past medical history. This solves the limitation of image-text fusion, which only superimposes features without achieving semantic synergy, and enables the fused features to have both visual recognizability and semantic relevance.

[0121] (4) The training process and parameters for machine learning using training and test sets;

[0122] The overall training process of the model utilizes the Adam optimizer to train the image model. The loss function is a composite loss function combining the cross-entropy loss function and the Dice coefficient weighted as described above. A stepwise adjustment strategy is employed for the learning rate to balance the model's training efficiency and stability. Specifically, the learning rate is adjusted as follows: the initial learning rate is set to 0.0001 to help the model quickly enter the effective training phase; if the model loss no longer decreases within 80 epochs, the learning rate is reduced to one-fifth of its original value to avoid instability during model training; the maximum number of epochs is set to 800.

[0123] This invention introduces a novel modality—breast ultrasound segmentation—which fully leverages the complementary value of different imaging modalities to further enhance the recognition of relevant features and effectively improve the accuracy of intelligent assisted breast cancer identification. Simultaneously, by combining clinical text with semantic association fusion of image features, it reduces diagnostic bias caused by insufficient information from a single modality, providing doctors with more comprehensive and reliable diagnostic references and better assisting them in breast cancer diagnosis. Furthermore, this invention possesses strong generalization ability, applicable to intelligent assisted identification of various cancers, further enhancing its effectiveness.

Claims

1. A method for breast cancer assisted identification based on multimodal information collaboration, characterized in that: Includes the following steps: S1. Collect the patient's breast ultrasound images, breast X-ray images, and clinical text data; S2. Perform preprocessing operations on clinical text data, breast ultrasound images, and breast X-ray images; S3. Use the U-Net model with fusion attention mechanism to segment the preprocessed breast ultrasound image to obtain the breast ultrasound segmentation mask. S4. Use the ResNet50 network model to perform feature extraction on breast ultrasound images to obtain the original ultrasound features and low-level visual features; use the ResNet50 network model to perform feature extraction on the breast ultrasound segmentation mask to obtain segmentation map features and lesion morphology features. S5. The underlying visual features and lesion morphology features are initially fused, and the fusion results are input into the Vision Transformer model along with the original ultrasound features and segmentation map features to mine intramodal correlation information and obtain enhanced ultrasound joint features. S6. Use the DenseNet-121 network model to extract features from the preprocessed mammograms to obtain X-ray image features; input the enhanced ultrasound combined features and X-ray image features into the cross-modal fusion module to establish the anatomical structure association between images and obtain image fusion features. S7. Use the BERT model to extract features from the preprocessed clinical text data to obtain clinical text features; The cross-modal attention distillation module is designed to fuse image fusion features with clinical text features, achieving deep synergy between semantic and visual information, and obtaining deeply fused multimodal features; S8. Input the deeply fused multimodal features into the MLP network model, and use the MLP network model to identify and output the corresponding probability values ​​of normal, attention, Luminal A, Luminal B, HER2 positive and triple negative breast cancer subtypes.

2. The breast cancer-assisted identification method based on multimodal information collaboration as described in claim 1, characterized in that: In step S1, several sets of breast ultrasound images, breast X-ray images, and clinical text data are collected and divided into training sets and test sets for operation.

3. The breast cancer-assisted identification method based on multimodal information collaboration as described in claim 2, characterized in that: Step S2 specifically includes the following steps: S21. The clinical text data is transformed into structured semantic data by performing medical terminology normalization, dirty data cleaning, natural language processing, and missing value imputation in sequence. S22. Analyze breast ultrasound and mammogram images into pixel matrices using the Pydicom library, and perform grayscale normalization to eliminate device grayscale deviations. The normalization calculation formula is as follows: ； in, These are the original pixel values. , These represent the minimum and maximum grayscale values ​​of the image, respectively.

4. The breast cancer-assisted identification method based on multimodal information collaboration as described in claim 3, characterized in that: Step S3 specifically includes the following steps: S31. Embed a hybrid attention module after each convolutional block in the U-Net model's contraction path. The channel attention module compresses the feature map of the breast ultrasound image into a channel description vector through global average pooling. The calculation formula is as follows: ； Where H represents the height of the feature map, and W represents the width of the feature map. The feature map of the c-th channel is represented in coordinates Eigenvalues ​​at; S32. Channel weights are generated through two fully connected layers and a Sigmoid activation function. The spatial attention module generates a spatial weight map through channel dimension compression and Sigmoid activation. The hybrid attention module enhances the ability to represent lesion features by fusing element-wise multiplication with the original feature map. S33. A composite loss function using a weighted combination of the cross-entropy loss function and the Dice coefficient; its calculation formula is as follows: ； in, These are the weighting coefficients. Let cross-entropy be the loss function. The actual mask value, The value is the model prediction, where N is the total number of pixels; The formula for calculating the Dice coefficient is: ； in, The actual mask value, The value is the model prediction, where N is the total number of pixels; S34. Minimize the composite loss using the Adam optimizer, iterate until the Dice coefficients on the validation set are stable, and then output the breast ultrasound segmentation mask.

5. The breast cancer-assisted identification method based on multimodal information collaboration as described in claim 4, characterized in that: In step S4, the ResNet50 network model is used to extract features from breast ultrasound images. The calculation formula is as follows: ； Where x is the input feature tensor of the residual block; Let F(x) be the feature tensor after residual fusion, and F(x) be the feature tensor after convolution.

6. The breast cancer-assisted identification method based on multimodal information collaboration as described in claim 5, characterized in that: In step S6, dilated convolutions with variable dilation rates are introduced into the densely connected layers of the DenseNet-121 network model; the formula for calculating dilated convolutions is: ； in, For convolution kernel weights, For the input feature map, is the dilation rate of the i-th layer; n represents the number of kernel elements involved in the dilated convolution calculation; Furthermore, a channel attention mechanism is embedded in the DenseNet-121 network model, which calculates the importance weights of each channel feature through global average pooling and fully connected layers. This enhances key feature channels, and the calculation formula is as follows: ； in, The feature map output by DenseNet. This is the channel attention weight vector.

7. The breast cancer-assisted identification method based on multimodal information collaboration as described in claim 6, characterized in that: In step S7, the BERT model contains 12 Transformer encoder layers. The hidden state update formula for the layer Transformer encoder is: ； in, Let be the hidden state of the (l-1)th layer, MultiHead(⋅) be the multi-head attention module, and LayerNorm(⋅) be the layer normalization operation.

8. The breast cancer-assisted identification method based on multimodal information collaboration as described in claim 7, characterized in that: In step S7, the design of the cross-modal attention distillation module to fuse image fusion features with clinical text features specifically includes the following steps: S71, Image Fusion Features Features of clinical texts Dimensional unification was performed, and clinical text features were represented by linear projection. Mapping to 512 dimensions yields ; ； in, (512×384) (512×1) represents the projection parameters. S72, Image Fusion Features LayerNorm normalization is performed to eliminate differences in modal feature distributions. The calculation formula is as follows: ； in, , Image fusion features The mean and variance, To prevent constants with a denominator of zero; S73. Based on the Transformer multi-head attention mechanism, As a semantic benchmark, Key, For the feature Query to be optimized, the association weight between the dimension of each image fusion feature and the clinical text feature is calculated using the following formula: ； ； ； ； in, , , For attention parameters, For feature dimension, This is the cross-modal attention weight matrix; S74, cross-modal attention weight matrix Applying to image fusion features to generate multimodal features after deep fusion. The calculation formula is as follows: ； Where λ=0.3 is the distillation intensity coefficient.

Images