A cross-modal fusion classification method based on digital breast images and ultrasound images
Patent Information
- Application Number
- CN202411284572.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-13
AI Technical Summary
Single digital breast images or ultrasound images are not effective in the molecular classification and classification task of breast cancer, and cannot effectively guide doctors to formulate treatment plans, especially in the judgment of Basal-like breast cancer types with poor prognosis.
Using the two-way generation network DCLGAN based on contrast learning improvement, add channel attention mechanism to generate pseudo-ultrasound images and pseudo-DBT images, and realize the feature fusion of DBT and pseudo-ultrasound images, ultrasound and pseudo-DBT images through multimodal Transformer model to improve the Classification ability.
It improves the accuracy of Basal-like classification of breast cancer, helps doctors formulate more accurate treatment plans, and improves the treatment effect.
Smart Images

Figure CN118799658B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a cross-modal fusion classification method based on digital breast images and ultrasonic images. Background Art
[0002] With the continuous advancement of science and technology and the continuous upgrading of medical equipment, the application of medical imaging technology in medical imaging diagnosis is becoming more and more extensive. Medical imaging technology can obtain image information of the internal structure of the human body through various imaging technologies, and diagnose and treat diseases based on this information. Different imaging technologies can observe the morphology, size, location, density and tissue structure of lesions, thereby helping doctors to develop treatment plans for individual patients and improve the treatment effect of diseases.
[0003] Digital Breast Tomosynthesis (DBT) uses a principle similar to computed tomography (CT) to create high-resolution three-dimensional tissue structure images through multiple X-ray projection images. Compared with traditional 2D mammography, DBT can reduce the interference of overlapping tissues and more clearly display the microstructures and lesions in the breast, especially for dense breast tissue. Ultrasound (US) Ultrasound is a high-frequency sound wave. Ultrasound pulses are sent to human tissues through an ultrasound probe (also called an ultrasound transducer). When the pulse passes through tissues of different densities and boundaries, reflections will occur. These reflections are received by the probe and converted into electrical signals. After processing, the electrical signals form an image, which can provide real-time, radiation-free soft tissue assessment. Molecular classification of breast cancer is a method of classification based on the molecular characteristics of tumor cells. Basal-like breast cancer is a representative of breast cancer types with poor prognosis in molecular classification, with a 5-year survival rate of less than 15%. Molecular classification can effectively guide personalized treatment options and predict patient prognosis. However, the image information contained in a single type of image is limited. At present, single DBT images or ultrasound images cannot achieve good results in molecular typing and classification tasks, and cannot provide effective reference for doctors to judge molecular typing categories. Summary of the invention
[0004] The purpose of the present invention is to solve the above problems and propose a cross-modal fusion classification method based on digital breast images and ultrasound images. At present, the cross-modal adversarial generation of image information using GAN generative adversarial networks can further enrich the original image information and obtain better results in specific classification tasks. Therefore, this method adds a channel attention mechanism based on the DCLGAN network, a bidirectional generative network improved based on contrastive learning, and uses the generative network to realize the cross-modal generation of pseudo-ultrasound images containing DBT image information from DBT images. At the same time, pseudo-DBT images with ultrasound information are generated from ultrasound images across modalities, and then the multimodal Transformer model is used to realize the feature fusion of DBT images and pseudo-ultrasound images, and pseudo-ultrasound images and DBT images, so as to further improve the classification ability for basal-like types. It helps doctors to implement more accurate treatment plans for patients to improve the treatment effect. The present invention has important practical value and significance.
[0005] To achieve the above object, the technical solution of the present invention is:
[0006] A cross-modal fusion classification method based on digital breast images and ultrasound images includes the following steps:
[0007] Step 1: Obtain ultrasound images and DBT image data of breast cancer patients, segment the marked lesion parts, and ensure the integrity of the segmented lesion information by performing dot multiplication between the square mask and the ultrasound image and DBT image data.
[0008] Step 2: Divide the segmented image data into a training set and a test set. Then, perform data enhancement and sample balancing on the sample data of the training set, and enhance the training set through rotation and mirroring operations.
[0009] Step 3: Improve the DCLGAN structure based on contrastive learning to construct the EDCLGAN network model, add a channel attention mechanism module to the original residual structure, input the DBT images and ultrasound images in the training data set into the EDCLGAN model, generate pseudo ultrasound images and pseudo DBT images, which helps to adaptively adjust the channel weights to enhance the network's feature extraction capabilities. Input the image pairs of DBT images and pseudo ultrasound images in the training data set into the EDCLGAN model, and input the image pairs of ultrasound images and pseudo DBT images into the EDCLGAN model.
[0010] Step 4: Construct image pairs of DBT images and pseudo-ultrasound images, and image pairs of ultrasound images and pseudo-DBT images, realize cross-modal feature fusion by constructing a multimodal Transformer model based on cross-attention, and then classify them through the classifier, and complete the training with the training set.
[0011] Step 5: Performance testing is performed on the test set, and the classification prediction effect is compared with the generated images generated by the generative adversarial network, the image pairs based on DBT images and pseudo-ultrasound images, and the image pairs of ultrasound images and pseudo-DBT images fused by the multimodal Transformer model to further verify the effectiveness and practicality of the present invention.
[0012] Compared with the prior art, the present invention has the following beneficial effects:
[0013] The present invention relates to a cross-modal ultrasound image generation method based on contrastive learning and channel attention mechanism to realize the technology of cross-modal generation based on DBT and ultrasound images, and uses the channel attention mechanism module to improve the network feature extraction performance, so that the generated image quality is better. The multimodal Transformer model based on the cross-attention mechanism realizes the cross-fusion of the original image and the generated image features to obtain better classification performance, thereby helping doctors and even artificial intelligence to make more accurate judgments. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0015] Figure 1 The specific segmentation process of DBT images of breast cancer provided by the embodiment of the present invention;
[0016] Figure 2 A DCLGAN generative adversarial network model structure diagram provided by an embodiment of the present invention;
[0017] Figure 3 A design diagram of the generative adversarial network attention residual block generator model provided in an embodiment of the present invention.
[0018] Figure 4 A structural diagram of a multimodal Transformer model provided in an embodiment of the present invention;
[0019] Figure 5 AUC curve diagram of classification effects of DBT images, ultrasound images, and pseudo ultrasound images on the basal-like task provided by an embodiment of the present invention;
[0020] Figure 6 An AUC curve diagram of the classification effect of the multimodal Transformer model on the basal-like task provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0022] A cross-modal fusion classification method based on digital breast images and ultrasound images includes the following steps:
[0023] Step 1: Obtain digital breast imaging and ultrasound imaging data
[0024] The medical images used in this embodiment are from the Affiliated Tumor Hospital of Fudan University. The DBT images are taken using the Hologic digital breast tomosynthesis fusion machine of Hologic, USA, and the ultrasound images are collected using the Resona 7t device of Mindray, China. The collected breast cancer image data are first screened to remove images with poor imaging clarity and unclear lesion areas, and then the original image data are segmented to extract the lesion area.
[0025] The specific operations are as follows:
[0026] Since DBT images contain multiple slices, the shapes and sizes of lesions in different slices are different. In addition, the lesion area may not be visible in the pre- and post-fraction slices. Therefore, in the DBT image files annotated by the doctor, the lesion area also appears in the form of slice slices. The json file information obtained by the doctor's annotation can accurately locate the image corresponding to the lesion area, thereby realizing the automatic segmentation of the lesion area. The specific image segmentation process is as follows: Figure 1 As shown. The lesion information is complete by performing dot multiplication between the square mask and the image. The processing flow of the square mask matrix is as follows:
[0027] (1) Generate an all-zero matrix with exactly the same shape and size as the original image.
[0028] (2) According to the annotation information, all zero matrices in the lesion area are set to .
[0029] (3) Use the rectangle to find the boundary and find the longest side along the x-axis or y-axis as the side length of the square.
[0030] (4) Adjust the entire rectangle along the short side to ensure that the center of the square mask remains the center of the original rectangular mask.
[0031] (5) Set all zero matrices in the rectangular area to 1.
[0032] (6) Finally, the generated mask matrix is saved as an npy format file to obtain the mask matrix.
[0033] Doctors manually segment the ultrasound images to obtain an image dataset containing the lesion area.
[0034] Step 2: Preprocess the segmented image data
[0035] Since the amount of DBT and ultrasound image data used is relatively small, and there is sample imbalance in the data, we need to perform data enhancement and data balancing on the segmented breast image data. First, the data set is randomly divided into a training set and a test set in a ratio of 7:3. When expanding the data set, only the training set is expanded, and the test set data remains unchanged. Secondly, for the sample imbalance problem in the training set, we choose to increase the weight of negative class samples during training to achieve sample balance for positive and negative classes. For the problem of small quantity, we choose to use the data enhancement method, that is, to rotate and mirror the image data after sample balancing to achieve the purpose of expanding the sample. Finally, a total of 214 patient data were used. There were 150 cases in the training set and 64 cases in the test set.
[0036] Step 3: Build a deep adversarial generative network model
[0037] like Figure 2 As shown, the DCLGAN model used in this method is an input non-matching recurrent adversarial neural network model, which mainly includes two generators and, and two discriminators. The generator mainly includes an input feature encoder module composed of an encoder, a residual network deep feature extraction block composed of residual blocks, an output generated image feature decoding block composed of a decoder, and a prediction classification module composed of a classifier. The discriminator is a convolutional neural network using the Patch-GAN mode. The advantage of Patch-GAN is that it reduces the receptive area of the neurons of the output layer nodes for the entire image, which improves the discrimination ability of the entire image compared to the traditional GAN structure, and helps to improve the stability of the training of the generative adversarial network model, making the output results more stable and reducing the possibility of mode collapse. For each mapping, we extract the features of the image from the four layers of the encoder to a two-layer MLP projection head ( and ), the projection head maps the high-dimensional features extracted by the encoder into a low-dimensional space to facilitate effective contrastive learning.
[0038] The focus of the entire network model is on the generator model module. In order to take into account the accuracy of the model for cross-modal information conversion and the accuracy of the prediction effect of the final basal-like classification, it means that the classification generator model needs to have both good classification effect and good image generation effect. In the study, the method of reducing the complexity of the model while still retaining the classification effect of the model is to introduce the channel attention mechanism in the deep convolutional neural network. The channel attention mechanism is a concept proposed in machine vision research based on the way the human eye pays attention to things. When observing things, the human eye always focuses on the area of interest and ignores other areas of secondary information. In the field of artificial intelligence, it enables artificial neural networks to quickly pay attention to the information in the effective area, thereby improving the problem-solving effect of artificial neural networks.
[0039] Therefore, the proposed entire classification generator module also introduces a channel attention mechanism in the convolutional downsampling neural network in addition to the aforementioned modules to take into account both the generation effect and the image generation effect. The entire feature downsampling module of the classification generator adopts the classic ResNet residual network structure. Different from the original ResNet structure, the present invention adds a channel attention mechanism module to the custom residual structure. Its structure is as follows Figure 3 shown.
[0040] The adversarial neural network model is specifically as follows:
[0041] Recurrent Adversarial Neural Network Adversarial Loss:
[0042]
[0043]
[0044] Where X represents the real DBT image.
[0045] Y represents the real ultrasound image; represents the generation of adversarial loss, which is the name of the loss; E represents the mathematical expectation; D represents the discriminator function, Indicates that the discriminant is generated positively by X That is, to generate the forward discriminant function of the ultrasound image. Indicates that it is generated in reverse from Y That is, the reverse discriminant function of the pre-chemotherapy image is generated; G represents the forward generator function; F represents the reverse generator function.
[0046] Patch-NCE Loss:
[0047]
[0048]
[0049] in represents the corresponding features of each selected layer, represents all other features except the selected layer, Represents the corresponding features of the selected layer of the corresponding generation domain represents the cross entropy loss. The contrastive learning problem consists of three signals: the query sample and positive samples , negative samples ,let and Signal correlation and In contrast, and And N Mapped into K-dimensional vectors , , , and use Represents the nth negative sample, and the cross entropy calculation formula is as follows:
[0050]
[0051] Through two layers of network The resulting set of features is , Indicates Layer output features. Sequence , ,in express Layer The corresponding feature is recorded as , and other features are marked as ,in is the number of channels per layer, which will output the image Encoded as .
[0052] The cyclic adversarial neural network ensures that the generated results are consistent with the original image loss:
[0053]
[0054] in Represents the generation consistency loss.
[0055] Finally, the loss function of this DCLGAN network is:
[0056]
[0057] We set =1, = 2, = 1 for better performance.
[0058] The training parameters of the improved generative adversarial network model are shown in Table 1.
[0059] Table 1. Training parameters of the GAN model
[0060]
[0061] Step 4: Construction of multimodal Transformer model based on cross-attention mechanism
[0062] With the rapid development of deep learning, multimodal feature fusion technology has been widely used in the fields of computer vision and natural language processing. Multimodal feature fusion refers to combining data from different modalities to obtain richer and more accurate features, thereby achieving better prediction and analysis results. In medical image analysis, fusing medical image information from different modalities can improve the accuracy and reliability of medical image analysis and diagnosis.
[0063] The present invention designs a cross-attention multimodality transformer model (CAMT), which can use the original image and the generated image to predict the molecular typing of the patient. It is mainly composed of two branches. The structure of the model is as follows Figure 4 shown.
[0064] (1) The DBT image and pseudo ultrasound image are respectively subjected to feature extraction through two branches with ResNet convolutional layers as encoders to obtain feature maps F1 and F2.
[0065] (2) The feature maps F1 and F2 are expanded into sequences consisting of feature vectors, and the position code and classification label CLS token are added to the sequences. They are input into two Transformer modules to obtain feature maps F1' and F2' respectively.
[0066] The position coding is specifically implemented as follows: position marking, i.e., position coding, is performed in sequence according to the order of the feature vectors in the sequence.
[0067] (3) A cross-attention module is added after the Transformer module of every N DBT images and M pseudo-ultrasound images to fuse the features of the two modalities of feature maps F1' and F2' to obtain the fused features.
[0068] (4) After passing through the Transformer model and the cross-attention module, the fused features are output to the classifier to obtain the classification results.
[0069] (5) Perform the same operations as steps (1) to (4) above on the ultrasound image and pseudo-DBT image, and output the classification results.
[0070] The feature fusion process of the cross attention is as follows Figure 5 , the patch embeddings of the original image branch and the generated image branch are respectively , Indicates that the patch embedding is the local feature of the two branches. The classification labels of the two branches are global features of different modalities, using and Indicates. It mainly includes the following steps:
[0071] (1) Feature channel alignment and cross-feature merging. First, the global features of the original image branch are Projection to obtain the global features of the generated image branch The same number of channels, denoted as . Then the global features after the original image branch projection and generate patch embeddings for the image branch Combined into cross features .
[0072]
[0073] (2) Cross-attention feature fusion. First, the global features of the original image branch are Projection is performed to obtain the query vector , the cross features composed of the global features of the generated image branch and the patch embedding of the generated image are projected to obtain the key vector and value vector ( ). The cross attention operation formula is defined as:
[0074]
[0075]
[0076]
[0077] in is the trainable projection matrix, is the number of channels of the feature vector, is the number of multi-head attention. Multiple Heads Cross Attention (MCA) transforms the input linearly, divides it into multiple heads for attention calculation, and finally concatenates the attention results of multiple heads and transforms them linearly again to get the final output.
[0078] (3) Layer normalization and residual connection. Layer normalization normalizes the features of each sample in the channel dimension so that the features have similar distribution, which can help the model learn and generalize better. Its output feature expression is:
[0079]
[0080] in Represents the layer normalization function.
[0081] (4) Feature back projection. represent The inverse projection function projects the global features fused with the generated image information back to the original dimension. The feature inverse projection and feature splicing operation formulas are defined as
[0082]
[0083] Generating images follows the same process.
[0084] Step 5: Generate images and evaluate the classification results of the multimodal Transformer model
[0085] Basal-like breast cancer is a special subtype of breast cancer, named because its tissue morphology and gene expression profile are similar to the basal cells or myoepithelial cells of normal breast cancer tissue. The 5-year survival rate of Basal-like breast cancer is less than 15%, and it is often clinically regarded as a type of breast cancer with poor prognosis. Basal-like breast cancer is more common in young premenopausal patients, with a high incidence of visceral metastasis and brain metastasis, poor pathological histological grading, mostly grade 3, a high proportion of cell proliferation, and often accompanied by p53 mutations. p53 and EGFR expression are mostly positive, and basal cell markers CK5 / 6 and CKI7 are also mostly positive. Compared with other subtypes of breast cancer, Basal-like breast cancer has the characteristics of poor biological behavior, limited treatment methods, and poor prognosis. Basal-like typing is an important factor that affects doctors in formulating patient treatment plans, so we use the Basal-like typing and non-Basal-like typing of breast cancer as prediction tasks.
[0086] Firstly, DBT and ultrasound images are classified into Basal-like and non-Basal-like types based on the Resnet network. Then, the EDCLGAN network with attention mechanism is used to realize the generation of ultrasound images from DBT images and the evaluation of classification effect. Finally, the generated pseudo ultrasound images are fused with DBT images using a cross fusion network to further optimize the classification effect. The results are shown in Table 1. The classification AUC curves of DBT, ultrasound and pseudo ultrasound images are shown in Table 1. Figure 5As shown, the classification AUC curve of the multimodal Transformer model is as follows Figure 6 As shown in the table data, it can be found that compared with DBT images, ultrasound images have better results in Basal-like classification tasks. The classification performance of pseudo ultrasound images generated by adversarial means is improved compared with that of original DBT images. The multimodal Transformer model can effectively integrate the features of original DBT and pseudo ultrasound images to significantly improve the classification effect.
[0087] Table 2 Classification results of different data
[0088]
[0089] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.
Claims
1. A cross-modal fusion classification method based on digital breast images and ultrasound images. It is characterized in that The following steps are involved: Step 1: Obtain ultrasound images and DBT image data of breast cancer patients and segment the marked lesion parts; Step 2: Divide the segmented image data into a training set and a test set, and perform data enhancement and sample balancing on the sample data of the training set; Step 3: Improve the generative adversarial network DCLGAN based on contrastive learning to obtain the EDCLGAN model, input the DBT images and ultrasound images in the training data set into the EDCLGAN model respectively, and generate pseudo ultrasound images and pseudo DBT images; The improvement of the generative adversarial network DCLGAN based on contrastive learning is specifically as follows: adding a channel attention mechanism module to the original residual structure of the DCLGAN network model; Step 4: Construct image pairs of DBT images and pseudo-ultrasound images, and image pairs of ultrasound images and pseudo-DBT images, realize cross-modal feature fusion by constructing a multimodal Transformer model based on cross-attention, and then classify them through a classifier, and complete the training with the training set; The cross-attention based multimodal Transformer model is specifically implemented as follows: S4.1: The DBT image and the pseudo ultrasound image are respectively subjected to feature extraction by two branches with the convolutional layers of ResNet as encoders to obtain feature maps F1 and F2; S4.2: Expand the feature maps F1 and F2 into sequences consisting of feature vectors, add position encoding and classification label CLS token to the sequences, and input them into two Transformer modules to obtain feature maps F1' and F2' respectively; S4.3: Add a cross-attention module after the Transformer module of every N DBT images and M pseudo-ultrasound images to fuse the features of the two modalities of feature maps F1' and F2' to obtain the fused features; During the feature fusion process of the cross-attention module, the patch embedding of the original image branch and the generated image branch in the cross-attention module is respectively used Indicates that the patch embedding is the local feature of the two branch data, and the classification labels of the two branches are global features of different modalities. and Indicates that the following steps are included: S4.3.1: Global features of the original image branch Through the function f d (.) Projection obtains the global features of the generated image branch The same number of channels, denoted as Then the global features after the original image branch projection and generate patch embeddings for the image branch Combined into cross feature x′ d : S4.3.2: Global features of the original image branch Projection is performed to obtain the query vector q, and the cross features composed of the global features of the generated image branch and the patch embedding of the generated image are projected to obtain the key vector k and the value vector v respectively; the cross attention operation formula is defined as: k=x′ d W k ,v=x′ d W v CA=Off MCA=Concat(CA 1 ,…,THAT h ) Where W q ,W k ,W v ∈R C×(C / h) is the trainable projection matrix, C is the number of channels of the feature vector, and h is the number of multi-head attentions; the multi-head cross attention MCA transforms the input linearly, divides it into multiple heads for attention calculation, and finally concatenates the attention results of multiple heads and transforms them linearly again to obtain the final output; S4.3.3: Layer normalization normalizes the features of each sample in the channel dimension so that the features have similar distribution. The output feature expression is: Where LN(·) represents the layer normalization function; S4.3.4: g d (·) represents f d The inverse projection function of (·) projects the global features fused with the generated image information back to the original dimension. The feature inverse projection and feature splicing operation formulas are defined as: Generating images follows the same process; S4.4: After the Transformer model and the cross-attention module, the fused features are output to the classifier to obtain the classification result; S4.5: Perform the same operations as steps S4.1-S4.4 above on the ultrasound image and the pseudo-DBT image, and output the classification results; Step 5: Perform performance testing through the test set to verify effectiveness and practicality.
2. The cross-modality fusion classification method based on digital breast images and ultrasound images according to claim 1, It is characterized in that The segmentation of the marked lesion part in step 1 is specifically performed by performing dot multiplication between a square mask and the ultrasound image and DBT image data to complete the segmentation.
3. The cross-modality fusion classification method based on digital breast images and ultrasound images according to claim 2, It is characterized in that The data enhancement described in step 2 includes rotation and mirroring operations.
4. The cross-modality fusion classification method based on digital breast images and ultrasound images according to claim 3, It is characterized in that The position coding is specifically implemented as follows: position marking, i.e., position coding, is performed in sequence according to the order of the feature vectors in the sequence.
