A multi-modal breast ultrasound diagnosis system based on a dynamic feature fusion network

Abstract

The application belongs to the field of medical image processing, and particularly relates to a multi-modal breast ultrasound diagnosis system based on a dynamic feature fusion network. The application aims to solve the problem of low recognition accuracy of existing methods for breast ultrasound. The system comprises a multi-modal data preprocessing main module for preprocessing collected multi-modal breast ultrasound images to obtain preprocessed multi-modal breast ultrasound images; a multi-modal convolutional neural network main module for building a dynamic feature fusion network model; a model training main module for training the built dynamic feature fusion network model using the preprocessed multi-modal breast ultrasound images to obtain a trained dynamic feature fusion network model; and a model testing main module for loading the trained dynamic feature fusion network model, classifying to-be-diagnosed multi-modal breast ultrasound images, and identifying the benignity or malignancy of breast tumors in the multi-modal ultrasound images.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical image processing, and particularly relates to a multi-modal breast ultrasound diagnosis system based on a dynamic feature fusion network. BACKGROUND

[0002] Breast cancer is one of the most common malignant tumors in women worldwide and the second leading cause of death from cancer in women. The latest data from the World Health Organization shows that there were 2.26 million new cases of breast cancer worldwide in 2020, resulting in 685,000 deaths. This makes breast cancer more common than lung cancer worldwide. Early diagnosis and timely treatment can significantly improve the survival rate and cure rate of breast cancer patients. Currently, common diagnostic methods for breast cancer include ultrasound examination, mammography, magnetic resonance imaging, and pathological examination. Among them, breast ultrasound has become an important medical imaging technology in breast cancer screening and diagnosis due to its low cost, non-invasive, and no radiation advantages. However, traditional breast ultrasound diagnosis largely depends on the rich experience and subjective evaluation of clinicians, which can easily lead to missed diagnosis or misdiagnosis, and even experienced professionals cannot completely avoid it. In challenging cases, the morphological features of benign and malignant lesions in B-mode ultrasound (Brightness-mode Ultrasound, BUS) images are very small or even contradictory, which to some extent limits the accuracy of diagnosis.

[0003] To solve the problem of limited accuracy of single-mode ultrasound diagnosis, doctors integrate color Doppler flow imaging (CDFI) and ultrasonic elastography (UE) into the standard diagnostic examination of breast cancer based on traditional BUS. This integration significantly improves the diagnostic accuracy of breast cancer. Specifically, BUS provides morphological information about the shape, direction, and edge of breast nodules; CDFI provides detailed information about the blood flow inside and around breast nodules, while UE provides quantitative information about the hardness of breast nodules and their surrounding tissues. Especially in malignant tumors, CDFI and UE images often show rich blood flow signals and obvious hardening. The combination of these ultrasound modes enables doctors to make more accurate decisions. However, integrating these breast ultrasound images from different modes to accurately identify malignant tumors requires additional expertise and clinical experience. Therefore, it is of great practical significance to develop a high-performance computer-aided diagnosis system that can accurately diagnose breast cancer through multi-modal ultrasound images.

[0004] In recent years, with the rapid development of artificial intelligence technology, scholars have conducted in-depth research on computer-aided diagnosis systems, which can achieve accurate diagnosis of breast cancer through multi-modal ultrasound. However, the existing multi-modal diagnosis method still faces many challenges. On the one hand, the traditional diagnosis method based on convolutional neural network usually has difficulty in extracting global information from multi-modal ultrasound images, and can only focus on local information within a relatively small receptive field. On the other hand, the contribution of different modalities to the diagnosis result usually differs significantly, and the existing algorithm treats all modal information equally, and cannot realize dynamic fusion of different modal features. SUMMARY

[0005] The present application is to solve the problem that the existing multi-modal diagnosis method based on deep learning lacks global feature extraction capability and cannot realize dynamic fusion of multi-modal features, resulting in low accuracy of breast ultrasound recognition. A multi-modal breast ultrasound diagnosis system based on a dynamic feature fusion network is proposed.

[0006] A multi-modal breast ultrasound diagnosis system based on a dynamic feature fusion network comprises:

[0007] A multi-modal data preprocessing main module, a multi-modal convolutional neural network main module, a model training main module and a model testing main module;

[0008] The multi-modal data preprocessing main module is used for preprocessing the collected multi-modal breast ultrasound images to obtain preprocessed multi-modal breast ultrasound images;

[0009] The multi-modal breast ultrasound images include B-mode ultrasound, color Doppler flow imaging and ultrasonic elastography;

[0010] The multi-modal convolutional neural network main module is used for building a dynamic feature fusion network model;

[0011] The model training main module uses the preprocessed multi-modal breast ultrasound images to train the built dynamic feature fusion network model to obtain a trained dynamic feature fusion network model;

[0012] The model testing main module is used for loading the trained dynamic feature fusion network model to classify the multi-modal breast ultrasound images to be diagnosed and identify the benign and malignant of breast tumors in the multi-modal ultrasound images.

[0013] The present application has the following advantages:

[0014] The application provides a multi-modal breast ultrasound diagnosis system based on a dynamic feature fusion network, avoids a complex ultrasound diagnosis process in a clinic, constructs a computer-aided diagnosis system capable of being used for breast cancer diagnosis by using multi-modal breast ultrasound image data, realizes rapid classification of multi-modal breast ultrasound images, and significantly improves the diagnosis efficiency and accuracy of a clinician.

[0015] The application aims to provide a multi-modal breast ultrasound diagnosis system based on a dynamic feature fusion network, which can assist doctors in rapidly diagnosing the benignity and malignancy of breast nodules in multi-modal ultrasound images.

[0016] The multi-modal breast ultrasound diagnosis system based on the dynamic feature fusion network has the following advantages:

[0017] 1. The application provides a multi-modal breast ultrasound diagnosis system based on a dynamic feature fusion network, which can sufficiently integrate the unique information of multi-modal breast ultrasound images, realize information complementation, and thus improve the performance of a model.

[0018] 2. The application adopts a deformable spatial attention module, which can adjust the positions of sampling points through a learnable offset, concentrate the attention of the network on a tumor area rich in information, and realize the attention of important information.

[0019] 3. The application provides a dynamic feature fusion module, which quantifies the contribution difference of features of different modalities to a diagnosis result through a Dirichlet distribution, dynamically allocates importance weights to the features of different modalities according to different samples, and realizes dynamic fusion of multi-modal features. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The flowchart of the system of the application is shown in the figure;

[0021] Figure 2 The structural diagram of the dynamic feature fusion network in the system of the application is shown in the figure;

[0022] Figure 3 The schematic diagram of the deformable spatial attention module in the system of the application is shown in the figure. DETAILED DESCRIPTION

[0023] Specific embodiment one: the multi-modal breast ultrasound diagnosis system based on the dynamic feature fusion network comprises:

[0024] a multi-modal data preprocessing main module, a multi-modal convolutional neural network main module, a model training main module and a model testing main module;

[0025] The multi-modal data preprocessing main module is configured to preprocess the collected multi-modal breast ultrasound images to obtain preprocessed multi-modal breast ultrasound images (with benign and malignant labels of breast tumors);

[0026] The multi-modal breast ultrasound images include brightness-mode ultrasound (BUS), color Doppler flow imaging (CDFI), and ultrasonic elastography (UE);

[0027] The multi-modal convolutional neural network main module is configured to build a dynamic feature fusion network model;

[0028] The model training main module is configured to train the built dynamic feature fusion network model using the preprocessed multi-modal breast ultrasound images to obtain a trained dynamic feature fusion network model;

[0029] The model testing main module is configured to load the trained dynamic feature fusion network model to classify the multi-modal breast ultrasound images to be diagnosed and identify the benign and malignant breast tumors in the multi-modal ultrasound images.

[0030] Specific implementation method two: the difference between this implementation method and the specific implementation method one is that the multi-modal data preprocessing main module is configured to preprocess the collected multi-modal breast ultrasound images to obtain preprocessed multi-modal breast ultrasound images; the specific process is as follows:

[0031] Step A1, multi-modal breast ultrasound images of different patients are obtained, and the regions of interest of each modality of breast ultrasound images are extracted to obtain each modality of breast ultrasound imaging region;

[0032] Each modality of breast ultrasound image is brightness-mode ultrasound, color Doppler flow imaging, or ultrasonic elastography;

[0033] The histogram equalization is performed on each modality of breast ultrasound imaging region to obtain an image after histogram equalization;

[0034] In view of the low contrast problem existing in each modality of ultrasound image, the histogram equalization method is used to enhance the contrast between breast nodules and surrounding tissues;

[0035] Step A2, the image after histogram equalization is scaled to make the short side length of the image 224 pixels;

[0036] Step A3, the center region of the image after scaling in step A2 is cropped by center cropping to obtain a standard image with a size of 224x224.

[0037] a standard image with a size of 224x224 as the final obtained preprocessed multi-modal breast ultrasound image;

[0038] the final obtained preprocessed multi-modal breast ultrasound image as a training set.

[0039] In order to enable the diagnostic model to effectively process multi-modal breast ultrasound images of different sizes, the preprocessed multi-modal ultrasound images are scaled and center cropped, and it is ensured that the breast tumor is located in the central region of each modality ultrasound image as much as possible, and finally a standard image with a size of 224x224 is obtained.

[0040] The preprocessed multi-modal breast ultrasound image is used as the input of the dynamic feature fusion network model.

[0041] The other steps and parameters are the same as those in the first embodiment.

[0042] The third embodiment is different from the first or second embodiment in that the multi-modal convolutional neural network main module is used to build a dynamic feature fusion network model, and the specific process is as follows:

[0043] The dynamic feature fusion network model sequentially comprises a multi-modal convolutional neural network, a deformable spatial attention module, a dynamic feature fusion module, a global average pooling layer, a full connection layer, and a softmax function layer.

[0044] The specific working process of the dynamic feature fusion network model is as follows:

[0045] The preprocessed multi-modal breast ultrasound image is input into the multi-modal convolutional neural network, the multi-modal convolutional neural network outputs the features which are input into the deformable spatial attention module, the deformable spatial attention module outputs the features which are input into the dynamic feature fusion module, and the dynamic feature fusion module outputs the features.

[0046] The features output by the dynamic feature fusion module are sequentially input into the global average pooling layer, the full connection layer, and the softmax function layer, and the softmax function layer outputs the probability value of each class of the preprocessed multi-modal breast ultrasound image.

[0047] The other steps and parameters are the same as those in the first or second embodiment.

[0048] The fourth embodiment is different from any one of the first to third embodiments in that the multi-modal convolutional neural network comprises branch 1, branch 2, and branch 3.

[0049] The branch 1 comprises, in sequence, a ResNet50 network, a first convolutional layer, a ReLU activation function, a batch normalization layer, a second convolutional layer, a ReLU activation function, a batch normalization layer, a third convolutional layer, a ReLU activation function, and a batch normalization layer.

[0050] The branch 2 comprises, in sequence, a ResNet50 network, a fourth convolutional layer, a ReLU activation function, a batch normalization layer, a fifth convolutional layer, a ReLU activation function, a batch normalization layer, a sixth convolutional layer, a ReLU activation function, and a batch normalization layer.

[0051] The branch 3 comprises, in sequence, a ResNet50 network, a seventh convolutional layer, a ReLU activation function, a batch normalization layer, an eighth convolutional layer, a ReLU activation function, a batch normalization layer, a ninth convolutional layer, a ReLU activation function, and a batch normalization layer.

[0052] The first convolutional layer has a convolution kernel size of 1x1, the second convolutional layer has a convolution kernel size of 3x3, and the third convolutional layer has a convolution kernel size of 1x1.

[0053] The fourth convolutional layer has a convolution kernel size of 1x1, the fifth convolutional layer has a convolution kernel size of 3x3, and the sixth convolutional layer has a convolution kernel size of 1x1.

[0054] The seventh convolutional layer has a convolution kernel size of 1x1, the eighth convolutional layer has a convolution kernel size of 3x3, and the ninth convolutional layer has a convolution kernel size of 1x1.

[0055] The specific working process of the multi-modal convolutional neural network is as follows:

[0056] The preprocessed B-mode ultrasound image is input into the branch 1, and the branch 1 outputs feature 1.

[0057] The preprocessed color Doppler blood flow imaging is input into the branch 2, and the branch 2 outputs feature 2.

[0058] The preprocessed ultrasound elastography is input into the branch 3, and the branch 3 outputs feature 3.

[0059] The multi-modal convolutional neural network is composed of three branch networks with the same structure, each branch network adopts a residual network ResNet50 as a backbone network, and the weight parameters between the branch networks are completely shared.

[0060] The branch network is composed of a ResNet50 and residual convolution modules with different channel numbers, and each residual convolution module is composed of three convolutional layers with different convolution kernels, and a ReLU activation function and a batch normalization layer are used after each convolutional layer.

[0061] The other steps and parameters are the same as one of the first to third specific embodiments.

[0062] Embodiment five: different from one of embodiments one to four is that: the deformable spatial attention module includes branch 1, branch 2 and branch 3;

[0063] The specific working process of each of branch 1, branch 2 and branch 3 is as follows:

[0064] The deformable spatial attention module is constructed as a unique feature extraction layer to enhance the network's ability to extract different modal unique features;

[0065] 1) Given an input feature map x, a reference position point p is set on the feature map x;

[0066] The input feature map x is projected into a query token q through linear mapping, and the expression is as follows:

[0067] q = x x w q

[0068] Wherein, w q is the projection matrix of the query token;

[0069] The query token q is input into a lightweight bias network θ offset , and the lightweight bias network θ offset outputs the offset Δp of the reference position point p, and the expression is as follows:

[0070] Δp = θ offset (q)

[0071] The lightweight bias network θ offset includes a convolution layer, a ReLU activation function and a convolution layer in sequence; the convolution kernel size of the convolution layer is 1x1;

[0072] The feature sampled at the offset position is transformed into a deformable key token k and a deformable value token v through linear mapping transformation, and the expression is as follows:

[0073]

[0074] Wherein, w k and w v are the projection matrices of the deformable key token and the deformable value token respectively;

[0075] is the feature sampled at the offset position;

[0076] φ(,;,) is a bilinear interpolation feature sampling function;

[0077] 2) The dimensions of the query token q, the deformable key token k and the deformable value token v are transformed into

[0078] The matrix multiplication is performed between the query token q and the transpose of the deformable key token k, and the spatial attention map s is calculated by applying the softmax function, and the calculation process is as follows:

[0079] s = softmax(q x k T )

[0080] Wherein, s represents the spatial attention map, the upper index T represents the transpose, represents a real matrix of C rows and N columns, C is the number of channels, and N is the number of features (length x width); N = H x W, H is the length of the feature map, and W is the width of the feature map;

[0081] 3) The spatial attention map s is multiplied by the deformable value token v, and the dimension of the value after the matrix multiplication is transformed to

[0082] The value after the dimension transformation is element-wise summed with the input feature map x to obtain the final output feature z (feature A1, feature A2, feature A3):

[0083] z = γ x (s x v) + x

[0084] Wherein, γ is the coefficient of the attention feature; represents a real matrix, C is the number of channels, H is the length of the feature map, and W is the width of the feature map;

[0085] The feature at each position of the final output feature z is the weighted sum of the features at all sampling positions and the input feature x.

[0086] The specific working process of the deformable spatial attention module is as follows:

[0087] The feature 1 output by branch 1 of the multi-modal convolutional neural network is input into branch 1 of the deformable spatial attention module, and branch 1 of the deformable spatial attention module outputs feature A1;

[0088] The feature 2 output by branch 2 of the multi-modal convolutional neural network is input into branch 2 of the deformable spatial attention module, and branch 2 of the deformable spatial attention module outputs feature A2;

[0089] The feature 3 output by branch 3 of the multi-modal convolutional neural network is input into branch 3 of the deformable spatial attention module, and branch 3 of the deformable spatial attention module outputs feature A3.

[0090] The other steps and parameters are the same as one of the first to fourth embodiments.

[0091] Embodiment six: The difference between this embodiment and one of the first to fifth embodiments is that the specific working process of the dynamic feature fusion module is as follows:

[0092] The dynamic feature fusion module quantifies the contribution of each modal feature to the diagnostic results under different samples, achieving dynamic weighted fusion of multimodal features. To fully utilize the extracted features of each modality, the multimodal features passed through the deformable spatial attention module are used as input to the dynamic feature fusion module. Specifically, the dynamic feature fusion module uses a subjective logic algorithm to model the output of the branch network as a Dirichlet distribution under the guidance of label information, thereby obtaining the diagnostic probability and uncertainty of the single-modal diagnosis result;

[0093] 1) Input the deformable spatial attention module output feature A1 into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability that the B-type ultrasound is judged as category k.

[0094] The output feature A2 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability that the color Doppler blood flow imaging is judged as category k.

[0095] The output feature A3 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability that the ultrasound elastic imaging is judged as category k.

[0096] The probability of being classified as category k based on modality m Calculate the Dirichlet distribution strength S m ; Based on the Dirichlet distribution strength S m Calculation correspondence The probability of diagnosis and corresponding The uncertainty u m ;

[0097] m = 1, 2, 3;

[0098] When m=1, the mode is B-mode ultrasound;

[0099] When m = 2, the modality is color Doppler flow imaging;

[0100] When m = 3, the mode is ultrasound elastography;

[0101] Based on the deformable spatial attention module, branch 1 determines the probability of the B-mode ultrasound image being classified as category k and calculates the Dirichlet distribution strength. Based on the deformable spatial attention module, branch 2 determines the probability of the color Doppler blood flow imaging being classified as category k and calculates the Dirichlet distribution strength. Based on the deformable spatial attention module, branch 3 determines the probability of the ultrasound elasticity imaging being classified as category k and calculates the Dirichlet distribution strength.

[0102] The expression is:

[0103]

[0104] in, is the probability that mode m is judged as category k (each branch corresponds to a mode);

[0105] The total number of categories K is 2, including benign and malignant;

[0106] To correspond The probability of diagnosis;

[0107] u m To correspond uncertainty;

[0108] is the concentration parameter of the Dirichlet distribution, is the Dirichlet distribution parameter;

[0109] is the Dirichlet distribution strength, and

[0110] Known, through get based on Get S m ;

[0111] Then calculate and u m ;

[0112] 2) Using uncertainty to dynamically adjust the proportion of each modal feature representation in the joint modal feature representation to achieve dynamic weighted fusion of multimodal features;

[0113] The output features of the dynamic feature fusion module are obtained based on uncertainty; the calculation formula is as follows:

[0114]

[0115] Among them, A i It is the input of the dynamic feature fusion module (also the output feature A1 of the deformable spatial attention module, the output feature A2 of the deformable spatial attention module, and the output feature A3 of the deformable spatial attention module);

[0116] F f Output features for the dynamic feature fusion module;

[0117] In addition, in order to optimize the branch network after introducing the Dirichlet distribution parameter, an improved weighted cross-entropy loss is adopted, and the loss function of the dynamic feature fusion module is as follows:

[0118]

[0119] wherein y k is the real category after one-hot encoding;

[0120] is the digamma function, θ is the weight factor, Dir is the Dirichlet distribution, and KL represents the Kullback-Leibler divergence, which is a measure of the difference between two probability distributions;

[0121] || is a separator;

[0122] is the centralized parameter of the Dirichlet distribution after replacing the label corresponding with 1 (that is If the category is 1, replace with 1, and if the category is 0, replace with 1);

[0123] This loss can make the diagnostic model generate more evidence on the output corresponding to the correct category and reduce the evidence generated on the wrong category.

[0124] The other steps and parameters are the same as one of the first to fifth embodiments.

[0125] Specific embodiment seven: different from one of the first to sixth embodiments is that the output features of the dynamic feature fusion module are sequentially input into the global average pooling layer, the fully connected layer, and the softmax function layer, and the softmax function layer outputs the probability value of the preprocessed multi-modal breast ultrasound image classified into each category.

[0126] The classification loss function is calculated based on the probability value and the real category.

[0127] The global average pooling layer compresses the extracted features of each modality branch network, and then the fully connected layer and the softmax function are used to obtain the probability value of the multi-modal breast ultrasound image classified into each category by the network.

[0128] The definition of the classification loss function is as follows:

[0129]

[0130] wherein, is the predicted probability value, y i is the real category.

[0131] The other steps and parameters are the same as one of embodiments one to six.

[0132] Embodiment eight: different from one of embodiments one to seven, the overall loss function of the dynamic feature fusion network is represented as follows:

[0133]

[0134] Wherein, M is the number of modalities (taking 3), and λ is the weight coefficient.

[0135] The other steps and parameters are the same as one of embodiments one to seven.

[0136] Embodiment nine: different from one of embodiments one to eight, the model training main module trains the built dynamic feature fusion network model by using the preprocessed multi-modal breast ultrasound images, and obtains the trained dynamic feature fusion network model; the specific process is as follows:

[0137] Step C1, initialize the parameters of the dynamic feature fusion network model;

[0138] Set the learning rate, and adjust the learning rate in the training process by using the cosine annealing learning rate adjustment strategy;

[0139] Set the maximum number of training times;

[0140] Set the weight decay parameter;

[0141] Step C2, train the dynamic feature fusion network model by using the preprocessed multi-modal breast ultrasound images as the training sample set, until the maximum training number is reached or the overall loss function of the model converges, obtain the trained dynamic feature fusion network model, and save the parameters of the trained dynamic feature fusion network model.

[0142] The other steps and parameters are the same as one of embodiments one to eight.

[0143] Embodiment ten: different from one of embodiments one to nine, the model test main module is used to load the trained dynamic feature fusion network model, classify the multi-modal breast ultrasound images to be diagnosed, and identify the benign and malignant of the breast tumors in the multi-modal ultrasound images; the specific process is as follows:

[0144] Step D1, region of interest extraction is performed on the multi-modal ultrasound images of the test set to obtain breast ultrasound imaging regions of each modality; histogram equalization is performed on the breast ultrasound imaging regions of each modality to obtain images after histogram equalization; the images after histogram equalization are scaled to obtain images after scaling processing; the center region of the images after scaling processing is cropped by a center cropping method to obtain standard images with a size of 224x224;

[0145] Step D2, the standard images with a size of 224x224 obtained in step D1 are input into the trained dynamic feature fusion network model, and the trained dynamic feature fusion network model outputs the category of the breast tumor in the multi-modal ultrasound image.

[0146] The accuracy, recall rate, precision rate and other performance indicators of the multi-modal diagnosis network on the test set are calculated.

[0147] The other steps and parameters are the same as one of the first to ninth specific embodiments.

[0148] The beneficial effects of the present application are verified by the following examples:

[0149] Example 1:

[0150] The specific embodiments of the present application will be described below in conjunction with specific examples and drawings. It should be understood that these examples are only used to illustrate the present application and not to limit the scope of the present application.

[0151] The multi-modal breast ultrasound data set used in the present study contains 1035 breast ultrasound images of different modalities from 345 patients (including 192 benign cases and 153 malignant cases). Each case in the data set contains ultrasound images of three different modalities: B-mode ultrasound, color Doppler flow, and ultrasound elastography. The data set is randomly divided into a training set and a test set in a ratio of 8:2. Among them, 276 cases are used as the training set to update the model parameters; 69 cases are used as the test set to test the final model.

[0152] Step 1 is performed:

[0153] Step A1, multi-modal breast ultrasound images of different patients are obtained, region of interest extraction is performed on the breast ultrasound images of each modality, and histogram equalization is performed on the breast ultrasound imaging regions of each modality;

[0154] Step A2, the images after histogram equalization are scaled to have a short side length of 224 pixels; the center region of the images after scaling processing is cropped by a center cropping method to obtain standard images with a size of 224x224, which are used as the final preprocessed multi-modal breast ultrasound images;

[0155] Performing step two:

[0156] Performing step B1, constructing a multi-modal convolutional neural network, including branch 1, branch 2 and branch 3;

[0157] Branch 1 includes, in sequence, a ResNet50 network, a first convolutional layer, a ReLU activation function, a batch normalization layer, a second convolutional layer, a ReLU activation function, a batch normalization layer, a third convolutional layer, a ReLU activation function, and a batch normalization layer;

[0158] Branch 2 includes, in sequence, a ResNet50 network, a fourth convolutional layer, a ReLU activation function, a batch normalization layer, a fifth convolutional layer, a ReLU activation function, a batch normalization layer, a sixth convolutional layer, a ReLU activation function, and a batch normalization layer;

[0159] Branch 3 includes, in sequence, a ResNet50 network, a seventh convolutional layer, a ReLU activation function, a batch normalization layer, an eighth convolutional layer, a ReLU activation function, a batch normalization layer, a ninth convolutional layer, a ReLU activation function, and a batch normalization layer;

[0160] The first convolutional layer has a convolution kernel size of 1x1, the second convolutional layer has a convolution kernel size of 3x3, and the third convolutional layer has a convolution kernel size of 1x1;

[0161] The fourth convolutional layer has a convolution kernel size of 1x1, the fifth convolutional layer has a convolution kernel size of 3x3, and the sixth convolutional layer has a convolution kernel size of 1x1;

[0162] The seventh convolutional layer has a convolution kernel size of 1x1, the eighth convolutional layer has a convolution kernel size of 3x3, and the ninth convolutional layer has a convolution kernel size of 1x1;

[0163] The specific working process of the multi-modal convolutional neural network is as follows:

[0164] The preprocessed B-mode ultrasound image is input into branch 1, and branch 1 outputs feature 1;

[0165] The preprocessed color Doppler blood flow imaging is input into branch 2, and branch 2 outputs feature 2;

[0166] The preprocessed ultrasound elastography is input into branch 3, and branch 3 outputs feature 3.

[0167] Performing step B2, constructing a deformable spatial attention module.

[0168] The specific working process of the deformable spatial attention module is as follows:

[0169] Feature 1 output by branch 1 of the multi-modal convolutional neural network is input into branch 1 of the deformable spatial attention module, and branch 1 of the deformable spatial attention module outputs feature A1;

[0170] The feature 2 output by branch 2 in the multi-modal convolutional neural network is input into branch 2 of the deformable spatial attention module, and branch 2 of the deformable spatial attention module outputs feature A2;

[0171] The feature 3 output by branch 3 in the multi-modal convolutional neural network is input into branch 3 of the deformable spatial attention module, and branch 3 of the deformable spatial attention module outputs feature A3.

[0172] Step B3 is performed, and the dynamic feature fusion module is constructed.

[0173] The feature A1 output by the deformable spatial attention module is sequentially input into a full connection layer and a ReLU activation function layer, and the ReLU activation function layer outputs the probability of the B-mode ultrasound being classified as class k

[0174] The feature A2 output by the deformable spatial attention module is sequentially input into a full connection layer and a ReLU activation function layer, and the ReLU activation function layer outputs the probability of the color Doppler blood flow imaging being classified as class k

[0175] The feature A3 output by the deformable spatial attention module is sequentially input into a full connection layer and a ReLU activation function layer, and the ReLU activation function layer outputs the probability of the ultrasound elastography being classified as class k

[0176] The probability of the modality m being classified as class k The Dirichlet distribution strength S is calculated m The diagnostic probability corresponding to m is calculated The uncertainty u corresponding to m ;

[0177] m=1, 2, 3;

[0178] When m=1, the modality is B-mode ultrasound;

[0179] When m=2, the modality is color Doppler blood flow imaging;

[0180] When m=3, the modality is ultrasound elastography;

[0181] The expression is:

[0182]

[0183] wherein, is the probability of the modality m being classified as class k;

[0184] ​​The total number of categories K is 2, including benign and malignant;

[0185] For the diagnostic probability of ;

[0186] u m For the uncertainty of ;

[0187] is the concentration parameter of the Dirichlet distribution, is the Dirichlet distribution parameter;

[0188] is the intensity of the Dirichlet distribution, and

[0189] It is known that is obtained by Based on , S m is obtained;

[0190] Then and u m are calculated;

[0191] The dynamic feature fusion module output feature is obtained based on the uncertainty; the calculation formula is as follows:

[0192]

[0193] Wherein, A i is the input of the dynamic feature fusion module (also the deformable spatial attention module output feature A1, the deformable spatial attention module output feature A2, the deformable spatial attention module output feature A3);

[0194] F f is the dynamic feature fusion module output feature; F f is input into the global average pooling layer, the fully connected layer and the Softmax function layer in turn to obtain the predicted probability value

[0195] The loss function of the dynamic feature fusion module is as follows:

[0196]

[0197] Wherein, y k is the real category after one-hot encoding;

[0198] is the digamma function, θ is the weight factor, Dir is the Dirichlet distribution, and KL represents the KL divergence;

[0199] || is a separator;

[0200] is the label corresponding to the concentration parameter of Dirichlet distribution after replacing 1 with 1 if the category is 1, and with 1 if the category is 0;

[0201] Step B4 is performed, and a classification loss function is calculated based on the probability value and the true category; the definition of the classification loss function is as follows:

[0202]

[0203] wherein, is the predicted probability value, y i is the true category.

[0204] Step three is performed:

[0205] Step C1 is performed, the network is initialized, the weights of the dynamic feature fusion network are trained using the Adam optimizer, and the learning rate is adjusted using the cosine annealing strategy, and the cycle number and cycle multiplier are set to 10 and 2 respectively. The diagnostic model is trained with an initial learning rate of 2e-5, the weight decay is set to 1e-5, the batch size is set to 16, and the iteration is 150 rounds.

[0206] Step C2 is performed, the pre-processed multi-modal breast ultrasound images are used as the training sample set to train the dynamic feature fusion network model until the maximum training round is reached or the model overall loss function converges, the trained dynamic feature fusion network model is obtained, and the trained dynamic feature fusion network model parameters are saved.

[0207] Step four is performed:

[0208] Step D1, the region of interest of the test set of multi-modal ultrasound images is extracted, and the histogram equalization is performed on each modality of breast ultrasound imaging region; the image after histogram equalization is scaled, and the center region of the scaled image is cropped by center cropping, to obtain a standard image with a size of 224x224;

[0209] Step D2, the 224x224 standard image obtained in step D1 is input into the trained dynamic feature fusion network model, and the trained dynamic feature fusion network model outputs the category of breast tumor in the multi-modal ultrasound image.

[0210] The performance indicators such as accuracy, recall rate and precision rate of the multi-modal diagnostic network on the test set are calculated.

[0211] ​The present application can have other various embodiments, and those skilled in the art can make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application, and these corresponding changes and modifications shall all belong to the protection scope of the claims of the present application.

Claims

1. A multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network, characterized by: The system comprises: Multimodal data preprocessing main module, multimodal convolutional neural network main module, model training main module and model testing main module; The multimodal data preprocessing main module is used to preprocess the collected multimodal breast ultrasound images to obtain the preprocessed multimodal breast ultrasound images; The multimodal breast ultrasound image includes B-mode ultrasound, color Doppler blood flow imaging and ultrasound elastography; The multimodal convolutional neural network main module is used to build a dynamic feature fusion network model; The model training main module uses the pre-processed multimodal breast ultrasound images to train the constructed dynamic feature fusion network model to obtain a trained dynamic feature fusion network model; The model testing main module is used to load the trained dynamic feature fusion network model, classify the multimodal breast ultrasound images to be diagnosed, and identify the benign and malignant breast tumors in the multimodal ultrasound images; The multimodal convolutional neural network main module is used to build a dynamic feature fusion network model; the specific process is: The dynamic feature fusion network model includes a multimodal convolutional neural network, a deformable spatial attention module, a dynamic feature fusion module, a global average pooling layer, a fully connected layer, and a softmax function layer. The specific working process of the dynamic feature fusion network model is as follows: The preprocessed multimodal breast ultrasound image is input into the multimodal convolutional neural network, the output features of the multimodal convolutional neural network are input into the deformable spatial attention module, the output features of the deformable spatial attention module are input into the dynamic feature fusion module, and the dynamic feature fusion module outputs features; The output features of the dynamic feature fusion module are sequentially input into the global average pooling layer, the fully connected layer, and the softmax function layer. The softmax function layer outputs the probability value of each category of the preprocessed multimodal breast ultrasound image. The multimodal convolutional neural network includes branch 1, branch 2 and branch 3; Branch 1 includes the ResNet50 network, the first convolutional layer, the ReLU activation function, the batch normalization layer, the second convolutional layer, the ReLU activation function, the batch normalization layer, the third convolutional layer, the ReLU activation function, and the batch normalization layer. Branch 2 includes the ResNet50 network, the fourth convolutional layer, the ReLU activation function, the batch normalization layer, the fifth convolutional layer, the ReLU activation function, the batch normalization layer, the sixth convolutional layer, the ReLU activation function, and the batch normalization layer. Branch 3 includes the ResNet50 network, the seventh convolutional layer, the ReLU activation function, the batch normalization layer, the eighth convolutional layer, the ReLU activation function, the batch normalization layer, the ninth convolutional layer, the ReLU activation function, and the batch normalization layer. The convolution kernel size of the first convolution layer is 1×1, the convolution kernel size of the second convolution layer is 3×3, and the convolution kernel size of the third convolution layer is 1×1; The convolution kernel size of the fourth convolution layer is 1×1, the convolution kernel size of the fifth convolution layer is 3×3, and the convolution kernel size of the sixth convolution layer is 1×1; The convolution kernel size of the seventh convolution layer is 1×1, the convolution kernel size of the eighth convolution layer is 3×3, and the convolution kernel size of the ninth convolution layer is 1×1; The specific working process of the multimodal convolutional neural network is as follows: The preprocessed B-mode ultrasound image is input into branch 1, and branch 1 outputs feature 1; The preprocessed color Doppler blood flow imaging is input into branch 2, and branch 2 outputs feature 2; The preprocessed ultrasound elastography is input into branch 3, and branch 3 outputs feature 3; The deformable spatial attention module includes branch 1, branch 2 and branch 3; The specific working process of the deformable spatial attention module is as follows: Input feature 1 output by branch 1 in the multimodal convolutional neural network into branch 1 of the deformable spatial attention module, and branch 1 of the deformable spatial attention module outputs feature A1; Input feature 2 output by branch 2 in the multimodal convolutional neural network into branch 2 of the deformable spatial attention module, and branch 2 of the deformable spatial attention module outputs feature A2; Input feature 3 output by branch 3 in the multimodal convolutional neural network into branch 3 of the deformable spatial attention module, and branch 3 of the deformable spatial attention module outputs feature A3; The output feature A1 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability of B-type ultrasound being judged as category k. The output feature A2 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability that the color Doppler blood flow imaging is judged as category k. The output feature A3 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability that the ultrasound elastic imaging is judged as category k.

2. The multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network according to claim 1, characterized in that: The multimodal data preprocessing main module is used to preprocess the collected multimodal breast ultrasound images to obtain preprocessed multimodal breast ultrasound images; the specific process is: Step A1: acquiring multimodal breast ultrasound images of different patients, extracting regions of interest from breast ultrasound images of each modality, and obtaining breast ultrasound imaging regions of each modality; Each modality of breast ultrasound image was B-mode ultrasound, color Doppler flow imaging, or ultrasound elastography; Performing histogram equalization on breast ultrasound imaging regions of each modality to obtain an image after histogram equalization; Step A2: scaling the image after histogram equalization so that the length of the short side of the image is 224 pixels; Step A3: cropping the central area of ​​the image after the scaling process in step A2 by means of center cropping to obtain a standard image of size 224×224; The standard image of 224×224 size is used as the final preprocessed multimodal breast ultrasound image; The final preprocessed multimodal breast ultrasound images are used as the training set.

3. The multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network according to claim 2, characterized in that: The specific working process of each branch in Branch 1, Branch 2 and Branch 3 is as follows: 1) Given an input feature map x, set a reference position point p on the feature map x; The input feature map x is projected into the query token q through a linear mapping, which is expressed as: q=x×w q Among them, w q is the projection matrix of the query token; Input the query token q to the lightweight bias network θ offset , lightweight bias network θ offset Output the offset Δp of the reference position point p, expressed as: Δp=θ offset (q) The lightweight bias network θ offset It includes convolution layer, ReLU activation function, and convolution layer in sequence; the convolution kernel size of the convolution layer is 1×1; The features sampled at the offset position are transformed by linear mapping Transformed into a deformable key token k and a deformable value token v; the expression is: Among them, w k and w v are the projection matrices for the deformable key token and the deformable value token, respectively; is the feature sampled at the offset position; φ(,;,) is the bilinear interpolation feature sampling function; 2) Transform the dimensions of query token q, deformable key token k and deformable value token v into A matrix multiplication is performed between the query token q and the transpose of the deformable key token k, and a softmax function is applied to compute the spatial attention map, which is calculated as follows: s=softmax(q×k T ) Among them, s represents the spatial attention map, and the superscript T represents the transposition. Represents a real matrix with C rows and N columns, where C is the number of channels and N is the number of features; 3) Perform matrix multiplication on the spatial attention map s and the deformable value token v, and transform the dimension of the value after matrix multiplication to Perform element-wise summation on the transformed value and the input feature map x to obtain the final output feature z: z=γ×(s×v)+x Among them, γ is the coefficient of attention feature; Represents a real matrix, C is the number of channels, H is the length of the feature map, and W is the width of the feature map.

4. The multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network according to claim 3, characterized in that: The specific working process of the dynamic feature fusion module also includes: 1)、 The probability of being classified as category k based on modality m Calculate the Dirichlet distribution strength S m ; Based on the Dirichlet distribution strength S m Calculation correspondence The probability of diagnosis and corresponding The uncertainty u m ; m＝1、2、3； When m=1, the mode is B-mode ultrasound; When m = 2, the modality is color Doppler flow imaging; When m = 3, the mode is ultrasound elastography; The expression is: in, is the probability that mode m is judged as category k; The total number of categories K is 2, including benign and malignant; To correspond The probability of diagnosis; u m To correspond uncertainty; is the concentration parameter of the Dirichlet distribution, is the Dirichlet distribution parameter; is the Dirichlet distribution strength, and 2)、 The output features of the dynamic feature fusion module are obtained based on uncertainty; the calculation formula is as follows: Among them, A i It is the input of the dynamic feature fusion module; F f Output features for the dynamic feature fusion module; The loss function of the dynamic feature fusion module is as follows: Among them, y k is the true category after one-hot encoding; is the digamma function, θ is the weight factor, Dir is the Dirichlet distribution, and KL represents the KL divergence; || is a separator; The label corresponds to The concentration parameter of the Dirichlet distribution after being replaced by 1.

5. The multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network according to claim 4, characterized in that: The output features of the dynamic feature fusion module are sequentially input into the global average pooling layer, the fully connected layer, and the softmax function layer, and the softmax function layer outputs the probability value of each category of the preprocessed multimodal breast ultrasound image; Calculate the classification loss function based on the probability value and the true category; The classification loss function is defined as follows: in, is the predicted probability value, y i is the real category.

6. The multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network according to claim 5, characterized in that: The overall loss function of the dynamic feature fusion network is expressed as follows: Where M is the number of modes and λ is the weight coefficient.

7. The multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network according to claim 6, characterized in that: The model training main module uses the pre-processed multimodal breast ultrasound images to train the established dynamic feature fusion network model and obtains the trained dynamic feature fusion network model; the specific process is: Step C1, initializing the parameters of the dynamic feature fusion network model; Set the learning rate; Set the maximum number of training sessions; Set weight decay parameters; Step C2: Using the preprocessed multimodal breast ultrasound images as a training sample set to train the dynamic feature fusion network model until the maximum number of training rounds is reached or the overall loss function of the model converges, a trained dynamic feature fusion network model is obtained, and the trained dynamic feature fusion network model parameters are saved.

8. The multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network according to claim 7, characterized in that: The model testing main module is used to load the trained dynamic feature fusion network model, classify the multimodal breast ultrasound images to be diagnosed, and identify the benign and malignant breast tumors in the multimodal ultrasound images. The specific process is as follows: Step D1: extracting regions of interest from the multimodal ultrasound images of the test set to obtain breast ultrasound imaging regions of each modality; performing histogram equalization on the breast ultrasound imaging regions of each modality to obtain a histogram-equalized image; scaling the histogram-equalized image to obtain a scaled image; and cropping the center region of the scaled image by center cropping to obtain a standard image of 224×224 size. Step D2: Input the 224×224 size standard image obtained in step D1 into the trained dynamic feature fusion network model, and the trained dynamic feature fusion network model outputs the category of breast tumors in the multimodal ultrasound image.

Images