Multi-modal breast ultrasonic diagnosis system based on dynamic feature fusion network

Abstract

The invention discloses a multi-mode breast ultrasonic diagnosis system based on a dynamic feature fusion network, belongs to the field of medical image processing, and particularly relates to the multi-mode breast ultrasonic diagnosis system based on the dynamic feature fusion network. The invention aims to solve the problem that the existing method is low in breast ultrasonic recognition accuracy. The system comprises a multi-modal data preprocessing main module used for preprocessing a collected multi-modal mammary gland ultrasound image to obtain a preprocessed multi-modal mammary gland ultrasound image; the multi-modal convolutional neural network main module is used for building a dynamic feature fusion network model; the model training main module is used for training an established dynamic feature fusion network model by using the preprocessed multi-modal breast ultrasound image to obtain a trained dynamic feature fusion network model; and the model test main module is used for loading the trained dynamic feature fusion network model, classifying a to-be-diagnosed multi-modal breast ultrasound image, and identifying benign and malignant breast tumors in the multi-modal ultrasound image.

Description

Technical Field

[0001] The invention belongs to the field of medical image processing, and in particular relates to a multimodal breast ultrasound diagnosis system based on a dynamic feature fusion network. Background Art

[0002] Breast cancer is one of the most common malignant tumors in women worldwide and the second leading cause of cancer death in women. The latest data from the World Health Organization shows that in 2020, there were 2.26 million new cases of breast cancer worldwide, resulting in 685,000 deaths. This makes breast cancer surpass lung cancer and become the most common cancer in the world. Early diagnosis and timely treatment can significantly improve the survival and cure rates of breast cancer patients. At present, common diagnostic methods for breast cancer include ultrasound examination, mammography, magnetic resonance imaging and pathological examination. Among them, breast ultrasound has become an important medical imaging technology in breast cancer screening and diagnosis due to its advantages such as low price, non-invasiveness and no radiation. However, traditional breast ultrasound diagnosis relies heavily on the rich experience and subjective evaluation of clinicians, which can easily lead to missed diagnosis or misdiagnosis, which cannot be completely avoided even by experienced professionals. In challenging cases, the morphological characteristics of benign and malignant lesions in B-mode ultrasound (Brightness-mode Ultrasound, BUS) images are very small or even contradictory, which limits the accuracy of diagnosis to a certain extent.

[0003] In order to solve the problem of limited diagnostic accuracy of single-modality ultrasound, doctors have incorporated color Doppler flow imaging (CDFI) and ultrasonicelastography (UE) into the standard diagnostic examination of breast cancer on the basis of traditional BUS. This integration has significantly improved the diagnostic accuracy of breast cancer. Specifically, BUS provides morphological information about the shape, direction, and edge of breast nodules; CDFI provides detailed information about the blood flow inside and around breast nodules, while UE provides quantitative information about the hardness of breast nodules and their surrounding tissues. Especially in malignant tumors, CDFI and UE images usually show rich blood flow signals and obvious hardening. The combination of these ultrasound modes enables doctors to make more accurate decisions. However, the process of integrating these breast ultrasound images from different modes to accurately identify malignant tumors requires additional expertise and clinical experience. Therefore, it is of great practical significance to develop a high-performance computer-aided diagnosis system that can accurately diagnose breast cancer through multimodal ultrasound images.

[0004] In recent years, with the rapid development of artificial intelligence technology, scholars have conducted in-depth research on computer-aided diagnosis systems, which can achieve accurate diagnosis of breast cancer through multimodal ultrasound. However, existing multimodal diagnostic methods still face many challenges. On the one hand, traditional diagnostic methods based on convolutional neural networks are usually difficult to extract global information from multimodal ultrasound images and can only focus on local information within a relatively small receptive field. On the other hand, the contribution of different modalities to the diagnostic results usually varies significantly, and existing algorithms treat all modal information equally and cannot achieve dynamic fusion of different modal features. Summary of the invention

[0005] In order to solve the problem that the existing multimodal diagnosis method based on deep learning lacks the ability to extract global features and cannot realize the dynamic fusion of multimodal features, resulting in low accuracy of breast ultrasound recognition, the present invention proposes a multimodal breast ultrasound diagnosis system based on a dynamic feature fusion network.

[0006] A multimodal breast ultrasound diagnosis system based on a dynamic feature fusion network includes:

[0007] Multimodal data preprocessing main module, multimodal convolutional neural network main module, model training main module and model testing main module;

[0008] The multimodal data preprocessing main module is used to preprocess the collected multimodal breast ultrasound images to obtain the preprocessed multimodal breast ultrasound images;

[0009] The multimodal breast ultrasound image includes B-mode ultrasound, color Doppler blood flow imaging and ultrasound elasticity imaging;

[0010] The multimodal convolutional neural network main module is used to build a dynamic feature fusion network model;

[0011] The model training main module uses the preprocessed multimodal breast ultrasound images to train the constructed dynamic feature fusion network model to obtain a trained dynamic feature fusion network model;

[0012] The model test main module is used to load the trained dynamic feature fusion network model, classify the multimodal breast ultrasound images to be diagnosed, and identify the benign and malignant nature of breast tumors in the multimodal ultrasound images.

[0013] The beneficial effects of the present invention are:

[0014] The present invention proposes a multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network, which avoids the complicated ultrasound diagnostic process in the clinic. By using multimodal breast ultrasound image data, a computer-aided diagnosis system that can be used for breast cancer diagnosis is constructed, which realizes the rapid classification of multimodal breast ultrasound images and significantly improves the diagnostic efficiency and accuracy of clinicians.

[0015] The purpose of the present invention is to propose a multimodal breast ultrasound diagnosis system based on a dynamic feature fusion network, which can assist doctors in quickly diagnosing the benign and malignant nature of breast nodules in multimodal ultrasound images. Existing auxiliary diagnosis systems can usually only consider traditional B-type ultrasound images, but cannot make full use of different modal ultrasound images for joint diagnosis. The present invention can overcome these shortcomings.

[0016] The multimodal breast ultrasound diagnosis system based on dynamic feature fusion network of the present invention has the following advantages:

[0017] 1. The present invention proposes a multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network. This method can fully integrate the unique information of multimodal breast ultrasound images to achieve information complementarity, thereby improving the performance of the model.

[0018] 2. The present invention adopts a deformable spatial attention module, which can adjust the position of the sampling points through a learnable offset, focus the network's attention on the information-rich tumor area, and achieve attention to important information.

[0019] 3. The present invention proposes a dynamic feature fusion module, which quantifies the difference in contribution of each modal feature to the diagnosis result through Dirichlet distribution, thereby dynamically assigning importance weights to each modal feature according to different samples, thereby realizing dynamic fusion of multimodal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of the system of the present invention;

[0021] Figure 2 It is a schematic diagram of the architecture of the dynamic feature fusion network in the system of the present invention;

[0022] Figure 3 Schematic diagram of the deformable spatial attention module in the system of the present invention. DETAILED DESCRIPTION

[0023] Specific implementation method 1: In this implementation method, a multimodal breast ultrasound diagnosis system based on a dynamic feature fusion network includes:

[0024] Multimodal data preprocessing main module, multimodal convolutional neural network main module, model training main module and model testing main module;

[0025] The multimodal data preprocessing main module is used to preprocess the collected multimodal breast ultrasound images to obtain the preprocessed multimodal breast ultrasound images (with benign or malignant labels of breast tumors);

[0026] The multimodal breast ultrasound images include B-mode ultrasound (Brightness-mode Ultrasound, BUS), color Doppler flow imaging (color Doppler flow imaging, CDFI) and ultrasonicelastography (ultrasonicelastography, UE);

[0027] The multimodal convolutional neural network main module is used to build a dynamic feature fusion network model;

[0028] The model training main module uses the preprocessed multimodal breast ultrasound images to train the constructed dynamic feature fusion network model to obtain a trained dynamic feature fusion network model;

[0029] The model test main module is used to load the trained dynamic feature fusion network model, classify the multimodal breast ultrasound images to be diagnosed, and identify the benign and malignant nature of breast tumors in the multimodal ultrasound images.

[0030] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that: the multimodal data preprocessing main module is used to preprocess the collected multimodal breast ultrasound images to obtain the preprocessed multimodal breast ultrasound images; the specific process is:

[0031] Step A1, obtaining multimodal breast ultrasound images of different patients, extracting regions of interest from breast ultrasound images of each modality, and obtaining breast ultrasound imaging regions of each modality;

[0032] Each modality of breast ultrasound image was B-mode ultrasound, color Doppler blood flow imaging, or ultrasound elasticity imaging;

[0033] Performing histogram equalization on breast ultrasound imaging regions of each modality to obtain an image after histogram equalization;

[0034] Aiming at the low contrast problem of ultrasound images of various modalities, the histogram equalization method is used to enhance the contrast between breast nodules and surrounding tissues.

[0035] Step A2, scaling the image after histogram equalization so that the length of the short side of the image is 224 pixels;

[0036] Step A3, cropping the central area of ​​the image after the scaling process in step A2 by means of center cropping, to obtain a standard image of size 224×224;

[0037] The standard image of 224×224 size is used as the final preprocessed multimodal breast ultrasound image;

[0038] The final preprocessed multimodal breast ultrasound images are used as the training set.

[0039] In order to enable the diagnostic model to effectively process multimodal breast ultrasound images of different sizes, the preprocessed multimodal ultrasound images were scaled and center-cropped, and the breast tumor was ensured to be located in the central area of ​​each modality ultrasound image as much as possible, and finally a standard image of 224×224 size was obtained.

[0040] The preprocessed multimodal breast ultrasound images are used as the input of the dynamic feature fusion network model.

[0041] The other steps and parameters are the same as those in the first embodiment.

[0042] Specific implementation method three: This implementation method is different from specific implementation method one or two in that: the multimodal convolutional neural network main module is used to build a dynamic feature fusion network model; the specific process is:

[0043] The dynamic feature fusion network model includes a multimodal convolutional neural network, a deformable spatial attention module, a dynamic feature fusion module, a global average pooling layer, a fully connected layer, and a softmax function layer in sequence;

[0044] The specific working process of the dynamic feature fusion network model is as follows:

[0045] The preprocessed multimodal breast ultrasound image is input into the multimodal convolutional neural network, the output features of the multimodal convolutional neural network are input into the deformable spatial attention module, the output features of the deformable spatial attention module are input into the dynamic feature fusion module, and the dynamic feature fusion module outputs features;

[0046] The output features of the dynamic feature fusion module are sequentially input into the global average pooling layer, the fully connected layer, and the softmax function layer. The softmax function layer outputs the probability value of each category of the preprocessed multimodal breast ultrasound image.

[0047] The other steps and parameters are the same as those in the first or second embodiment.

[0048] Specific embodiment 4: This embodiment is different from any one of specific embodiments 1 to 3 in that: the multimodal convolutional neural network includes branch 1, branch 2 and branch 3;

[0049] Branch 1 includes ResNet50 network, first convolution layer, ReLU activation function, batch normalization layer, second convolution layer, ReLU activation function, batch normalization layer, third convolution layer, ReLU activation function, batch normalization layer in sequence;

[0050] Branch 2 includes ResNet50 network, fourth convolution layer, ReLU activation function, batch normalization layer, fifth convolution layer, ReLU activation function, batch normalization layer, sixth convolution layer, ReLU activation function, batch normalization layer in sequence;

[0051] Branch 3 includes ResNet50 network, seventh convolution layer, ReLU activation function, batch normalization layer, eighth convolution layer, ReLU activation function, batch normalization layer, ninth convolution layer, ReLU activation function, batch normalization layer in sequence;

[0052] The convolution kernel size of the first convolution layer is 1×1, the convolution kernel size of the second convolution layer is 3×3, and the convolution kernel size of the third convolution layer is 1×1;

[0053] The convolution kernel size of the fourth convolution layer is 1×1, the convolution kernel size of the fifth convolution layer is 3×3, and the convolution kernel size of the sixth convolution layer is 1×1;

[0054] The convolution kernel size of the seventh convolution layer is 1×1, the convolution kernel size of the eighth convolution layer is 3×3, and the convolution kernel size of the ninth convolution layer is 1×1;

[0055] The specific working process of the multimodal convolutional neural network is as follows:

[0056] The preprocessed B-mode ultrasound image is input into branch 1, and branch 1 outputs feature 1;

[0057] The preprocessed color Doppler blood flow imaging is input into branch 2, and branch 2 outputs feature 2;

[0058] The preprocessed ultrasound elastic imaging is input into branch 3, and branch 3 outputs feature 3.

[0059] The multimodal convolutional neural network consists of three branch networks with the same structure. Each branch network uses the residual network ResNet50 as the backbone network, and the weight parameters between the branch networks are completely shared.

[0060] The branch network consists of ResNet50 and residual convolution modules with different numbers of channels, and each residual convolution module consists of three convolution layers with different convolution kernels, and each convolution layer is followed by a ReLU activation function and a batch normalization layer.

[0061] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0062] Specific implementation five: This implementation differs from any one of specific implementations one to four in that: the deformable spatial attention module includes branch 1, branch 2 and branch 3;

[0063] The specific working process of each branch in branch 1, branch 2 and branch 3 is as follows:

[0064] Construct a deformable spatial attention module as a unique feature extraction layer to enhance the network's ability to extract unique features of different modalities;

[0065] 1) Given an input feature map x, set a reference position point p on the feature map x;

[0066] The input feature map x is projected into the query token q through a linear mapping, expressed as:

[0067] q=x×w q

[0068] Among them, w q is the projection matrix of the query token;

[0069] The query token q is fed into the lightweight bias network θ offset , lightweight bias network θ offset Output the offset Δp of the reference position point p, expressed as:

[0070] Δp=θ offset (q)

[0071] The lightweight bias network θ offset It includes convolution layer, ReLU activation function, and convolution layer in sequence; the convolution kernel size of the convolution layer is 1×1;

[0072] The features sampled at the offset position are transformed by linear mapping Transformed into a morphable key token k and a morphable value token v; the expression is:

[0073]

[0074] Among them, w k and w v are the projection matrices for the morphable key token and the morphable value token, respectively;

[0075] is the feature sampled at the offset position;

[0076] φ(,;,) is the bilinear interpolation feature sampling function;

[0077] 2) Transform the dimensions of query token q, deformable key token k and deformable value token v into

[0078] A matrix multiplication is performed between the query token q and the transpose of the morphable key token k, and a softmax function is applied to compute the spatial attention map, which is calculated as follows:

[0079] s=softmax(q×k T )

[0080] Among them, s represents the spatial attention map, and the superscript T represents the transposition. Represents a real matrix with C rows and N columns, where C is the number of channels and N is the number of features (length times width); N = H × W, where H is the length of the feature map and W is the width of the feature map;

[0081] 3) Perform matrix multiplication on the spatial attention map s and the deformable value token v, and transform the dimension of the value after matrix multiplication to

[0082] The transformed value is element-wise summed with the input feature map x to obtain the final output feature z (feature A1, feature A2, feature A3):

[0083] z=γ×(s×v)+x

[0084] Among them, γ is the coefficient of the attention feature; represents a real matrix, C is the number of channels, H is the length of the feature map, and W is the width of the feature map;

[0085] The feature of each position on the final output feature z is the weighted sum of the features at all sampled positions and the input feature x.

[0086] The specific working process of the deformable spatial attention module is as follows:

[0087] Input feature 1 output by branch 1 in the multimodal convolutional neural network into branch 1 of the deformable spatial attention module, and branch 1 of the deformable spatial attention module outputs feature A1;

[0088] Input feature 2 output by branch 2 in the multimodal convolutional neural network into branch 2 of the deformable spatial attention module, and branch 2 of the deformable spatial attention module outputs feature A2;

[0089] Feature 3 output by branch 3 in the multimodal convolutional neural network is input into branch 3 of the deformable spatial attention module, and branch 3 of the deformable spatial attention module outputs feature A3.

[0090] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0091] Specific implementation method 6: This implementation method is different from the specific implementation methods 1 to 5 in that: the specific working process of the dynamic feature fusion module is:

[0092] The dynamic feature fusion module quantifies the contribution of each modal feature to the diagnosis result under different samples, and realizes the dynamic weighted fusion of multimodal features. In order to make full use of the extracted features of each modality, the multimodal features after the deformable spatial attention module are used as the input of the dynamic feature fusion module. Specifically, the dynamic feature fusion module can use the subjective logic algorithm under the guidance of label information to model the output of the branch network as a Dirichlet distribution, thereby obtaining the diagnostic probability and uncertainty of the single-modal diagnosis result;

[0094] 1) Input the deformable spatial attention module output feature A1 into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability that the B-type ultrasound is judged as category k.

[0095] The output feature A2 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability of color Doppler blood flow imaging being judged as category k.

[0096] The output feature A3 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability of ultrasound elastic imaging being judged as category k.

[0097] The probability of being classified as category k based on mode m Calculate the Dirichlet distribution strength S m ; Based on the Dirichlet distribution strength S m Calculation correspondence The probability of diagnosis and correspondence The uncertainty u m ;

[0098] m = 1, 2, 3;

[0099] When m=1, the mode is B-mode ultrasound;

[0100] When m=2, the mode is color Doppler blood flow imaging;

[0101] When m=3, the mode is ultrasound elastography;

[0102] Based on the probability of the B-type ultrasound image being classified as category k, branch 1 of the deformable spatial attention module calculates the Dirichlet distribution intensity; based on the probability of the color Doppler blood flow imaging being classified as category k, branch 2 of the deformable spatial attention module calculates the Dirichlet distribution intensity; based on the probability of the ultrasound elastic imaging being classified as category k, branch 3 of the deformable spatial attention module calculates the Dirichlet distribution intensity;

[0103] The expression is:

[0104]

[0105] in, is the probability that mode m is judged as category k (each branch corresponds to a mode);

[0106] The total number of categories K is 2, including benign and malignant;

[0107] For the corresponding The probability of diagnosis;

[0108] u m For the corresponding uncertainty;

[0109] is the concentration parameter of the Dirichlet distribution, is the Dirichlet distribution parameter;

[0110] is the Dirichlet distribution strength, and

[0111] Known, through get based on Get S m ;

[0112] Then calculate and u m ;

[0113] 2) Using uncertainty to dynamically adjust the proportion of each modal feature representation in the joint modal feature representation to achieve dynamic weighted fusion of multi-modal features;

[0114] The output features of the dynamic feature fusion module are obtained based on uncertainty; the calculation formula is as follows:

[0115]

[0116] Among them, A i It is the input of the dynamic feature fusion module (also the output feature A1 of the deformable spatial attention module, the output feature A2 of the deformable spatial attention module, and the output feature A3 of the deformable spatial attention module);

[0117] F f Output features for the dynamic feature fusion module;

[0118] In addition, in order to optimize the branch network after introducing the Dirichlet distribution parameters, an improved weighted cross entropy loss is used. The loss function of the dynamic feature fusion module is as follows:

[0119]

[0120] Among them, y k is the true category after one-hot encoding;

[0121] is the digamma function, θ is the weight factor, Dir is the Dirichlet distribution, and KL represents the Kullback-Leibler divergence, which is a measure of the difference between two probability distributions.

[0122] || is a separator;

[0123] The label corresponds to The concentration parameter of the Dirichlet distribution after being replaced by 1 (that is If the category is 1, Replace with 1, if the category is 0, Replace with 1);

[0124] This loss enables the diagnostic model to generate more evidence on the output corresponding to the correct category and reduce the evidence generated on the wrong category.

[0125] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0126] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that: the output features of the dynamic feature fusion module are sequentially input into the global average pooling layer, the fully connected layer, and the softmax function layer, and the softmax function layer outputs the probability value of each category of the preprocessed multimodal breast ultrasound image;

[0127] Calculate the classification loss function based on the probability value and the true category;

[0128] The global average pooling layer compresses the extracted features of each modality branch network, and then the fully connected layer and softmax function are used to obtain the probability value of the network classifying the multimodal breast ultrasound image into each category.

[0129] The classification loss function is defined as follows:

[0130]

[0131] in, is the predicted probability value, y i is the real category.

[0132] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0133] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the overall loss function of the dynamic feature fusion network is expressed as follows:

[0134]

[0135] Among them, M is the number of modes (take 3), and λ is the weight coefficient.

[0136] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0137] Specific implementation method 9: This implementation method is different from specific implementation methods 1 to 8 in that the model training main module uses the pre-processed multimodal breast ultrasound image to train the built dynamic feature fusion network model and obtain the trained dynamic feature fusion network model; the specific process is:

[0138] Step C1, initializing the parameters of the dynamic feature fusion network model;

[0139] Set the learning rate and use the cosine annealing learning rate adjustment strategy to adjust the learning rate during training;

[0140] Set the maximum number of training times;

[0141] Set weight decay parameters;

[0142] Step C2: train the dynamic feature fusion network model using the preprocessed multimodal breast ultrasound images as a training sample set until the maximum number of training rounds is reached or the overall loss function of the model converges, obtain the trained dynamic feature fusion network model, and save the trained dynamic feature fusion network model parameters.

[0143] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0144] Specific implementation method ten: This implementation method is different from specific implementation methods one to nine in that the model test main module is used to load the trained dynamic feature fusion network model, classify the multimodal breast ultrasound images to be diagnosed, and identify the benign and malignant breast tumors in the multimodal ultrasound images; the specific process is:

[0145] Step D1, extracting the region of interest from the multimodal ultrasound image of the test set to obtain the breast ultrasound imaging region of each modality; performing histogram equalization on the breast ultrasound imaging region of each modality to obtain the image after the histogram equalization; scaling the image after the histogram equalization to obtain the scaled image; cropping the central area of ​​the scaled image by center cropping to obtain a standard image of 224×224 size;

[0146] Step D2: input the 224×224 size standard image obtained in step D1 into the trained dynamic feature fusion network model, and the trained dynamic feature fusion network model outputs the category of breast tumors in the multimodal ultrasound image.

[0147] Calculate the performance indicators of the multimodal diagnosis network on the test set, such as accuracy, recall, and precision.

[0148] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0149] The following examples are used to verify the beneficial effects of the present invention:

[0150] Embodiment 1:

[0151] The specific implementation of the present invention is described below in conjunction with specific embodiments and drawings. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0152] The multimodal breast ultrasound dataset used in this study contains 1035 breast ultrasound images of different modalities from 345 patients (including 192 benign cases and 153 malignant cases). Each case in this dataset contains ultrasound images of three different modalities: B-mode ultrasound, color Doppler blood flow, and ultrasound elastography. The dataset was randomly divided into a training set and a test set in a ratio of 8:2. Among them, 276 cases were used as training sets to update model parameters; 69 cases were used as test sets to test the final model.

[0153] Perform step 1:

[0154] Execute step A1, obtain multimodal breast ultrasound images of different patients, extract regions of interest from breast ultrasound images of each modality, and perform histogram equalization on breast ultrasound imaging regions of each modality;

[0155] Execute step A2, scale the image after histogram equalization so that the length of the short side of the image is 224 pixels; crop the central area of ​​the scaled image by center cropping to obtain a standard image of 224×224 size as the final preprocessed multimodal breast ultrasound image;

[0156] Execute step 2:

[0157] Execute step B1, construct a multimodal convolutional neural network, including branch 1, branch 2 and branch 3;

[0158] Branch 1 includes ResNet50 network, first convolution layer, ReLU activation function, batch normalization layer, second convolution layer, ReLU activation function, batch normalization layer, third convolution layer, ReLU activation function, batch normalization layer in sequence;

[0159] Branch 2 includes ResNet50 network, fourth convolution layer, ReLU activation function, batch normalization layer, fifth convolution layer, ReLU activation function, batch normalization layer, sixth convolution layer, ReLU activation function, batch normalization layer in sequence;

[0160] Branch 3 includes ResNet50 network, seventh convolution layer, ReLU activation function, batch normalization layer, eighth convolution layer, ReLU activation function, batch normalization layer, ninth convolution layer, ReLU activation function, batch normalization layer in sequence;

[0161] The convolution kernel size of the first convolution layer is 1×1, the convolution kernel size of the second convolution layer is 3×3, and the convolution kernel size of the third convolution layer is 1×1;

[0162] The convolution kernel size of the fourth convolution layer is 1×1, the convolution kernel size of the fifth convolution layer is 3×3, and the convolution kernel size of the sixth convolution layer is 1×1;

[0163] The convolution kernel size of the seventh convolution layer is 1×1, the convolution kernel size of the eighth convolution layer is 3×3, and the convolution kernel size of the ninth convolution layer is 1×1;

[0164] The specific working process of the multimodal convolutional neural network is as follows:

[0165] The preprocessed B-mode ultrasound image is input into branch 1, and branch 1 outputs feature 1;

[0166] The preprocessed color Doppler blood flow imaging is input into branch 2, and branch 2 outputs feature 2;

[0167] The preprocessed ultrasound elastic imaging is input into branch 3, and branch 3 outputs feature 3.

[0168] Execute step B2 and build a deformable spatial attention module.

[0169] The specific working process of the deformable spatial attention module is as follows:

[0170] Input feature 1 output by branch 1 in the multimodal convolutional neural network into branch 1 of the deformable spatial attention module, and branch 1 of the deformable spatial attention module outputs feature A1;

[0171] Input feature 2 output by branch 2 in the multimodal convolutional neural network into branch 2 of the deformable spatial attention module, and branch 2 of the deformable spatial attention module outputs feature A2;

[0172] Feature 3 output by branch 3 in the multimodal convolutional neural network is input into branch 3 of the deformable spatial attention module, and branch 3 of the deformable spatial attention module outputs feature A3.

[0173] Execute step B3 to build a dynamic feature fusion module.

[0174] The output feature A1 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability of B-type ultrasound being judged as category k.

[0175] The output feature A2 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability of color Doppler blood flow imaging being judged as category k.

[0176] The output feature A3 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability of ultrasound elastic imaging being judged as category k.

[0177] The probability of being classified as category k based on mode m Calculate the Dirichlet distribution strength S m ; Based on the Dirichlet distribution strength S m Calculation correspondence The probability of diagnosis and correspondence The uncertainty u m ;

[0178] m = 1, 2, 3;

[0179] When m=1, the mode is B-mode ultrasound;

[0180] When m=2, the mode is color Doppler blood flow imaging;

[0181] When m=3, the mode is ultrasound elastography;

[0182] The expression is:

[0183]

[0184] in, is the probability that mode m is judged as category k;

[0185] The total number of categories K is 2, including benign and malignant;

[0186] For the corresponding The probability of diagnosis;

[0187] u m For the corresponding uncertainty;

[0188] is the concentration parameter of the Dirichlet distribution, is the Dirichlet distribution parameter;

[0189] is the Dirichlet distribution strength, and

[0190] Known, through get based on Get S m ;

[0191] Then calculate and u m ;

[0192] The output features of the dynamic feature fusion module are obtained based on uncertainty; the calculation formula is as follows:

[0193]

[0194] Among them, A i It is the input of the dynamic feature fusion module (also the output feature A1 of the deformable spatial attention module, the output feature A2 of the deformable spatial attention module, and the output feature A3 of the deformable spatial attention module);

[0195] F f Output features of the dynamic feature fusion module; F f Input the global average pooling layer, the fully connected layer and the Softmax function layer in sequence to obtain the predicted probability value

[0196] The loss function of the dynamic feature fusion module is as follows:

[0197]

[0198] Among them, y k is the true category after one-hot encoding;

[0199] is the digamma function, θ is the weight factor, Dir is the Dirichlet distribution, and KL represents the KL divergence;

[0200] || is a separator;

[0201] The label corresponds to The concentration parameter of the Dirichlet distribution after being replaced by 1 (that is If the category is 1, Replace with 1, if the category is 0, Replace with 1);

[0202] Execute step B4, calculate the classification loss function based on the probability value and the true category; the classification loss function is defined as follows:

[0203]

[0204] in, is the predicted probability value, y i is the real category.

[0205] Perform step three:

[0206] Execute step C1, initialize the network, use the Adam optimizer to train the weights of the dynamic feature fusion network, and use the cosine annealing strategy to adjust the learning rate, with the number of cycles and cycle multiplier set to 10 and 2 respectively. The diagnostic model is trained with an initial learning rate of 2e-5, weight decay set to 1e-5, batch size set to 16, and iterated for 150 rounds.

[0207] Execute step C2, use the preprocessed multimodal breast ultrasound images as the training sample set to train the dynamic feature fusion network model until the maximum number of training rounds is reached or the overall loss function of the model converges, obtain the trained dynamic feature fusion network model, and save the trained dynamic feature fusion network model parameters.

[0208] Execute step 4:

[0209] Step D1, extracting the region of interest from the multimodal ultrasound image of the test set, and performing histogram equalization on the breast ultrasound imaging area of ​​each modality; scaling the image after the histogram equalization, and cropping the central area of ​​the scaled image by center cropping to obtain a standard image of 224×224 size;

[0210] Step D2: input the 224×224 size standard image obtained in step D1 into the trained dynamic feature fusion network model, and the trained dynamic feature fusion network model outputs the category of breast tumors in the multimodal ultrasound image.

[0211] Calculate the performance indicators of the multimodal diagnosis network on the test set, such as accuracy, recall, and precision.

[0212] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network, characterized in that: The system comprises: Multimodal data preprocessing main module, multimodal convolutional neural network main module, model training main module and model testing main module; The multimodal data preprocessing main module is used to preprocess the collected multimodal breast ultrasound images to obtain the preprocessed multimodal breast ultrasound images; The multimodal breast ultrasound image includes B-mode ultrasound, color Doppler blood flow imaging and ultrasound elasticity imaging; The multimodal convolutional neural network main module is used to build a dynamic feature fusion network model; The model training main module uses the preprocessed multimodal breast ultrasound images to train the constructed dynamic feature fusion network model to obtain a trained dynamic feature fusion network model; The model test main module is used to load the trained dynamic feature fusion network model, classify the multimodal breast ultrasound images to be diagnosed, and identify the benign and malignant nature of breast tumors in the multimodal ultrasound images.

2. The multimodal breast ultrasound diagnostic system based on a dynamic feature fusion network according to claim 1, characterized in that: The multimodal data preprocessing main module is used to preprocess the collected multimodal breast ultrasound images to obtain the preprocessed multimodal breast ultrasound images; the specific process is: Step A1, obtaining multimodal breast ultrasound images of different patients, extracting regions of interest from breast ultrasound images of each modality, and obtaining breast ultrasound imaging regions of each modality; Each modality of breast ultrasound image was B-mode ultrasound, color Doppler blood flow imaging, or ultrasound elasticity imaging; Performing histogram equalization on breast ultrasound imaging regions of each modality to obtain an image after histogram equalization; Step A2, scaling the image after histogram equalization so that the length of the short side of the image is 224 pixels; Step A3, cropping the central area of ​​the image after the scaling process in step A2 by means of center cropping, to obtain a standard image of size 224×224; The standard image of 224×224 size is used as the final preprocessed multimodal breast ultrasound image; The final preprocessed multimodal breast ultrasound images are used as the training set.

3. The multimodal breast ultrasound diagnostic system based on dynamic feature fusion network according to claim 2, characterized in that: The multimodal convolutional neural network main module is used to build a dynamic feature fusion network model; the specific process is: The dynamic feature fusion network model includes a multimodal convolutional neural network, a deformable spatial attention module, a dynamic feature fusion module, a global average pooling layer, a fully connected layer, and a softmax function layer in sequence; The specific working process of the dynamic feature fusion network model is as follows: The preprocessed multimodal breast ultrasound image is input into the multimodal convolutional neural network, the output features of the multimodal convolutional neural network are input into the deformable spatial attention module, the output features of the deformable spatial attention module are input into the dynamic feature fusion module, and the dynamic feature fusion module outputs features; The output features of the dynamic feature fusion module are sequentially input into the global average pooling layer, the fully connected layer, and the softmax function layer. The softmax function layer outputs the probability value of each category of the preprocessed multimodal breast ultrasound image.

4. The multimodal breast ultrasound diagnostic system based on dynamic feature fusion network according to claim 3 is characterized in that: The multimodal convolutional neural network includes branch 1, branch 2 and branch 3; Branch 1 includes ResNet50 network, first convolution layer, ReLU activation function, batch normalization layer, second convolution layer, ReLU activation function, batch normalization layer, third convolution layer, ReLU activation function, batch normalization layer in sequence; Branch 2 includes ResNet50 network, fourth convolution layer, ReLU activation function, batch normalization layer, fifth convolution layer, ReLU activation function, batch normalization layer, sixth convolution layer, ReLU activation function, batch normalization layer in sequence; Branch 3 includes ResNet50 network, seventh convolution layer, ReLU activation function, batch normalization layer, eighth convolution layer, ReLU activation function, batch normalization layer, ninth convolution layer, ReLU activation function, batch normalization layer in sequence; The convolution kernel size of the first convolution layer is 1×1, the convolution kernel size of the second convolution layer is 3×3, and the convolution kernel size of the third convolution layer is 1×1; The convolution kernel size of the fourth convolution layer is 1×1, the convolution kernel size of the fifth convolution layer is 3×3, and the convolution kernel size of the sixth convolution layer is 1×1; The convolution kernel size of the seventh convolution layer is 1×1, the convolution kernel size of the eighth convolution layer is 3×3, and the convolution kernel size of the ninth convolution layer is 1×1; The specific working process of the multimodal convolutional neural network is as follows: The preprocessed B-mode ultrasound image is input into branch 1, and branch 1 outputs feature 1; The preprocessed color Doppler blood flow imaging is input into branch 2, and branch 2 outputs feature 2; The preprocessed ultrasound elastic imaging is input into branch 3, and branch 3 outputs feature 3.

5. The multimodal breast ultrasound diagnostic system based on dynamic feature fusion network according to claim 4, characterized in that: The deformable spatial attention module includes branch 1, branch 2 and branch 3; The specific working process of each branch in branch 1, branch 2 and branch 3 is as follows: 1) Given an input feature map x, set a reference position point p on the feature map x; The input feature map x is projected into the query token q through a linear mapping, expressed as: q=x×w q Among them, w q is the projection matrix of the query token; The query token q is fed into the lightweight bias network θ offset , lightweight bias network θ offset Output the offset Δp of the reference position point p, expressed as: Δp=θ offset (q) The lightweight bias network θ offset It includes convolution layer, ReLU activation function, and convolution layer in sequence; the convolution kernel size of the convolution layer is 1×1; The features sampled at the offset position are transformed by linear mapping Transformed into a morphable key token k and a morphable value token v; the expression is: Among them, w k and w v are the projection matrices for the morphable key token and the morphable value token, respectively; is the feature sampled at the offset position; φ(,;,) is the bilinear interpolation feature sampling function; 2) Transform the dimensions of query token q, deformable key token k and deformable value token v into A matrix multiplication is performed between the query token q and the transpose of the morphable key token k, and a softmax function is applied to compute the spatial attention map, which is calculated as follows: s=softmax(q×k T ) Among them, s represents the spatial attention map, and the superscript T represents the transposition. represents a real matrix with C rows and N columns, where C is the number of channels and N is the number of features; 3) Perform matrix multiplication on the spatial attention map s and the deformable value token v, and transform the dimension of the value after matrix multiplication to Perform element-wise summation of the transformed dimension value with the input feature map x to obtain the final output feature z: z=γ×(s×v)+x Among them, γ is the coefficient of the attention feature; represents a real matrix, C is the number of channels, H is the length of the feature map, and W is the width of the feature map; The specific working process of the deformable spatial attention module is as follows: Input feature 1 output by branch 1 in the multimodal convolutional neural network into branch 1 of the deformable spatial attention module, and branch 1 of the deformable spatial attention module outputs feature A1; Input feature 2 output by branch 2 in the multimodal convolutional neural network into branch 2 of the deformable spatial attention module, and branch 2 of the deformable spatial attention module outputs feature A2; Feature 3 output by branch 3 in the multimodal convolutional neural network is input into branch 3 of the deformable spatial attention module, and branch 3 of the deformable spatial attention module outputs feature A3.

6. The multimodal breast ultrasound diagnostic system based on dynamic feature fusion network according to claim 5, characterized in that: The specific working process of the dynamic feature fusion module is as follows: 1)、 The output feature A1 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability of B-type ultrasound being judged as category k. The output feature A2 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability of color Doppler blood flow imaging being judged as category k. The output feature A3 of the deformable spatial attention module is input into the fully connected layer and the ReLU activation function layer in sequence. The ReLU activation function layer outputs the probability of ultrasound elastic imaging being judged as category k. The probability of being classified as category k based on mode m Calculate the Dirichlet distribution strength S m ; Based on the Dirichlet distribution strength S m Calculation correspondence The probability of diagnosis and correspondence The uncertainty u m ; m＝1、2、3； When m=1, the mode is B-mode ultrasound; When m=2, the mode is color Doppler blood flow imaging; When m=3, the mode is ultrasound elastography; The expression is: in, is the probability that mode m is judged as category k; The total number of categories K is 2, including benign and malignant; For the corresponding The probability of diagnosis; u m For the corresponding uncertainty; is the concentration parameter of the Dirichlet distribution, is the Dirichlet distribution parameter; is the Dirichlet distribution strength, and 2)、 The output features of the dynamic feature fusion module are obtained based on uncertainty; the calculation formula is as follows: Among them, A i It is the input of the dynamic feature fusion module; F f Output features for the dynamic feature fusion module; The loss function of the dynamic feature fusion module is as follows: Among them, y k is the true category after one-hot encoding; is the digamma function, θ is the weight factor, Dir is the Dirichlet distribution, and KL represents the KL divergence; || is a separator; The label corresponds to The concentration parameter of the Dirichlet distribution after replacement by 1.

7. The multimodal breast ultrasound diagnostic system based on dynamic feature fusion network according to claim 6, characterized in that: The output features of the dynamic feature fusion module are sequentially input into the global average pooling layer, the fully connected layer, and the softmax function layer, and the softmax function layer outputs the probability value of each category of the preprocessed multimodal breast ultrasound image; Calculate the classification loss function based on the probability value and the true category; The classification loss function is defined as follows: in, is the predicted probability value, y i is the real category.

8. The multimodal breast ultrasound diagnosis system based on dynamic feature fusion network according to claim 7, characterized in that: The overall loss function of the dynamic feature fusion network is expressed as follows: Among them, M is the number of modes and λ is the weight coefficient.

9. The multimodal breast ultrasound diagnosis system based on dynamic feature fusion network according to claim 8, characterized in that: The model training main module uses the pre-processed multimodal breast ultrasound images to train the built dynamic feature fusion network model and obtains the trained dynamic feature fusion network model; the specific process is: Step C1, initializing the parameters of the dynamic feature fusion network model; Set the learning rate; Set the maximum number of training times; Set weight decay parameters; Step C2: train the dynamic feature fusion network model using the preprocessed multimodal breast ultrasound images as a training sample set until the maximum number of training rounds is reached or the overall loss function of the model converges, obtain the trained dynamic feature fusion network model, and save the trained dynamic feature fusion network model parameters.

10. The multimodal breast ultrasound diagnostic system based on dynamic feature fusion network according to claim 9, characterized in that: The model test main module is used to load the trained dynamic feature fusion network model, classify the multimodal breast ultrasound images to be diagnosed, and identify the benign and malignant breast tumors in the multimodal ultrasound images; the specific process is: Step D1, extracting the region of interest from the multimodal ultrasound image of the test set to obtain the breast ultrasound imaging region of each modality; performing histogram equalization on the breast ultrasound imaging region of each modality to obtain the image after the histogram equalization; scaling the image after the histogram equalization to obtain the scaled image; cropping the central area of ​​the scaled image by center cropping to obtain a standard image of 224×224 size; Step D2: input the 224×224 size standard image obtained in step D1 into the trained dynamic feature fusion network model, and the trained dynamic feature fusion network model outputs the category of breast tumors in the multimodal ultrasound image.

Images