Multi-modal ultrasonic prediction method based on hierarchical attention fusion

Abstract

The invention discloses a multi-modal ultrasonic prediction method based on hierarchical attention fusion, and belongs to the field of medical information processing.The prediction method comprises the following steps that firstly, ultrasonic image features of each modal are extracted through Swin Transform V2; secondly, splicing the ultrasonic image features of the four modes, and inputting the spliced ultrasonic image features into a constructed ultrasonic self-attention and convolution fusion module to realize coarse fusion of multi-mode-multi-scale features; a designed multi-modal cross attention fusion module is adopted to realize hierarchical fusion of global-local information, and fine fusion of multi-modal information is realized; and finally, inputting the final cross attention fusion feature into a classifier, and further outputting a tumor prediction result. According to the method, the application of tumor prediction on the ultrasonic image based on the hierarchical attention fusion model is realized, the accuracy and robustness of tumor diagnosis are remarkably improved, and the method has clinical popularization and application values.

Description

Technical Field

[0001] The present invention belongs to the field of medical information processing, and in particular relates to a multimodal ultrasound prediction method based on hierarchical attention fusion. Background Art

[0002] Breast cancer is one of the most common malignant tumors in women worldwide, and its early and accurate diagnosis is crucial to reducing mortality. Ultrasound imaging has become the preferred imaging technology for breast cancer screening due to its advantages such as being non-invasive, low-cost and radiation-free. In clinical practice, the comprehensive analysis of four types of modal images, namely B-type, color Doppler, elastic imaging and contrast ultrasound, is widely used. Among them, B-type ultrasound and contrast ultrasound can provide global anatomical structure and blood flow distribution information of the lesion, while color Doppler and elastic imaging focus on local hemodynamics and tissue hardness characteristics. However, existing technologies still have significant limitations in the analysis and fusion of multimodal ultrasound images.

[0003] Although traditional convolutional neural networks are widely used in medical image processing, it is difficult to effectively model the long-range dependencies between multimodal ultrasound images, resulting in insufficient utilization of cross-modal complementary information. Existing fusion methods mostly rely on simple feature concatenation or weighted summation, fail to fully combine local details with global semantic features, and easily ignore subtle differences in lesions. Although the Transformer model (such as Vision Transformer) performs well in natural image processing, it faces problems such as high computational complexity and loss of local details when directly applied to medical images, and it is difficult to adapt to the high resolution and noise interference characteristics of ultrasound images. In addition, existing methods are insufficient in the optimization of hierarchical feature interactions of multimodal data, resulting in classification accuracy and AUC values ​​that are difficult to meet clinical needs.

[0004] In recent years, researchers have tried to improve the multimodal fusion effect by improving the model architecture. The Swin Transformer reduces computational complexity through the window self-attention mechanism, but its cross-modal interaction capabilities in medical multimodal scenarios are still insufficient. Although the cross-attention-based fusion method can enhance modal complementarity, it does not combine the local feature extraction advantages of the convolutional network, resulting in information redundancy or missing in the fusion results. In addition, the existing methods do not fully explore the feature complementarity between global modalities (such as B-ultrasound, contrast imaging) and local modalities (such as color Doppler, elastic imaging), making it difficult to capture key diagnostic patterns.

[0005] The above problems restrict the further application of multimodal ultrasound in breast cancer diagnosis. How to achieve efficient fusion of multi-scale features, balance the interaction of global and local information, and improve the robustness and interpretability of the model have become technical bottlenecks that need to be solved urgently. Summary of the invention

[0006] To solve the above problems, the present invention proposes a multi-modal ultrasound prediction method based on hierarchical attention fusion, which can extract convolutional features and long-range dependence features of multi-modal ultrasound, achieve effective fusion of multi-scale features, and through designing a multi-modal ultrasound cross-attention fusion module, perform hierarchical fusion on the global information and local information of multi-modal ultrasound, fully exploit the complementary features between different modalities, and accurately capture key information beneficial to diagnosis.

[0007] A multi-modal ultrasound prediction method based on hierarchical attention fusion of the present invention includes the following steps:

[0008] Step 1, obtain multi-modal ultrasound data of a tumor, where the multi-modal ultrasound data includes B-mode ultrasound, color Doppler, elastography, and contrast-enhanced ultrasound images, and perform unified formatting processing to obtain the formatted B-mode ultrasound, color Doppler, elastography, and contrast-enhanced ultrasound images;

[0009] Step 2, respectively input the formatted B-mode ultrasound, color Doppler, elastography, and contrast-enhanced ultrasound images into the Swin Transformer V2 model to respectively extract ultrasound image features, and obtain the features of the B-mode ultrasound, color Doppler, elastography, and contrast-enhanced ultrasound images;

[0010] Step 3, after splicing the ultrasound image features of the four modalities, obtain the spliced ultrasound image features; respectively extract the long-range dependence features and local features of the spliced ultrasound image features by using a self-attention and convolutional dual-branch feature extraction structure, and perform rough fusion of multi-modal - multi-scale features based on the long-range dependence features and local features;

[0011] Step 4, expand the rough fusion features along the channel dimension into 4 unfolded features F′ B , F′ D , F′ C and F′ E ;

[0012] Use the multi-modal cross-attention fusion module to respectively implement the following hierarchical fusion: perform global information fusion on the B-mode ultrasound feature F′ B and the contrast-enhanced ultrasound image feature F′ E to obtain the global cross-attention fusion feature; perform local information fusion on the color Doppler feature F′ D and the elastography feature F′ C to obtain the local cross-attention fusion feature; perform cross-fusion on the global cross-attention fusion feature and the local cross-attention fusion feature to obtain the final cross-attention fusion feature.

[0013] Step 5: Input the final cross-attention fusion feature into a classifier to obtain the prediction result for the multimodal ultrasound data.

[0014] Furthermore, the B-ultrasound and contrast-enhanced ultrasound images of the tumor only include the tumor area; the unified formatting process refers to performing pixel size normalization on the B-ultrasound, color Doppler, elastography, and contrast-enhanced ultrasound images of the tumor, and simultaneously performing data augmentation through color jittering, random cropping, random erasing, and mirror transformation.

[0015] Furthermore, there are a total of four Swin Transformer V2 models in Step 2, and the structures of the four Swin Transformer V2 models are the same; the formatted B-ultrasound, color Doppler, elastography, and contrast-enhanced ultrasound images are respectively input into the Swin Transformer V2 models to extract ultrasound image features, obtaining the feature F of B-ultrasound B , the feature F of color Doppler D , the feature F of elastography C and the feature F of contrast-enhanced ultrasound image E .

[0016] Furthermore, Step 3 is specifically as follows:

[0017] Step 3.1: Concatenate the feature F of B-ultrasound B , the feature F of color Doppler D , the feature F of elastography C and the feature F of contrast-enhanced ultrasound image E along the channel dimension to generate an initial fusion feature matrix F I ∈R H×W×4C , where H×W is the feature space dimension and C is the number of single-modal feature channels;

[0018] Perform channel projection on the initial fusion feature matrix F I through three groups of 1×1 convolutional kernels to generate three groups of intermediate feature matrices {F1, F2, F3} ∈R H×W×C , and concatenate them along the channel dimension to obtain the concatenated ultrasound image feature F N ∈R H ×W×3C .

[0019] Step 3.2: Input the concatenated ultrasound image feature F N into the self-attention feature extraction branch to extract the long-range dependence feature of the concatenated ultrasound image feature F N , that is, the self-attention feature F Att ;

[0020] Input the spliced ultrasonic image feature F N into the convolutional feature extraction branch to extract the local features of the spliced ultrasonic image feature F N specifically: use a convolutional kernel of size k×k to perform local feature extraction on F N to generate the convolutional feature F Con ; where k≥3;

[0021] Step 3.3, perform weighted summation on the self-attention feature F Att and the convolutional feature F Con to obtain the rough fusion feature F AC of the multi-modal multi-scale feature, that is:

[0022] F AC =αF Att +βF Con ;

[0023] where α and β are learnable weight parameters, dynamically optimized through backpropagation, and satisfy α + β = 1.

[0024] Further, the specific steps of Step 3.2 are as follows:

[0025] Divide the spliced ultrasonic image feature F N evenly into N groups, and each group contains three self-attention sub-features where, is the query feature of the self-attention of the i-th group,. is the key feature of the self-attention of the i-th group, is the value feature of the self-attention of the i-th group;

[0026] The self-attention feature of the i-th group is calculated as:

[0027]

[0028] where d is the dimension of the key feature , is the scaling factor, T is the transpose of the matrix, and Softmax is the activation function; splice the N groups of self-attention features to form the self-attention feature F Att The self-attention feature F Att is the output feature of the self-attention feature extraction branch.

[0029] Further, the steps of implementing the fusion using the multi-modal cross-attention fusion module are as follows:

[0030] Step 4.1: Suppose there are two modal features U1 and U2. First, perform linear projections on U1 and U2 respectively to generate the query vector Q1, the key vector K1, and the value vector V1. The calculation formula is:

[0031]

[0032] Among them, and are the learnable weight matrices of Q1, K1, and V1 respectively;

[0033] Step 4.2: Calculate the preliminary cross-attention fusion feature F1 based on the query vector Q1, key vector K1, and value vector V1 using cross-attention, that is:

[0034]

[0035] Among them, is the dimension of the key vector K1, is the scaling factor.

[0036] Step 4.3: Perform quadratic linear projection on the preliminary cross-attention fusion feature F1 and the modality feature U1 respectively to generate the query vector Q2, key vector K2, and value vector V2. The calculation formula is:

[0037]

[0038] Among them, are the learnable weight matrices of Q2, K2, and V2 respectively;

[0039] Step 4.4: Calculate the secondary cross-attention fusion feature F2 based on the query vector Q2, key vector K2, and value vector V2 using cross-attention, that is

[0040]

[0041] Among them, is the dimension of the key vector K2, is the scaling factor.

[0042] Step 4.5: Exchange the fusion order of the modality features U1 and U2, and repeat the above steps 4.1 - 4.4 to generate another set of secondary cross-attention fusion features T2. The final output cross-attention fusion feature O is:

[0043] O = δF2 + (1 - δ)T2

[0044] where δ is a learnable weight parameter.

[0045] Furthermore, a multi-modal cross-attention fusion module is adopted to achieve three-level hierarchical fusion: the B-mode ultrasound feature F′ B and the contrast-enhanced ultrasound image feature F′ E perform global information fusion, that is, the B-mode ultrasound feature F′ B and the contrast-enhanced ultrasound image feature F′E Respectively as feature U1 and feature U2, a multi-modal cross-attention fusion module is used to perform global information fusion according to steps 4.1 - 4.5 to obtain the global cross-attention fusion feature O g ;

[0046] Perform local information fusion on the color Doppler feature F′ D and the elastography feature F′ C , that is, take the color Doppler feature F′ D and the elastography feature F′ C as U1 and U2 respectively, and use a multi-modal cross-attention fusion module to perform local information fusion according to steps 4.1 - 4.5 to obtain the local cross-attention fusion feature O l ; Perform final cross-fusion on the global cross-attention fusion feature O g and the local cross-attention fusion feature O l to obtain the final cross-attention fusion feature O g-l .

[0047] Furthermore, the classifier described in step 5 is a joint classifier including a 1D convolutional neural network and a multi-layer perceptron. The specific steps of step 5 are as follows:

[0048] Step 5.1: Input the final cross-attention fusion feature O g-l into the 1D convolutional neural network to generate a dimensionality-reduced feature The specific formula is:

[0049]

[0050] where Conv1D is the 1D convolution operation.

[0051] Step 5.2: Input the dimensionality-reduced feature into the multi-layer perceptron module and perform the following operations in sequence:

[0052] Perform normalization on the dimensionality-reduced feature to obtain the normalized feature O norm :

[0053] Apply the ReLU activation function to reduce the dimension of the normalized feature O norm from 768 to 512 to obtain the activated feature O hidden1 :

[0054] O hidden = ReLU(Linear 768→512 (O norm ));

[0055] Randomly mask some neurons with probability p = 0.1 to prevent overfitting and obtain the overfitting-prevented feature O drop : O drop = Dropout(O hidden1 );

[0056] Apply the ReLU activation function again to compress the dimension of the overfitting-prevented feature O drop from 512 to 256 to obtain the activated feature O hidden2 :

[0057] O hidden2 = ReLU(Linear 512→256 (O drop ));

[0058] Finally, map the dimension of the activated feature O hidden2 from 256 to 2 dimensions to obtain the linear feature

[0059]

[0060] Step 5.3: Apply the Softmax activation function to the linear feature and convert the linear feature into the prediction probability P for tumors:

[0061]

[0062] Beneficial effects: The present invention proposes a multi-modal ultrasound prediction method based on hierarchical attention fusion, innovatively constructs a multi-modal data fusion module and a hierarchical attention mechanism model, performs multi-modal feature extraction, multi-scale feature fusion and classification tasks on tumor ultrasound images, and further realizes the accurate diagnosis of tumors. It has the following advantages:

[0063] (1) By introducing the Swin Transformer V2 structure to extract multi-modal features from B-mode ultrasound, color Doppler, elastography and contrast-enhanced imaging, it effectively captures the unique and rich features of each modality.

[0064] (2) Design an ultrasound self-attention and convolution fusion module, combine the self-attention mechanism and the convolution network, realize the adaptive learning of multi-scale features, and further improve the model's recognition ability for complex image patterns.

[0065] (3) Propose a multi-modal ultrasound cross-attention fusion module, which can efficiently integrate the complementary information of different modalities through hierarchical cross-modal feature fusion, fully consider the complex relationship between global and local features, and optimize the fusion effect of multi-modal data.

[0066] (4) Based on the multi-modal fusion data, a joint classifier combining a one-dimensional convolutional neural network and a multi-layer perceptron is designed. This joint architecture can not only reduce the dimension of high-dimensional data, reduce the number of parameters and the risk of overfitting of the multi-layer perceptron, but also efficiently achieve the task of diagnosing the benign and malignant of tumors. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The drawings generally illustrate various embodiments by way of example rather than limitation, and are used together with the description and the claims to explain the embodiments of the invention. Where appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the device or method.

[0068] Figure 1 is a schematic flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0070] An embodiment of the present invention provides a multi-modal ultrasound intelligent prediction method based on hierarchical attention fusion. The specific steps are as follows:

[0071] Step 1: Uniformly format the B-ultrasound, color Doppler, elastography and contrast-enhanced ultrasound images of the tumor collected by different devices.

[0072] When processing B-ultrasound and contrast-enhanced ultrasound images, there should be a certain outer contour area around the tumor to ensure the availability of global information; when processing color Doppler and elastography ultrasound images, only the tumor area is retained to fully display the colors of different internal tissue structures of the tumor and obtain the local information of the tumor.

[0073] Uniformly adjust the images of each modality to a size of 256×256 pixels and perform standardization processing. At the same time, data augmentation is performed by means of color jitter, random cropping, random erasing and mirror transformation.

[0074] Step 2: Input the B-ultrasound, color Doppler, elastography and contrast-enhanced ultrasound images of the four modalities into the Swin Transformer V2 model to extract the ultrasound image feature maps respectively. Specifically:

[0075] Four Swin Transformer V2 models with the same structure are used to process the ultrasonic images of four modalities respectively. The Swin Transformer V2 model captures features at different levels from coarse to fine through the window multi-head attention - shifted window multi-head attention module, and obtains the features F of B-mode ultrasound, color Doppler, elastography, and contrast-enhanced ultrasound images respectively B , F D , F C and F E .

[0076] Step 3: After splicing the ultrasonic image features of the four modalities, input them into the constructed ultrasonic self-attention and convolution fusion module. The self-attention and convolution dual-branch feature extraction structure is used to extract the long-range dependence features and local features of the spliced ultrasonic image features respectively, realizing the rough fusion of multi-modal and multi-scale features. The specific steps are as follows:

[0077] Step 3.1: Splice the features F B , F D , F C and F E of B-mode ultrasound, Doppler, elastography, and contrast-enhanced ultrasound along the channel dimension to generate the initial fusion feature matrix F I ∈R H×W×4C , where H×W is the feature space dimension and C is the number of single-modal feature channels;

[0078] Step 3.2: Perform channel projection on F I through three groups of 1×1 convolution kernels to generate three groups of intermediate feature matrices {F1, F2, F3}∈R H×W×C , and splice them along the channel dimension into F N ∈R H×W×3C .

[0079] Step 3.3: In the self-attention feature extraction branch, divide F N evenly into N groups, and each group contains three self-attention sub-features which are the query feature, key feature, and value feature of the self-attention of the i-th group respectively. The self-attention feature of the i-th group can be calculated as:

[0080]

[0081] where d is the dimension of the key feature , is the scaling factor, T is the transpose of the matrix, and Softmax is the activation function. Finally, splice the N groups of self-attention features to form the output feature F Att of the self-attention feature extraction branch.

[0082] In the convolutional feature extraction branch, a convolutional kernel of size k×k (k≥3) is used to perform local feature extraction on F N to generate the convolutional feature F Con .

[0083] Step 3.4: Perform weighted summation on the output features F Att and F Con of the self-attention branch and the convolutional branch to obtain the rough fusion feature F AC of the multi-modal multi-scale features, that is: F AC =αF Att +βF Con , where α and β are learnable weight parameters, dynamically optimized through backpropagation, and satisfy α+β=1. Their initial values are set to α=0.5 and β=0.5.

[0084] Step 4: First, unfold the rough fusion feature F AC along the channel dimension into four unfolded features F′ B , F′ D , F′ C and F′ E corresponding to B-mode ultrasound, color Doppler, elastography, and contrast-enhanced imaging; use the designed multi-modal cross-attention fusion module to achieve the following hierarchical fusion: perform global information fusion on B-mode ultrasound and contrast-enhanced ultrasound images with global modalities; perform local information fusion on color Doppler and elastography with local modalities; perform final cross-fusion on the above fused global information and local information. The designed multi-modal cross-attention fusion module is implemented by the following sub-steps:

[0085] Step 4.1: Assume that there are any two modal features U1 and U2. The multi-modal cross-attention module first performs linear projections on U1 and U2 respectively to generate query vector Q1, key vector K1, and value vector V1. The calculation formulas are as follows:

[0086]

[0087] where, and are the learnable weight matrices of Q1, K1, and V1 respectively;

[0088] Step 4.2: The multi-modal cross-attention module uses cross-attention to calculate the preliminary cross-attention fusion feature F1 based on Q1, K1, and V1, that is:

[0089]

[0090] where, is the dimension of the key vector K1, is the scaling factor.

[0091] Step 4.3: Perform secondary linear projection on F1 and U1 respectively to generate query vector Q2, key vector K2, and value vector V2. The calculation formula is as follows:

[0092]

[0093] where, are the learnable weight matrices of Q2, K2, and V2 respectively;

[0094] Step 4.4: Use cross-attention to calculate the secondary cross-attention fusion feature F2 based on Q2, K2, and V2, that is

[0095]

[0096] where, is the dimension of the key vector K2, is the scaling factor.

[0097] Step 4.5: Exchange the fusion order of U1 and U2, and repeat the above steps to generate another set of secondary cross-attention fusion features T2. The specific steps are as follows:

[0098] The multi-modal cross-attention module first performs linear projection on U2 and U1 respectively to generate query vector Q′1, key vector K′1, and value vector V′1. The calculation formula is as follows:

[0099]

[0100] where, and are the learnable weight matrices of Q′1, K′1, and V′1 respectively;

[0101] The multi-modal cross-attention module uses cross-attention to calculate the preliminary cross-attention fusion feature T1 based on Q′1, K′1, and V′1, that is:

[0102]

[0103] where, is the dimension of the key vector K′1, is the scaling factor.

[0104] Perform secondary linear projection on T1 and U2 respectively to generate query vector Q′2, key vector K′2, and value vector V′2. The calculation formula is as follows:

[0105]

[0106] where, and Learnable weight matrices Q′2, K′2, and V′2 respectively;

[0107] Use cross-attention to calculate the quadratic cross-attention fusion feature T2 based on Q′2, K′2, and V′2, that is

[0108]

[0109] where is the dimension of the key vector K′2, is the scaling factor.

[0110] The final output cross-attention fusion feature O is:

[0111] O = δF2+(1 - δ)T2

[0112] where δ is a learnable weight parameter, with an initial value set to 0.5 and dynamically optimized through backpropagation.

[0113] Step 4.6: According to the cross-attention fusion module in the above steps 4.1 - 4.5, implement the following three-level fusion: perform global information fusion on B-mode ultrasound and contrast-enhanced ultrasound images with global modalities to obtain the global cross-attention fusion feature O g ; perform local information fusion on color Doppler and elastography with local modalities to obtain the local cross-attention fusion feature O l ; perform a final cross-fusion on O g and O l to obtain the final cross-attention fusion feature O g-l .

[0114] Step 5, input the final cross-attention fusion feature into the constructed joint classifier that combines a 1D convolutional neural network and a multi-layer perceptron, and further output the diagnosis result of tumor benignity and malignancy. The specific implementation steps are as follows:

[0115] Step 5.1: Input the final cross-attention fusion feature O g-l (D is the spatial dimension, C is the number of channels) into the 1D convolutional neural network to generate a dimensionality-reduced feature The specific formula is: where Conv1D is the 1D convolutional operation.

[0116] Step 5.2: Input into the multi-layer perceptron module and perform the following operations in sequence:

[0117] Normalize the input dimensionality-reduced feature to enhance the training stability and obtain the normalized feature O norm : Apply the ReLU activation function to O normThe dimension is reduced from 768 to 512 to obtain the activation feature O hidden1 : O hidden = ReLU(Linear 768→512 (O norm )); Randomly mask some neurons with probability p = 0.1 to prevent overfitting, and obtain the overfitting-prevented feature O drop : O drop = Dropout(O hidden1 ); Apply the ReLU activation function again to compress the dimension of O drop from 512 to 256 to obtain the activation feature O hidden2 : O hidden2 = ReLU(Linear 512→256 (O drop )); Finally, map the dimension of O hidden2 from 256 to 2 dimensions to obtain the linear feature

[0118] Step 5.3: Apply the Softmax activation function to to convert the classification feature into the predicted probability P of the patient sample having a malignant tumor:

[0119] For example, when P is greater than 0.5, it is determined that the patient sample has a malignant tumor, and when P is less than 0.5, it is determined that the patient sample is a benign tumor, and further output the diagnosis result of tumor benignity and malignancy.

[0120] Finally, the prediction result output by the multi-modal ultrasound prediction model with hierarchical attention fusion is the tumor benignity and malignancy result predicted for the patient sample. This method can provide more accurate auxiliary decision-making support for clinical practice, improve the accuracy of tumor benignity and malignancy diagnosis. The multi-modal ultrasound prediction method based on hierarchical attention fusion of the present invention can also be used for the prediction of breast cancer, providing a reference for the formulation of personalized treatment plans. The above is only the preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes, should be covered within the protection scope of the present invention.

Claims

1. A multimodal ultrasound prediction method based on hierarchical attention fusion, characterized in that: The steps include: Step 1, obtaining multimodal ultrasound data of the tumor, wherein the multimodal ultrasound data includes B-ultrasound, color Doppler, elastic imaging and contrast ultrasound images, and performing unified formatting processing to obtain formatted B-ultrasound, color Doppler, elastic imaging and contrast ultrasound images; Step 2, inputting the formatted B-ultrasound, color Doppler, elastic imaging and contrast ultrasound images into the Swin Transformer V2 model, respectively, extracting ultrasound image features, and obtaining features of B-ultrasound, color Doppler, elastic imaging and contrast ultrasound images; Step 3, after splicing the ultrasonic image features of the four modes, a spliced ​​ultrasonic image feature is obtained; a self-attention and convolution dual-branch feature extraction structure is used to respectively extract the long-distance dependency features and local features of the spliced ​​ultrasonic image feature, and a coarse fusion of multi-modal-multi-scale features is performed based on the long-distance dependency features and local features; Step 4: Expand the coarse fusion feature along the channel dimension into four expanded features F corresponding to B-ultrasound, color Doppler, elastic imaging and contrast imaging B ', F D ', F C ' and F E '; The multimodal cross attention fusion module is used to achieve the following hierarchical fusion: B ' and contrast ultrasound imaging features E 'Perform global information fusion to obtain global cross-attention fusion features; for color Doppler feature F D ' and elastic imaging characteristics F C 'Perform local information fusion to obtain local cross-attention fusion features; cross-fuse the global cross-attention fusion features and the local cross-attention fusion features to obtain the final cross-attention fusion features; Step 5: Input the final cross-attention fusion feature into the classifier to obtain the prediction result of the tumor.

2. The multimodal ultrasound prediction method based on hierarchical attention fusion according to claim 1 is characterized in that: The B-ultrasound and contrast ultrasound images only include the tumor area; The unified formatting process refers to standardizing the pixel size of the B-ultrasound, color Doppler, elastic imaging and contrast ultrasound images of the tumor, and enhancing the data by means of color jittering, random cropping, random erasing and mirror transformation.

3. The multimodal ultrasound prediction method based on hierarchical attention fusion according to claim 1 is characterized in that: There are four Swin Transformer V2 models in step 2, and the four Swin Transformer V2 models have the same structure; The formatted B-ultrasound, color Doppler, elastic imaging and contrast ultrasound images are respectively input into the SwinTransformer V2 model to extract ultrasound image features and obtain the B-ultrasound feature F B 、Color Doppler characteristics F D , Characteristics of elastic imaging F C and the characteristics of contrast ultrasound images E .

4. The multimodal ultrasound prediction method based on hierarchical attention fusion according to claim 3 is characterized in that: The step 3 is specifically as follows: Step 3.1, the feature F of the B-ultrasound B 、Color Doppler characteristics F D , Characteristics of elastic imaging F C and the characteristics of contrast ultrasound images E , spliced ​​along the channel dimension to generate the initial fusion feature matrix F I ∈R H×W×4C , where H×W is the feature space dimension and C is the number of single-modal feature channels; The initial fusion feature matrix F is respectively fused through three groups of 1×1 convolution kernels I Perform channel projection to generate three sets of intermediate feature matrices {F1, F2, F3}∈R H×W×C , and splice them along the channel dimension to obtain the spliced ​​ultrasound image feature F N ∈R H×W×3C ; Step 3.2: The stitched ultrasound image feature F N Input into the self-attention feature extraction branch to extract the spliced ​​ultrasound image feature F N The long-distance dependency feature of self-attention feature F Att ; The stitched ultrasound image feature F N Input into the convolution feature extraction branch to extract the spliced ​​ultrasound image features F N Local features of Step 3.3: The self-attention feature F Att With the convolution feature F Con Perform weighted summation to obtain the coarse fusion feature F of multi-modal and multi-scale features AC ,Right now: F AC =αF Att +βF Con ; Among them, α and β are learnable weight parameters, which are dynamically optimized through back propagation and satisfy α+β=1.

5. The multimodal ultrasound prediction method based on hierarchical attention fusion according to claim 4 is characterized in that: The step 3.2 is specifically as follows: The stitched ultrasound image features F N Evenly divide N groups, each group contains three self-attention sub-features in, is the query feature of the i-th group of self-attention; is the key feature of the i-th group of self-attention, is the value feature of the i-th group of self-attention; Self-attention features of group i Calculated as: Where d is the key characteristic The dimension of is the scaling factor, T is the transpose of the matrix, and Softmax is the activation function; concatenate N groups of self-attention features to form a self-attention feature F Att , the self-attention feature F Att Output features of the self-attention feature extraction branch.

6. The multimodal ultrasound prediction method based on hierarchical attention fusion according to claim 1, characterized in that: The steps to achieve fusion using the multimodal cross-attention fusion module are as follows: Step 4.1: Assume that there are two modal features U1 and U2. First, linearly project U1 and U2 respectively to generate query vector Q1, key vector K1, and value vector V1. The calculation formula is: in, and are the learnable weight matrices for Q1, K1, and V1 respectively; Step 4.2: Use cross attention to calculate the preliminary cross attention fusion feature F1 based on the query vector Q1, key vector K1, and value vector V1, that is: in, is the dimension of key vector K1, is the scaling factor; Step 4.3: Perform secondary linear projection on the preliminary cross-attention fusion feature F1 and the modal feature U1 to generate the query vector Q2, key vector K2, and value vector V2. The calculation formula is: in, are the learnable weight matrices of Q2, K2 and V2 respectively; Step 4.4: Use cross attention to calculate the secondary cross attention fusion feature F2 based on the query vector Q2, key vector K2, and value vector V2, that is, in, is the key vector K2 dimension, is the scaling factor; Step 4.5: Exchange the fusion order of modal features U1 and U2, repeat the above steps 4.1-4.4 to generate another set of secondary cross-attention fusion features T2, and finally output the cross-attention fusion feature O as: O=δF2+(1-δ)T2 Where δ is a learnable weight parameter.

7. The multimodal ultrasound prediction method based on hierarchical attention fusion according to claim 6, characterized in that: A multimodal cross-attention fusion module is used to achieve three-level fusion: B ' and contrast ultrasound imaging features E 'Perform global information fusion to obtain the global cross-attention fusion feature O g ; For color Doppler feature F D ' and elastic imaging characteristics F C 'Perform local information fusion to obtain local cross-attention fusion feature O l ; Global cross attention fusion feature O g And the local cross attention fusion feature O l Perform the final cross fusion to obtain the final cross attention fusion feature O g-l .

8. The multimodal ultrasound prediction method based on hierarchical attention fusion according to claim 1, characterized in that: The classifier described in step 5 is a joint classifier including a 1-dimensional convolutional neural network and a multi-layer perceptron.

9. The multimodal ultrasound prediction method based on hierarchical attention fusion according to claim 1, characterized in that: The step 5 is specifically as follows: Step 5.1: Fusion the final cross-attention feature O g-l Input 1D convolutional neural network to generate dimensionality reduction features The specific formula is: Among them, Conv1D is a 1-dimensional convolution operation; Step 5.2: Reduce the features Enter the multi-layer perceptron module and perform the following operations in sequence: After dimension reduction, the features Perform normalization to obtain the normalized feature O norm : Apply the ReLU activation function to normalize the feature O norm The dimension is reduced from 768 to 512, and the activation feature O is obtained. hidden1 : O hidden =ReLU(Linear 768→512 (O norm )); Randomly shield some neurons with probability p = 0.1 to prevent overfitting and obtain the anti-overfitting feature O drop :O drop = Dropout(O hidden1 ); Applying the ReLU activation function again will prevent overfitting feature O drop The dimension is compressed from 512 to 256, and the activation feature O is obtained. hidden2 : O hidden2 =ReLU(Linear 512→256 (O drop )); Finally, feature O will be activated hidden2 The dimension is mapped from 256 to 2 dimensions to obtain linear features Step 5.3: Linear Features Apply the Softmax activation function to transform the linear features Converted into predicted probability P:

Images