Breast cancer NAC curative effect prediction method based on multi-modal multi-branch neural network

Abstract

The invention relates to a breast cancer NAC curative effect prediction method based on a multi-modal multi-branch neural network. The breast cancer NAC curative effect prediction method comprises the following steps of S1, obtaining multi-modal ultrasonic images of two NAC stages of a breast cancer and performing preprocessing; and S2, on the basis of the preprocessed multi-modal ultrasonic images in each NAC stage, a pre-trained multi-modal multi-branch neural network designed by the invention is utilized to obtain a prediction result of the breast cancer NAC curative effect on the premise of considering the correlation among the images in different NAC stages and the correlation among the ultrasonic images in different modals. Compared with the prior art, the method has the advantages that difference information among different stages and deeper features of a single stage can be effectively captured, strong correlation and complementarity information among multiple modes are fully considered, contribution of feature representation to classification tasks is better known, correlation among the modes is extracted, and accordingly prediction accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for predicting the NAC efficacy of breast cancer based on a multi-modal multi-branch neural network. Background Art

[0002] Neoadjuvant chemotherapy (NAC) is a component of systemic treatment for breast cancer. It can reduce the stage of breast cancer, reduce recurrence and metastasis, and improve the breast conservation rate. Whether pathologic complete response (pCR) can be achieved after NAC is closely related to the patient's prognosis. However, breast cancer is a highly heterogeneous tumor. Although chemotherapy regimens are constantly being optimized and improved, the current pCR rate is still low. How to use non-invasive imaging methods to objectively and effectively predict the efficacy of NAC for breast cancer at an early stage is a current research hotspot and a problem that needs to be solved urgently in clinical practice.

[0003] Ultrasound is a commonly used imaging method during NAC for breast cancer. It can reflect the morphology of the tumor and monitor tumor changes during NAC. However, the evaluation of ultrasound examination is mainly based on the subjective visual evaluation of clinicians. There is still a lack of an objective standard evaluation system, and there are large differences within and between observers. Traditional machine learning methods based on imaging features belong to feature engineering, which is time-consuming and labor-intensive and has low accuracy. In recent years, with the continuous development of deep learning technology (DL), DL-based ultrasound images have received widespread attention due to their outstanding performance in breast tumor identification, differential diagnosis, efficacy evaluation, and prognosis prediction.

[0004] At present, most DL systems for early prediction of NAC efficacy of breast cancer based on ultrasound images have the following shortcomings: First, most existing systems only analyze the gray scale ultrasound image (GUSI) of the lesion. Second, most existing systems only analyze the GUSI of a single section of the lesion, lacking a comprehensive evaluation of the entire lesion. Finally, the existing systems do not compare and analyze the characteristic changes of tumors before and after chemotherapy.

[0005] In summary, there is currently a lack of a method for predicting the efficacy of NAC for breast cancer to solve or partially solve the above-mentioned problems. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for predicting the NAC efficacy of breast cancer based on a multimodal multi-branch neural network, so as to solve or partially solve the problem that the current NAC efficacy prediction scheme does not fully consider the correlation between different NAC stages and between different modality features.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] One aspect of the present invention provides a method for predicting the efficacy of NAC for breast cancer based on a multimodal multi-branch neural network, comprising the following steps:

[0009] Step S1, acquiring multimodal ultrasound images of two NAC stages of breast cancer and performing preprocessing;

[0010] Step S2, based on the preprocessed multimodal ultrasound images of each NAC stage, using the pretrained multimodal multi-branch neural network, and considering the correlation between images of different NAC stages and the correlation between ultrasound images of different modalities, obtain the prediction result of the NAC efficacy of breast cancer.

[0011] As a preferred technical solution, in step S2, the process of using a pre-trained multi-modal multi-branch neural network to obtain the prediction result of the NAC efficacy of breast cancer includes:

[0012] Step S201, using a ResNeXt network to perform feature extraction on various types of multimodal ultrasound images at various NAC stages after preprocessing, to obtain feature vectors corresponding to various types of multimodal ultrasound images at various NAC stages;

[0013] Step S202, for the feature vectors corresponding to various multimodal ultrasound images at various NAC stages, extract the difference features between the two NAC stages corresponding to various modalities and the deep features of each NAC stage through contrast difference learning, and obtain the difference contrast feature vectors between the two stages corresponding to various modalities;

[0014] Step S203, fusing the difference contrast feature vectors corresponding to the ultrasound images of various modalities, respectively obtaining corresponding feature vectors using channel attention, spatial attention and self-attention, and obtaining a fused feature vector by splicing;

[0015] Step S204: obtaining a binary classification prediction result of the NAC efficacy of breast cancer based on the fused feature vector.

[0016] As a preferred technical solution, the difference comparison feature vector is obtained using the following formula:

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] in, and They represent the feature maps of the two NAC stages NAC0 and NAC2 after preprocessing, and the feature maps have C channels and a size of m×n. It represents the cosine similarity of the feature vectors of the pixel (i, j) and C channel between the two NAC stages, H standard is the normalization function, is the feature matrix, is the weight matrix, F global is the difference comparison feature vector, Conv(*) represents the convolution operation, Cat(*) represents the concatenation operation in the channel direction, Represents element-wise multiplication.

[0023] As a preferred technical solution, the fused feature vector is obtained using the following formula:

[0024]

[0025] W channel =σ(FC(GlobalAveragePooling(F fusion )))

[0026] W spatial =σ(Conv 3×3 (Cat(AveragePooling(F fusion ), MaxPooling(F fusion )))

[0027]

[0028] F final =Cat(W channel ·F fusion , W spatial ·F fusion , W self_att ·F fusion ·V)

[0029] in, are the difference contrast feature vectors corresponding to grayscale ultrasound modality images, color Doppler ultrasound modality images, and elastic ultrasound modality images, GlobalAveragePooling is the global average pooling operation, FC is the fully connected layer, σ is the activation function, AveragePooling and MaxPooling are the average pooling and maximum pooling operations, respectively, Conv 3×3is a 3x3 convolutional layer, Q, K, and V are query, key, and value feature matrices obtained through linear transformation, d k is the scaling factor, and Cat(*) indicates the concatenation operation in the channel direction.

[0030] As a preferred technical solution, the multimodal ultrasound image includes a grayscale ultrasound image, a color Doppler ultrasound image and an elastic ultrasound image.

[0031] As a preferred technical solution, the training process of the multimodal multi-branch neural network includes:

[0032] Step S001, obtaining a labeled and preprocessed data set, and dividing it into multiple subsets, and selecting a validation set and a training set from the subsets;

[0033] Step S002, using the training set to perform optimal performance training on the single-path feature extraction network, and using the obtained weights as the initial weights of the multi-branch neural network;

[0034] Step S003, inputting the multi-modal and multi-stage ultrasound images of each training set into the multi-modal and multi-branch neural network, and performing training by calculating the loss function;

[0035] Step S004, using the validation set to evaluate the performance of the network obtained from each training, recording the network accuracy, sensitivity and specificity indicators of each training, and selecting the network with the best performance as the final network.

[0036] As a preferred technical solution, the loss function is a weighted focal loss function.

[0037] Another aspect of the present invention provides a breast cancer NAC efficacy prediction system based on a multimodal multi-branch neural network, comprising:

[0038] An image preprocessing module, used for preprocessing the acquired multimodal ultrasound images of two NAC stages of breast cancer;

[0039] A multimodal multi-branch neural network is used to obtain the prediction results of breast cancer NAC efficacy based on the preprocessed multimodal ultrasound images of each NAC stage, taking into account the differences and similarities between images of different NAC stages.

[0040] As a preferred technical solution, the multimodal multi-branch neural network includes:

[0041] A feature extraction module is used to extract features from the preprocessed multimodal ultrasound images of each NAC stage to obtain feature vectors of each NAC stage;

[0042] The two-stage contrastive difference learning module is used to extract the difference features between different NAC stages and the deep features of each NAC stage through contrastive difference learning for the feature vectors corresponding to each NAC stage, and obtain the single-stage feature vector;

[0043] The multi-attention multi-modal fusion module is used to fuse the single-stage feature vectors corresponding to each modality of ultrasound image, and obtain the corresponding feature vectors using channel attention, spatial attention and self-attention respectively, and obtain the fused feature vector by splicing;

[0044] The decision module is used to obtain a binary classification prediction result of the NAC efficacy of breast cancer based on the fused feature vector.

[0045] As a preferred technical solution, during the training process of the multimodal multi-branch neural network, training is achieved by calculating weighted focal loss.

[0046] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0047] (1) Fully consider the correlation between different ultrasound modalities and different NAC stages: In view of the problem that the current efficacy prediction model only extracts information from a single dimension and has limited predictive value, the present invention is based on the preprocessed multimodal ultrasound images of each NAC stage, using a pre-trained multimodal multi-branch neural network, and obtains the prediction results of the NAC efficacy of breast cancer under the premise of considering the correlation between images of different NAC stages and the correlation between ultrasound images of different modalities. Through the above design, the difference information between different stages and the deeper features of a single stage can be effectively captured, and the strong correlation and complementary information between multimodalities can be fully considered, so as to better understand the contribution of their feature representation to the classification task, extract the correlation between modalities, and thus improve the prediction accuracy.

[0048] (2) Effectively extract features across ultrasound modalities and NAC stages: The present invention provides a new multi-modal multi-branch network that can fully consider the correlation between different modalities and different NAC stages. First, a two-stage contrast difference learning module is designed to effectively extract single-modality difference contrast feature vectors. Secondly, a fusion module based on a multi-attention mechanism is designed to discover the correlation between multiple modalities. Finally, after the fusion analysis of the multi-attention mechanism, a weighted focal loss function is used to help alleviate the result bias caused by sample imbalance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a multi-modal multi-branch neural network construction process in an embodiment;

[0050] Figure 2 It is a pre-processing flow chart in the embodiment;

[0051] Figure 3 This is a schematic diagram of median filtering in an embodiment;

[0052] Figure 4 It is a TMCN model framework diagram in the embodiment;

[0053] Figure 5 This is the ROC curve of the retrospective independent test set in the embodiment;

[0054] Figure 6 Schematic diagram of ROC curve of prospective test set in the embodiment. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0056] Example 1

[0057] In view of the lack of objective and effective early prediction methods for the efficacy of NAC for breast cancer, and the problem that the existing efficacy prediction models only extract information from a single dimension and have limited prediction value, this embodiment provides a method for predicting the efficacy of NAC for breast cancer based on a multimodal multi-branch neural network. A clinically applicable two-stage multimodal comparison network (TMCN) is used, combined with multimodal (GUSI, CDFI and EI) multi-section ultrasound images of the dual chemotherapy stages before chemotherapy (NAC0) and after two courses of chemotherapy (NAC2), to comprehensively evaluate the efficacy of NAC for breast cancer from different angles for reference by doctors and to improve the pCR rate.

[0058] Color Doppler flow imaging (CDFI) and elasticity imaging (EI) are two ultrasonic imaging methods. CDFI can reflect the blood perfusion degree and distribution characteristics of the lesion, and EI can reflect the hardness of the lesion. Both play an important role in the early prediction of the efficacy of NAC for breast cancer. For tumors that respond well to chemotherapy, there will be a decrease in the blood perfusion degree and the hardness of tumor tissue in the early stage. Taking into account the above characteristics, this embodiment combines GUSI, CDFI and EI to conduct a multimodal ultrasonic comprehensive evaluation of the efficacy of NAC for breast cancer, which is more in line with the clinical evaluation process and can provide more comprehensive information about the response of the lesion to chemotherapy. Combined with the comprehensive analysis of multi-section ultrasound images of different parts of the lesion, the chemotherapy sensitivity of the lesion can be better demonstrated.

[0059] It should be noted that this embodiment uses two NAC stages, NAC0 and NAC2, for illustration. Other NAC stages may be used without conflict. The main purpose of this method is to predict the efficacy of NAC for breast cancer and does not involve the diagnosis or treatment of the disease.

[0060] For details, see Figure 1 , the method comprises the following steps:

[0061] Step S1, retrospectively collect multi-modal (GUSI, CDFI and EI) multi-section high-quality ultrasound images of NAC0 and NAC2 of breast cancer patients who underwent NAC.

[0062] Specifically, this step includes the following sub-steps:

[0063] Step S101, retrospectively collect multimodal ultrasound images of NAC0 and NAC2 stages of 197 breast cancer patients who were diagnosed with breast cancer and underwent NAC, and all patients completed a full cycle of NAC.

[0064] Step S102 requires that each modality has at least three or more complete and high-quality images containing both transverse and longitudinal sections. All cases were randomly divided into a training set (n=132; 2376 images) and a retrospective test set (n=65; 1170 images).

[0065] In this example, conventional ultrasound and SE imaging examinations were performed using MyLab Twice (Esaote, Genoa, Italy) color Doppler ultrasound diagnostic instrument and 4-13-MHz LA523 linear array variable frequency probe. For retrospectively collected ultrasound images, each modality was required to have at least three or more complete and good-quality images containing at least two transverse and longitudinal sections. All cases were randomly assigned to a training set (n = 132; 2376 images) and a retrospective test set (n = 65; 1170 images).

[0066] In step S103, a doctor with 15 years of experience in breast ultrasound examination is responsible for controlling the quality of ultrasound images, removing some images containing artifacts, blur and non-lesional tissues, and selecting high-quality ultrasound images.

[0067] Step S2, ultrasound image preprocessing, the process includes: manually marking the region of interest (ROI), followed by image preprocessing, including: random cropping, adjusting image size, median filtering noise reduction and data enhancement.

[0068] The original ultrasound image has heavy speckle noise and may involve privacy information and some useless data, such as instrument model, examination time, etc. Therefore, the original data needs to be preprocessed first. This process includes manual annotation of ROI, followed by traditional processing (cropping, adjusting image size, grayscale and normalization), median filtering denoising and data enhancement.

[0069] For details, see Figure 2 , this step includes the following sub-steps:

[0070] Step S201, region of interest (ROI) annotation: a ultrasound physician with 15 years of experience in breast ultrasound examination manually annotates the ROI to eliminate irrelevant information such as text and icons to improve the learning efficiency of TMCN.

[0071] Step S202, image cropping and resizing: crop the image to remove irrelevant parts, and adjust the image size to 256x256 to meet the input requirements of the subsequent model.

[0072] In this embodiment, zero padding is performed on the periphery of the lesion area without changing the aspect ratio of the image to generate a square image. To facilitate model training, all images are normalized to 256×256 pixels before being input into the neural network. This process promotes the standardization of the model input image size and helps model training.

[0073] Step S203, median filtering denoising: applying median filtering to perform denoising processing on the image to remove salt and pepper noise and other interference in the image.

[0074] See also Figure 3 In this embodiment, the median filter method is used to reduce the noise of the image. The median filter is a nonlinear filter that can preserve the edge characteristics of the image while overcoming the noise. The implementation process is as follows: first, a filter Fjk with an odd window size is defined, and then the image pixels in the filter are arranged in order from small to large to obtain a filter queue {Fjk}. The median value of all pixel values ​​in {Fjk} is used to replace the pixel value at the center position G(i,y) of the filter.

[0075] Step S204, data enhancement: performing data enhancement on the image, including operations such as rotation, translation, and flipping, to expand the data set and increase the robustness of the model.

[0076] This embodiment uses preset data transformation rules to perform data augmentation based on existing data. It includes geometric and color transformation categories. The geometric transformation category includes: flipping, rotation, cropping, deformation and scaling. Through data augmentation, the number of images in the training set is increased to 10 times the original data amount.

[0077] Step S3, based on the processed image data set, a multi-branch neural network algorithm is used to construct a deep learning model (TMCN) based on dual-stage multimodal ultrasound images to achieve early intelligent prediction of the efficacy of NAC for breast cancer.

[0078] For details, see Figure 4 , TMCN consists of three modules: two-stage contrastive difference learning module (TCD), multi-attention multimodal fusion module (MAF) and decision module. TCD is designed between two channels of the same modality to effectively capture the difference information between different stages and the deeper features of a single stage; MAF is designed after the two-stage contrastive difference learning module to discover the correlation between modalities, allowing the model to selectively emphasize the feature representation of important modalities; after fusion analysis of the multi-attention mechanism, two fully connected layers are used to reduce the dimension of the features and obtain the output results. Each module is explained below. Figure 4 In the figure, GUSI represents grayscale ultrasound image; CDFI represents color Doppler blood flow image; EI represents elastic ultrasound image.

[0079] Contrastive Difference Learning Module:

[0080] The TCD module consists of two submodules: the difference learning submodule (DLS) and the multi-stage linear correlation fusion submodule (LCF).

[0081] DLS submodule: used to evaluate the difference in predicting chemotherapy sensitivity between the two stages. The module input is the feature map of the two stages obtained by the last convolution layer of the pre-trained backbone network ResNeXt, that is, and Both feature maps have C channels and size m×n. RexNetXt has a multi-layer deep architecture that enables it to learn complex patterns and representations from input data. Independent backbone networks can not only learn feature representations of specific modalities, but also prevent the model from mis-tuning parameters when training between different modalities. The C-channel feature vectors of the corresponding pixel (i, j) of the two-stage feature maps are compared for similarity, and the cosine similarity feature map is calculated. The formula is:

[0082]

[0083]

[0084] Among them, H standard It is a normalization function that maps the similarity score of each pixel to the range of 0-1.

[0085] LCF module: The cosine function is used to explore the difference response map of feature representation between the two stages. In addition to the difference features, single-stage specific features can also provide useful information for predicting the effect of chemotherapy, especially when the image quality of a certain stage is low. Considering the above characteristics, this module uses a bilinear fusion method to retain the independent features of the two-stage images and the difference features between them.

[0086] The DLS submodule can effectively capture information about the differences between different stages. In addition to the need to fuse features from different stages, deeper features of a single stage can also provide useful information for prediction, especially when we need to distinguish whether the difference in tumors after chemotherapy is due to a good response to chemotherapy or drug resistance. Therefore, highly inconsistent single-stage features are also retained in LCF.

[0087] Specifically, the similarity matrix obtained by the DLS submodule is first ROI feature matrix of NAC0 stage And the ROI feature matrix of the NAC2 stage Splicing is performed, and the number of channels of the splicing matrix is ​​changed back to length C through the convolution layer to obtain the weight matrix See formula (2); then, multiply the weight matrix and the single-stage feature matrix. Finally, the single-stage feature vector F is obtained. global , see formula (3):

[0088]

[0089]

[0090]

[0091] Where Conv(*) represents the convolution operation, Cat(*) represents the concatenation operation in the channel direction, Represents element-wise multiplication.

[0092] MAF Module:

[0093] In order to consider the strong correlation and complementary information between multiple modalities and better understand the contribution of their feature representation to the classification task, this embodiment provides a fusion module MAF based on a multi-attention mechanism to discover the correlation between modalities. This module allows the model to selectively emphasize the feature representation of important modalities. The MAF module includes a channel attention module, a spatial attention module, and a self-attention module. The feature vectors of the three attention mechanisms are concatenated to obtain the fusion feature F. fusion , the formula is:

[0094]

[0095] Channel attention module: Representing F with features of three modalities fusion As input, the weight of each channel is calculated through a 1x1 convolutional layer to obtain the modality weight, thereby identifying the modality that contributes most to the classification task. The specific implementation is: W channel =σ(FC(GlobalAveragePooling(F fusion ))), where GlobalAveragePooling is the global average pooling operation, FC is the fully connected layer, and σ is the activation function.

[0096] Spatial attention module: focuses on the spatial location of the tumor, uses a 3x3 convolutional layer to obtain spatial weights, and reduces the interference of non-tumor tissue information in ultrasound images. The specific implementation is: W spatial =σ(Conv 3×3 (Cat(AveragePooling(F fusion ), MaxPooling(F fusion ))). Among them, AveragePooling and MaxPooling are average pooling and maximum pooling operations respectively, Conv 3×3 It is a 3x3 convolutional layer.

[0097] Self-attention module: Determine the importance of different modalities at each spatial position through the self-attention mechanism, capture the correlation between different positions in the series feature map, obtain self-attention weights, dynamically assign weights, and enhance feature representation, so as to better utilize the correlation and differences between different modalities. The specific implementation is: Among them, Q, K and V are the query, key and value feature matrices obtained by linear transformation, d k is the scaling factor.

[0098] MAF concatenates the feature vectors of the three attention mechanisms: channel attention, spatial attention, and self-attention modules to obtain the fusion feature F final , used for subsequent classification, the formula is:

[0099] Ffinal =Cat(W cha ·F fusion , W spatial ·F fusion , W self_att ·F fusion ·V) (4)

[0100] The most relevant features are discovered through the fusion module. MAF encourages the model to learn more meaningful representations in terms of spatial, channel, and multimodal features, which is better than using only one attention mechanism.

[0101] Decision module:

[0102] After fusion analysis of the multi-attention mechanism, F is final Dimensionality reduction is performed to reduce the number of features to 2048, and finally the output probability of binary classification is obtained. The greater the weight of the modality, the greater its contribution to the classification.

[0103] In this embodiment, the entire network is trained in an end-to-end manner. Considering that the imbalance in the number of categories will affect the classification results, the number of nonpCR patients is more than twice that of pCR patients. To solve this problem, weighted focal loss is used as the pCR prediction loss function to help alleviate the biased results caused by sample imbalance during training. Loss function formula:

[0104]

[0105] Here, p is the predicted probability. When the class is 1, we multiply it by the weight α=0.25, otherwise, we multiply it by the weight 1-α. In addition, we take γ=2 to reduce the loss of samples that are easily correctly classified and pay more attention to those samples that are more likely to be misclassified.

[0106] Step S4, using five-fold cross validation, the multi-modal and multi-stage ultrasound images of the training set are input into TMCN for training and the optimal classification model is selected using its validation set.

[0107] Specifically, this step includes the following sub-steps:

[0108] Step S401, data set division: the preprocessed data set is divided into five non-overlapping subsets. Each time one of the subsets is selected as the validation set, and the remaining four subsets are selected as the training set, and five training and verification are performed.

[0109] In this embodiment, a random sampling method was used to extract 60%, 10% and 30% of the samples from the pCR and nonpCR categories, respectively, to construct a training set, a validation set and a retrospective test set. The training set was divided into five non-overlapping subsets. Each time, one of the subsets was selected as the validation set, and the remaining four subsets were selected as the training set, and five training and validation were performed.

[0110] Step S402, pre-training weights: Considering the size of the sample and the multi-path network, the present invention uses the pre-trained weights from the ImageNet dataset on the single-path ResNeXt network, uses the training set to train the single-path network for optimal performance, and uses the obtained weights as the initial weights of the multi-path network.

[0111] Step S403, model training: input the multimodal and multi-stage ultrasound images of each training set into the TMCN model for training. During the training process, batch normalization and dropout are used to solve the overfitting problem, batch gradient descent method is used to train the model, and dynamic learning rate is used to ensure the robustness of the training model.

[0112] Hyperparameters have a great impact on the performance of the model. Appropriate hyperparameters can make the model converge quickly and achieve excellent performance. This example uses the adaptive learning Adam as the optimizer, the initial learning rate is set to 0.001, the weight decay factor is 0.1, the number of iterations is set to 500, and the learning rate is reduced to 1 / 10 of the previous one every 150 iterations. The loss function will use the cross entropy loss function used in classification tasks. The batch size is set to 32.

[0113] Step S404, model validation: Use the validation set to evaluate the model performance obtained from each training, record the model accuracy, sensitivity, specificity and other indicators of each training, and select the classification model with the best performance as the final model.

[0114] Step S5, prospectively collect and preprocess the multimodal multi-section ultrasound images of NAC0 and NAC2 of breast cancer patients who will undergo NAC.

[0115] Specifically, this step includes the following sub-steps:

[0116] Step S501: In order to further evaluate the applicability of the model in clinical practice, cases diagnosed with breast cancer and planned to undergo NAC were prospectively recruited as an internal prospective test set.

[0117] Step S502, for prospectively retained ultrasound images, the following standards are followed: at least three or more multimodal ultrasound images containing transverse and longitudinal sections are retained for each patient, and the GUSI, CDFI, and EI images of the same part of each lesion are ensured to be in the same section as much as possible. Conventional ultrasound and SE imaging examinations use the MyLab Twice (Esaote, Genoa, Italy) color Doppler ultrasound diagnostic instrument with a 4-13-MHz LA523 linear array variable frequency probe. The imaging planes of GUSI, CDFI, and EI are kept as consistent as possible. A total of 63 women (1134 images) were included in the prospective test set.

[0118] In step S503, a doctor with 15 years of experience in breast ultrasound examination is responsible for controlling the quality of ultrasound images, removing some images containing artifacts, blur and non-lesional tissues, and selecting high-quality ultrasound images.

[0119] In step S6, the multimodal and multi-stage ultrasound images belonging to the retrospective independent test set and the prospective test set are respectively input into the TMCN for testing to obtain the predictive efficacy, and the efficacy is compared with the clinical model and the visual evaluation of the clinician.

[0120] Specifically, this step includes the following sub-steps:

[0121] Step S601: In the retrospective and prospective test sets, all ultrasound images were analyzed by two ultrasound physicians without knowing the patient's ultrasound report, radiomics evaluation results, and pathological results. Based on their personal clinical experience and previous research results, they made subjective visual predictions on the possible treatment outcomes after NAC for each case by analyzing the dual-stage multimodal ultrasound images of each case, and divided all lesions into two categories: pCR and nonpCR.

[0122] Step S602: Input the multi-modal and multi-stage ultrasound images of the retrospective independent test set and the prospective test set into the TMCN model for prediction. The TMCN model processes the image data of different modalities and different stages through six branch paths to generate the prediction results of each sample. The prediction results and corresponding probability values ​​of each sample are recorded.

[0123] Step S603, performance evaluation: Evaluate the predictive performance of TMCN and compare it with the clinical model and the subjective visual assessment of clinicians. DeLong test was used to compare AUC. The accuracy, sensitivity, specificity, PPV, NPV and area under the ROC curve (AUC) of different models for pCR prediction were calculated.

[0124] As shown in Table 1, the performance of the clinical model in predicting pCR was unsatisfactory, with AUCs of 0.79 (95% CI: 0.71, 0.86), 0.69 (95% CI: 0.56, 0.80), and 0.73 (95% CI: 0.59, 0.83) in the training set and the retrospective and prospective internal test sets, respectively. Figure 5 and 6 As shown, compared with the clinical model, the AUC of the TMCN provided in this embodiment in the retrospective and prospective test sets were 0.93 (95% CI: 0.88, 0.96) and 0.88 (95% CI: 0.83, 0.93), respectively, and its predictive performance was significantly higher than that of the clinical model (p<0.001 and p=0.01). In the retrospective test set, the accuracy, sensitivity, specificity, PPV and NPV of TMCN prediction were 90.7%, 82.6%, 95.2%, 86.4% and 93.0%, respectively. TMCN performed well in the prospective test set, with the prediction accuracy, sensitivity, specificity, PPV and NPV of 87.3%, 83.3%, 89.7, 83.3% and 89.7%, respectively. Figure 5 and Figure 6 As shown in Table 1, based on the multimodal ultrasound image features, the AUCs of clinicians 1 and 2 for early prediction of pCR in the test set were 0.64 (95% CI: 0.51, 0.75) and 0.68 (95% CI: 0.54, 0.79), respectively, and their diagnostic performance was significantly lower than that of the TMCN model (p<0.001). There was no significant statistical difference in predictive efficacy between the two.

[0125] Table 1. The performance of TMCN, clinical model, and clinician in predicting pCR in retrospective and prospective test sets

[0126]

[0127] In Table 1, the prospective and retrospective test sets using the DeLong test were compared with the TMCN model. * P<0.001, RTS=retrospective test set; PTS=prospective test set.

[0128] Figure 5 and Figure 6 In the figure, Clinical method refers to the clinical model; Radiologist 1 refers to doctor 1; Radiologist 2 refers to doctor 2.

[0129] Example 2

[0130] Based on Example 1, this example provides a breast cancer NAC efficacy prediction system based on a multi-modal multi-branch neural network, including:

[0131] An image preprocessing module, used for preprocessing the acquired multimodal ultrasound images of two NAC stages of breast cancer;

[0132] The multimodal multi-branch neural network is used to obtain the prediction result of the NAC efficacy of breast cancer based on the pre-processed multimodal ultrasound images of each NAC stage, taking into account the similarity between the images of different NAC stages. During the training process of the multimodal multi-branch neural network, the training is achieved by calculating the weighted focal loss.

[0133] Among them, the multimodal multi-branch neural network includes:

[0134] The feature extraction module is used to extract features from the preprocessed multimodal ultrasound images of each NAC stage to obtain feature vectors of each NAC stage.

[0135] The two-stage contrastive difference learning module TCD is used to extract the difference features between different NAC stages and the deep features of each NAC stage through contrastive difference learning for the feature vectors corresponding to each NAC stage, and obtain the single-stage feature vector. TCD includes a differential learning submodule DLS and a multi-stage linear correlation fusion submodule LCF. TCD is designed between two channels of the same modality to effectively capture the difference information between different stages and the deeper features of a single stage; MAF is designed after the two-stage contrastive difference learning module to discover the correlation between modalities, allowing the model to selectively emphasize the feature representation of important modalities.

[0136] The multi-attention multimodal fusion module MAF is used to fuse the single-stage feature vectors corresponding to each modality of ultrasound image, and obtain the corresponding feature vectors by using channel attention, spatial attention and self-attention respectively, and obtain the fused feature vector by splicing. The MAF module includes channel attention module, spatial attention module and self-attention module

[0137] The decision module is used to obtain a binary classification prediction result of the NAC efficacy of breast cancer based on the fused feature vector.

[0138] Compared with the prior art, the present invention has the following beneficial effects:

[0139] The existing DL models for early prediction of the efficacy of NAC for breast cancer are mostly based on single-stage single-modal single ultrasound images, which can only extract information from a single dimension and have limited predictive value. The present invention combines multi-modal multi-section ultrasound images of NAC0 and NAC2 to provide a multi-branch neural network method (TMCN) for comprehensive evaluation of the efficacy of NAC for breast cancer from different angles. First, the present invention designs a two-stage contrast difference learning module between two channels of the same modality to effectively capture the difference information between different stages and the deeper features of a single stage; secondly, in order to consider the strong correlation and complementary information between multi-modalities and better understand the contribution of their feature representation to the classification task, after the two-stage contrast difference learning module, the present invention constructs a fusion module based on a multi-attention mechanism to discover the correlation between modalities. Finally, after the fusion analysis of the multi-attention mechanism, two fully connected layers are used to reduce the dimension of the features, and the weighted focal loss function is used to help reduce the result deviation caused by sample imbalance. Through experimental verification, the present invention has shown significant advantages in the task of early prediction of NAC efficacy. Compared with the single-branch network method, this method has achieved significant improvements in prediction accuracy and can provide a non-invasive tool for individualized pCR prediction, assisting clinicians to adjust treatment plans in a timely manner and improve the pCR rate.

[0140] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for predicting the efficacy of NAC for breast cancer based on a multimodal multi-branch neural network, characterized in that: The steps include: Step S1, acquiring multimodal ultrasound images of two NAC stages of breast cancer and performing preprocessing; Step S2, based on the preprocessed multimodal ultrasound images of each NAC stage, using the pretrained multimodal multi-branch neural network, and considering the correlation between images of different NAC stages and the correlation between ultrasound images of different modalities, obtain the prediction result of the NAC efficacy of breast cancer.

2. The method for predicting the efficacy of NAC for breast cancer based on a multimodal multi-branch neural network according to claim 1, characterized in that: In step S2, the process of using the pre-trained multi-modal multi-branch neural network to obtain the prediction result of the NAC efficacy of breast cancer includes: Step S201, using a ResNeXt network to perform feature extraction on various types of multimodal ultrasound images at various NAC stages after preprocessing, to obtain feature vectors corresponding to various types of multimodal ultrasound images at various NAC stages; Step S202, for the feature vectors corresponding to various types of multimodal ultrasound images at various NAC stages, extract the difference features between the two NAC stages corresponding to various types of modalities and the deep features of each NAC stage through contrast difference learning, and obtain the difference contrast feature vectors of the two stages corresponding to various types of modalities; Step S203, the difference contrast feature vectors corresponding to the ultrasound images of various modalities are fused, and the corresponding feature vectors are obtained by using the channel attention, spatial attention and self-attention mechanisms respectively, and the fused feature vector is obtained by splicing; Step S204: obtaining a binary classification prediction result of the NAC efficacy of breast cancer based on the fused feature vector.

3. The method for predicting the efficacy of NAC for breast cancer based on a multimodal multi-branch neural network according to claim 2, characterized in that: The difference comparison feature vector is obtained by the following formula: in, and They represent the feature maps of the two NAC stages NAC0 and NAC2 after preprocessing, and the feature maps have C channels and a size of m×n. It represents the cosine similarity of the feature vectors of the pixel (i, j) and C channel between the two NAC stages, H standard is the normalization function, is the feature matrix, is the weight matrix, F global is the difference comparison feature vector, Conv(*) represents the convolution operation, Cat(*) represents the concatenation operation in the channel direction, Represents element-wise multiplication.

4. The method for predicting the efficacy of NAC for breast cancer based on a multimodal multi-branch neural network according to claim 2, characterized in that: The fused feature vector is obtained using the following formula: W chann =σ(FC(GlobalAveragePooling(F fusion ))) W spatial =σ(Conv 3×3 (Cat(AveragePooling(F fusion ),MaxPooling(F fusion ))) F final =Cat(W channel ·F fusion ,W spatial ·F fusion ,W self_att ·F fusion ·V) in, are the difference contrast feature vectors corresponding to grayscale ultrasound modality images, color Doppler ultrasound modality images, and elastic ultrasound modality images, GlobalAveragePooling is the global average pooling operation, FC is the fully connected layer, σ is the activation function, AveragePooling and MaxPooling are the average pooling and maximum pooling operations, respectively, Conv 3×3 is a 3x3 convolutional layer, Q, K, and V are query, key, and value feature matrices obtained by linear transformation, d k is the scaling factor, and Cat(*) indicates the concatenation operation in the channel direction.

5. The method for predicting the efficacy of NAC for breast cancer based on a multimodal multi-branch neural network according to claim 2, characterized in that: The multimodal ultrasound image includes a grayscale ultrasound image, a color Doppler ultrasound image and an elastic ultrasound image.

6. The method for predicting the efficacy of NAC for breast cancer based on a multimodal multi-branch neural network according to claim 1, characterized in that: The training process of the multimodal multi-branch neural network includes: Step S001, obtaining a labeled and preprocessed data set, and dividing it into multiple subsets, and selecting a validation set and a training set from the subsets; Step S002, using the training set to perform optimal performance training on the single-path feature extraction network, and using the obtained weights as the initial weights of the multi-branch neural network; Step S003, inputting the multi-modal and multi-stage ultrasound images of each training set into the multi-modal and multi-branch neural network, and performing training by calculating the loss function; Step S004, using the validation set to evaluate the performance of the network obtained from each training, recording the network accuracy, sensitivity and specificity indicators of each training, and selecting the network with the best performance as the final network.

7. The method for predicting the efficacy of NAC for breast cancer based on a multi-modal multi-branch neural network according to claim 6, characterized in that: The loss function is a weighted focal loss function.

8. A breast cancer NAC efficacy prediction system based on a multi-modal multi-branch neural network, characterized in that: include: An image preprocessing module, used for preprocessing the acquired multimodal ultrasound images of two NAC stages of breast cancer; A multimodal multi-branch neural network is used to obtain the prediction results of the NAC efficacy of breast cancer based on the preprocessed multimodal ultrasound images of each NAC stage, taking into account the differences and similarities between images of different NAC stages.

9. The breast cancer NAC efficacy prediction system based on a multi-modal multi-branch neural network according to claim 8, characterized in that: The multimodal multi-branch neural network comprises: A feature extraction module is used to extract features from the preprocessed multimodal ultrasound images of each NAC stage to obtain feature vectors of each NAC stage; The two-stage contrastive difference learning module is used to extract the difference features between different NAC stages and the deep features of each NAC stage through contrastive difference learning for the feature vectors corresponding to each NAC stage, and obtain the single-stage feature vector; The multi-attention multi-modal fusion module is used to fuse the single-stage feature vectors corresponding to each modality of ultrasound image, and obtain the corresponding feature vectors by using channel attention, spatial attention and self-attention respectively, and obtain the fused feature vector by splicing; The decision module is used to obtain a binary classification prediction result of the NAC efficacy of breast cancer based on the fused feature vector.

10. The breast cancer NAC efficacy prediction system based on a multi-modal multi-branch neural network according to claim 8, characterized in that: During the training process of the multimodal multi-branch neural network, training is achieved by calculating weighted focal loss.

Images