Breast cancer neoadjuvant chemotherapy curative effect early prediction framework based on multiple modes and multiple stages

Abstract

The invention discloses a breast cancer neoadjuvant chemotherapy curative effect early prediction framework based on multi-mode and multi-stage, and the framework introduces a layer-by-layer feature fusion and time sequence mixed attention module to predict whether a patient can achieve complete pathology remission after being subjected to neoadjuvant chemotherapy (NAC). According to the framework, through connection between different modal data layers in the same stage by a bimodal layer-by-layer feature fusion module, complex relations between different modals can be captured while complementary information is learned, so that full fusion of multi-modal data is realized, and a time sequence mixed attention module is adopted to enable a network to focus on a significant focus area and to focus on a significant focus area. Meanwhile, the correlation between focus change characteristics in the chemotherapy process can be learned, and the learning ability of the model on key characteristics of breast tumors is enhanced. The method can effectively assist a doctor in evaluating the curative effect of neoadjuvant chemotherapy of a patient, and has certain clinical popularization significance.

Description

Technical Field

[0001] The present invention relates to the field of computers and proposes a multi-modal and multi-stage framework for early prediction of the efficacy of neoadjuvant chemotherapy for breast cancer. Background Art

[0002] As the most common invasive malignant tumor in women, breast cancer seriously threatens the physical and mental health of women around the world. The 2020 Global Cancer Statistics Report released by the International Agency for Research on Cancer (IARC) shows that in 2020, female breast cancer surpassed lung cancer for the first time and became the cancer with the highest incidence rate in the world. There are about 2.26 million new cases of female breast cancer and 685,000 deaths worldwide, ranking first in the world and fourth in my country in the spectrum of female cancer deaths. The incidence of breast cancer in women in my country has shown an upward trend in recent years, with the number of new cases increasing from 300,000 in 2015 to 420,000 in 2020, accounting for 18% of global breast cancer cases; moreover, the patient group is gradually becoming younger, and the high-incidence age has been concentrated between 45 and 59 years old. my country is facing various challenges in the prevention and treatment of breast cancer. Clinical experiments have found that early prevention and treatment are the key to controlling breast cancer mortality. Therefore, early diagnosis and treatment of breast cancer and timely evaluation of treatment results are important prerequisites for reducing the mortality rate and improving the survival rate of breast cancer patients.

[0003] Neoadjuvant chemotherapy (NAC) refers to a systemic chemotherapy regimen for breast cancer patients before surgery or radiotherapy. It can reduce the primary lesion to a certain extent, kill possible micro-metastatic lesions, reduce the tumor stage, and further reduce the scope of surgical treatment, thereby improving the success rate of surgery and the patient's postoperative survival rate. NAC is not only suitable for locally advanced breast cancer and high-risk operable patients with the risk of breast cancer recurrence or metastasis, but also has good effects in the treatment of inflammatory breast cancer and reducing breast cancer stage. Therefore, it is widely used in the clinical treatment of breast cancer. Compared with adjuvant chemotherapy performed after surgery, preoperative NAC has a better chemotherapy effect. Studies have confirmed that the efficacy of NAC can achieve pathological complete response (pCR) in breast cancer patients, and the disease-free survival rate and overall survival rate are significantly improved. However, the efficacy of NAC often varies due to different pathological genomics and molecular characteristics. Not all patients can achieve good efficacy after NAC. About 10%-35% of patients do not have a significant response after NAC treatment. If patients fail to achieve therapeutic effects after several courses of NAC, it will not only cause irreversible damage to the patient's body and mind, but may even miss the best opportunity for surgical treatment. If the efficacy of chemotherapy can be accurately evaluated in the early stages of NAC treatment, it will help doctors adjust the patient's treatment plan in a timely manner and greatly increase the possibility of patients achieving pCR. Therefore, timely and accurate efficacy evaluation of breast cancer patients receiving NAC treatment and finding a reliable evaluation and early prediction method are crucial to improving the success rate of NAC treatment and saving patients' lives.

[0004] Common breast imaging techniques currently include mammography (MG), ultrasound (US), and magnetic resonance imaging (MRI). Among them, ultrasound is widely used as the preferred examination method for breast diseases due to its convenience, non-invasiveness, no radiation, and low cost. Ultrasound examination is a commonly used detection method in the field of breast cancer NAC. Grayscale ultrasound (GUS) diagnoses changes in tumor size by comparing the changes in high and low echoes of the tumor and tumor margin tissue before and after chemotherapy, thereby evaluating the efficacy of neoadjuvant chemotherapy

[108] . At the same time, color Doppler flow imaging (CDFI) can clearly display the blood flow status inside the tumor and adjacent tissues, fully reflecting the changes in blood vessels before and after breast cancer chemotherapy. Moreover, the imaging data at different stages of chemotherapy can reflect the changing trend of the patient's tumor response during chemotherapy. Therefore, combining GUS and CDFI at multiple stages of chemotherapy can effectively improve the diagnostic accuracy of breast cancer NAC efficacy.

[0005] In clinical practice, ultrasound examination is a highly dependent examination on the operating physician and the diagnostic physician. In particular, its low imaging resolution, low imaging quality caused by speckle noise and artifacts, and insufficient detection of tissue details will affect the objectivity and accuracy of diagnosis to a certain extent. Computer-assisted diagnosis can reduce human interference factors and improve diagnostic efficiency and accuracy. Given that the current feature analysis of ultrasound images of breast cancer NAC has certain limitations and the accuracy of auxiliary diagnosis is not high, it is still a challenging task to develop a model for accurately evaluating the efficacy of breast tumor NAC based on early ultrasound images of patients. Summary of the invention

[0006] In view of the problem that existing models are difficult to accurately predict and evaluate the efficacy of neoadjuvant chemotherapy, a multimodal and multi-stage early prediction framework for the efficacy of neoadjuvant chemotherapy for breast cancer is proposed through a bimodal layer-by-layer feature fusion module and a temporal hybrid attention module. This framework can not only effectively fuse the features of multimodal data, but also effectively extract the time information in the data of different chemotherapy stages. This invention can effectively improve the accuracy of early prediction of neoadjuvant chemotherapy for breast cancer.

[0007] The technical solution of the present invention is:

[0008] A multi-modal and multi-stage framework for early prediction of the efficacy of neoadjuvant chemotherapy for breast cancer. This framework uses a dual-modal layer-by-layer feature fusion module to connect the layers of different modal data in the same stage. In addition to learning complementary information, it can capture the complex relationship between different modalities to achieve full fusion of multimodal data. The temporal mixed attention module is used to enable the network to not only focus on significant lesion areas, but also learn the correlation between lesion change characteristics during chemotherapy, thereby enhancing the model's ability to learn key features of breast tumors. The specific steps are as follows:

[0009] A. Data preprocessing: crop the non-imaging areas of the collected multi-modal and multi-stage ultrasound images and normalize the ultrasound images;

[0010] B. Multimodal feature fusion: The outputs of corresponding layers from different modal data streams are connected through the dual-modal layer-by-layer feature fusion module, so that the different modal data of each layer are related to each other, and the input of each layer in the same modal data stream corresponds to the output of all previous layers, which promotes the flow of information. Its advantage is that through the connection between the layers of different modalities, it can capture the complex relationship between different modalities while learning complementary information and produce richer feature representations;

[0011] C. Multi-stage feature fusion: The pre-NAC and NAC1 features obtained from the fusion of multimodal features are used through the temporal mixed attention module to capture the key features related to breast tumor-related properties and breast tumor lesion changes in the feature map in the three directions of channel, space, and time, reduce the impact of non-tumor lesion tissue information in ultrasound images, and enhance the network model's ability to learn the key features of breast tumor lesion changes;

[0012] D. Efficacy evaluation prediction: The fused features are input into the fully connected layer to calculate whether the patient’s neoadjuvant chemotherapy is effective.

[0013] The specific steps of step B are as follows:

[0014] like Figure 2 As shown in Figure 1, the framework receives grayscale ultrasound (GUS) and color Doppler flow imaging (CDFI) as input. Initially, convolution operations are performed on different modalities to obtain their respective initial feature maps, namely and Subsequently, the i-th layer feature map is fused by performing convolution and nonlinear activation function operations on the (i-1) layer feature map of the modal path and combining it with the (i-1) layer feature map of the other modal path. The fused feature map is calculated by combining all previous feature maps of the modal path with the (i-1) layer feature map of the other modal path. For example, for the GUS modality, the second layer feature map By calculating the feature map And the first layer fusion feature map To obtain, where the fusion feature map is calculated by calculating the characteristic graph of the GUS pathway and And the characteristic diagram of the CDFI path The calculation method of this layer-by-layer feature fusion module is as follows:

[0015]

[0016] in, and They represent the fused feature maps obtained through the layer-by-layer feature fusion module, Conv represents the dimension reduction achieved through the 1×1 convolution operation, and […] represents the connection operation of the feature map. Finally, The (i-1)th layer feature map of the GUS path Combined to calculate the i-th layer feature map Characteristic diagram of CDFI path The same is true. Feature map and It can be calculated as follows:

[0017]

[0018] in, and They represent the mapping functions consisting of convolution operations and non-linear activation functions respectively.

[0019] The specific steps of step C are as follows:

[0020] The temporal hybrid attention module includes the temporal attention module (TA) of the convolutional block attention module (CBAM). The CBAM attention mechanism is a more comprehensive feature attention method that combines the attention in the channel domain and the spatial domain. By adding the spatial attention module on the basis of the channel attention module, the performance of the target region is enhanced. Assume The output feature maps of pre-neoadjuvant chemotherapy (pre-NAC) and first neoadjuvant chemotherapy (NAC1) of GUS and CDFI, respectively, where g and c represent GUS and CDFI modal data, p and n represent pre-NAC and NAC1 data, and the corresponding feature maps are generated by the CBAM attention module These feature maps can be calculated as follows:

[0021]

[0022] Among them, M ca Represents the weighting factor of channel attention, M sa represents the weighting factor of spatial attention, Represents element-wise multiplication. Dual-modal fusion feature map of two chemotherapy stages It is obtained by fusing different modal data at the same stage. These feature maps can be calculated as follows:

[0023]

[0024] After obtaining the CBAM features between each stage, the feature map are concatenated together to obtain the CBAM feature sequence which can be represented by X, calculated as follows:

[0025]

[0026] Among them, stack means splicing feature maps according to the time dimension

[0027] Then input feature M∈R T×C×H×W , firstly, it is transformed into two feature spaces q(X) and k(X) through two sets of 1×1×1 convolutions, where C, H and W represent channels, height and width respectively, q(X) = W q X and k(X) = W k X(W q and W k is a trainable weight matrix). We then reshape both q(X) and k(X) into R M×C , where M = T × H × W, to calculate the attention map between any pair of regions through the time dimension. Attention map F T (X) is as follows:

[0028]

[0029] among them F T (X) j,i Represents X i With X j T represents the time dimension or the length of the sequence, and i and j represent the indices of the two regions or feature positions involved in the attention calculation. The output feature map of the temporal attention is in Feature vectors representing different positions in the sequence. As shown below:

[0030]

[0031] Where v(X) = W v X(W v is a learnable matrix), and X jare added back to retain more information. With such a design, the hybrid attention module can both focus on important tumor tissue features and mine the potential features of tumor lesion progression before and after NAC1.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] 1. The mainstream deep learning classification model is used to evaluate and predict the efficacy of neoadjuvant chemotherapy for breast cancer. The prediction model cannot effectively integrate the features of multimodal data, and is not effective in exploring the highly complex relationships and complementary feature information between different modalities. There is a lack of effective means of sharing the features of multimodal data. The newly proposed new bimodal layer-by-layer feature fusion (BLFFM). It can effectively mine the highly complex associations between different modal data and the complementary feature information between data, and is an effective method for sharing bimodal data features.

[0034] 2. Traditional models cannot effectively extract time information from data of different chemotherapy stages. This paper introduces a convolutional block attention module (CBAM) and a temporal attention module (TA) to form a temporal hybrid attention module (THAM). It not only learns the important features of the image, but also learns the association between lesion features during chemotherapy, enhancing the network's ability to understand the key features of breast tumor progression. This helps to enhance the network's performance in efficacy prediction tasks.

[0035] 3. The present invention establishes a multimodal and multi-stage early prediction framework for the efficacy of neoadjuvant chemotherapy for breast cancer through a bimodal layer-by-layer feature fusion module (BLFFM) and a temporal hybrid attention module, thereby further improving the accuracy of predicting and evaluating the efficacy of neoadjuvant chemotherapy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a structural diagram of the early prediction framework of the efficacy of neoadjuvant chemotherapy for breast cancer based on multi-modality and multi-stage of the present invention.

[0037] Figure 2 This is the structure diagram of the bimodal layer-by-layer feature fusion module (BLFFM) in step B.

[0038] Figure 3 It is a performance comparison of different data combinations used in step B.

[0039] Figure 4 It is the performance comparison of different fusion methods in step B.

[0040] Figure 5 The comparison of ablation experiments of the hybrid distillation strategy in step D is listed.

[0041] Figure 6A comparison between the privileged information-based dual-path distillation framework for carotid plaque echo classification of the present invention and the mainstream deep learning classification method is listed. DETAILED DESCRIPTION

[0042] The preferred embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0043] Embodiment 1:

[0044] The concept of the present invention is: first, pre-process the multimodal ultrasound image to remove irrelevant information. Then, through the dual-modal layer-by-layer feature fusion module, the connection between the different modal data layers at the same stage can be captured while learning complementary information. The complex relationship between different modalities can be fully integrated to achieve multimodal data, and the temporal mixed attention module is used to make the network not only focus on the significant lesion area, but also learn the correlation between the lesion change characteristics during chemotherapy, strengthen the model's learning ability for key features of breast tumors, and ultimately improve the accuracy of early prediction of neoadjuvant chemotherapy for breast cancer.

[0045] According to the above inventive concept, the present invention adopts the following technical solutions:

[0046] A. Data preprocessing: cropping the non-imaging areas of the collected multi-modal and multi-stage ultrasound images and normalizing the ultrasound images;

[0047] B. Multimodal feature fusion: The outputs of corresponding layers from different modal data streams are connected through the dual-modal layer-by-layer feature fusion module, so that the different modal data of each layer are related to each other, and the input of each layer in the same modal data stream corresponds to the output of all previous layers, which promotes the flow of information. Its advantage is that through the connection between the layers of different modalities, it can capture the complex relationship between different modalities while learning complementary information and produce richer feature representations;

[0048] C. Multi-stage feature fusion: The pre-NAC and NAC1 features obtained from the fusion of multimodal features are used through the temporal mixed attention module to capture the key features related to breast tumor-related properties and breast tumor lesion changes in the feature map in the three directions of channel, space, and time, reduce the impact of non-tumor lesion tissue information in ultrasound images, and enhance the network model's ability to learn the key features of breast tumor lesion changes;

[0049] D. Efficacy evaluation prediction: The fused features are input into the fully connected layer to calculate whether the patient's neoadjuvant chemotherapy is effective.

[0050] This method uses a four-branch network architecture, and inputs grayscale ultrasound (GUS) and color Doppler flow imaging (CDFI) before chemotherapy (pre-NAC) and the first stage of chemotherapy (NAC1). Multimodal and multi-stage data are modeled through a layer-by-layer feature fusion module and a temporal hybrid attention module, which not only allows the network to focus on significant lesion areas, but also learns the correlation between lesion change features during chemotherapy, and strengthens the model's ability to learn key features of breast tumors.

[0051] Example 2: In this example, the server running the experiment is specifically configured as follows: NVIDIA GeForce RTX2080Ti independent graphics card and two Intel Xeon Silver 4110CPUs, 64GB RAM. In terms of model building, this framework and the comparison model are both implemented based on the open source deep learning tool PyTorch1.9.0. In terms of experimental settings, the experimental model uses Adam with adaptive learning as the optimizer, the initial learning rate is set to 0.001, the weight decay factor is 0.1, the number of iterations is set to 500, and the learning rate is reduced to 1 / 10 of the previous one every 150 iterations. The loss function uses the cross entropy loss function used in the classification task, and the batch size is set to 8. There are a total of six classification evaluation indicators involved, namely class accuracy (Acc), sensitivity (Sen), specificity (Spec), positive predictive value (PPV), negative predictive value (NPV) and F1 score.

[0052] The data set used in this embodiment includes ultrasound videos (including GUS and CDFI) and pathological data from 101 patients with locally advanced breast cancer. This data set is a tracking ultrasound image data set, and the pathological results after chemotherapy are confirmed by pathological tissue sections. That is, each patient has video data of GUS and CDFI in the four stages of pre-NAC, NAC1, NAC2 and NAC4, as well as the final pathological reaction result report. In this experiment, only the ultrasound video data of the two chemotherapy stages of pre-NAC and NAC1 were used for research, and the pathological reaction result report was used as the gold standard. After data preprocessing, the original video data can obtain 3000 sets of NAC ultrasound image combinations, and each set of ultrasound image combinations contains the image data of GUS and CDFI in the pre-NAC and NAC1 stages of the patient and the corresponding pathological results of NAC reaction. Among them, there are 930 sets of ultrasound images with pathological results of pCR (37 patients), and there are 2070 sets of ultrasound images with pathological results of non-pCR (64 patients).

[0053] The present invention provides a multi-modal and multi-stage early prediction framework for the efficacy of neoadjuvant chemotherapy for breast cancer. Figure 1 As shown, the following steps are included:

[0054] A. Data preprocessing: The non-imaging areas of the collected multi-modal and multi-stage ultrasound images are cropped and normalized.

[0055] Ultrasound images;

[0056] B. Multimodal feature fusion: The outputs of corresponding layers from different modal data streams are connected through the dual-modal layer-by-layer feature fusion module, so that the different modal data of each layer are related to each other, and the input of each layer in the same modal data stream corresponds to the output of all previous layers, which promotes the flow of information. Its advantage is that through the connection between the layers of different modalities, it can capture the complex relationship between different modalities while learning complementary information and produce richer feature representations. The specific steps are as follows:

[0057] like Figure 2 As shown in Figure 1, the framework receives grayscale ultrasound (GUS) and color Doppler flow imaging (CDFI) as input. Initially, convolution operations are performed on different modalities to obtain their respective initial feature maps, namely and Subsequently, the i-th layer feature map is fused by performing convolution and nonlinear activation function operations on the (i-1) layer feature map of the modal path and combining it with the (i-1) layer feature map of the other modal path. The fused feature map is calculated by combining all previous feature maps of the modal path with the (i-1) layer feature map of the other modal path. For example, for the GUS modality, the second layer feature map By calculating the feature map And the first layer fusion feature map To obtain, where the fusion feature map is calculated by calculating the characteristic graph of the GUS pathway and And the characteristic diagram of the CDFI path The calculation method of this layer-by-layer feature fusion module is as follows:

[0058]

[0059] in, and They represent the fused feature maps obtained through the layer-by-layer feature fusion module, Conv represents the dimension reduction achieved through the 1×1 convolution operation, and […] represents the connection operation of the feature map. Finally, The (i-1)th layer feature map of the GUS path Combined to calculate the u-th layer feature map Characteristic diagram of CDFI path The same is true. Feature map and It can be calculated as follows:

[0060]

[0061] in, and They represent the mapping functions consisting of convolution operations and non-linear activation functions respectively.

[0062] C. Multi-stage feature fusion: The pre-NAC and NAC1 features obtained from the fusion of multimodal features are captured in the channel, space, and time directions by the temporal mixed attention module. The key features related to breast tumor-related properties and breast tumor lesion changes in the feature map are captured to reduce the impact of non-tumor lesion tissue information in ultrasound images and enhance the network model's ability to learn the key features of breast tumor lesion changes. The specific steps are as follows:

[0063] The temporal hybrid attention module includes the convolutional block attention module (CBAM) and the temporal attention module (TA). The CBAM attention mechanism is a more comprehensive feature attention method that combines the attention in the channel domain and the spatial domain. By adding the spatial attention module on the basis of the channel attention module, the performance of the target area is enhanced. Assume The output feature maps of pre-neoadjuvant chemotherapy (pre-NAC) and first neoadjuvant chemotherapy (NAC1) of GUS and CDFI, respectively, where g and c represent GUS and CDFI modal data, p and n represent pre-NAC and NAC1 data, and the corresponding feature maps are generated by the CBAM attention module These feature maps can be calculated as follows:

[0064]

[0065] Among them, M ca Represents the weighting factor of channel attention, M sa represents the weighting factor of spatial attention, Represents element-wise multiplication. Dual-modal fusion feature map of two chemotherapy stages It is obtained by fusing different modal data at the same stage. These feature maps can be calculated as follows:

[0066]

[0067] After obtaining the CBAM features between each stage, the feature map are concatenated together to obtain the CBAM feature sequence which can be represented by X, calculated as follows:

[0068]

[0069] Among them, stack means splicing feature maps according to the time dimension

[0070] Then the input feature X∈R T×C×H×W First, two sets of 1×1×1 convolutions are used to transform it into two feature spaces q(X) and k(X), where C, H and W represent the channel, height and width respectively, and q(X) = W q X and k(X) = W k X(W q and W k is a trainable weight matrix). We then reshape both q(X) and k(X) into R M×C , where M = T × H × W, to calculate the attention map between any pair of regions through the time dimension. Attention map F T (X) is as follows:

[0071]

[0072] among them F T (X) j,i Represents X i With X j T represents the time dimension or the length of the sequence, and i and j represent the indices of the two regions or feature positions involved in the attention calculation. The output feature map of the temporal attention is in Feature vectors representing different positions in the sequence. As shown below:

[0073]

[0074] Where v(X) = W v X(W v is a learnable matrix), and X j is added back to retain more information. With this design, the hybrid attention module can both focus on important tumor tissue features and mine potential features of tumor lesion progression before and after NAC1.

[0075] D. Efficacy evaluation prediction: The fused features are input into the fully connected layer to calculate whether the patient’s neoadjuvant chemotherapy is effective.

[0076] Reference Figure 2 , showing the flowchart of the dual-modality layer-by-layer feature fusion module for the fusion of grayscale ultrasound and color Doppler blood flow images.

[0077] Reference Figure 3 , showing the performance comparison of different data combinations. The purpose of this experiment is to demonstrate the rationality of the combined use of grayscale ultrasound (GUS) and color Doppler flow imaging (CDFI) data from preoperative neoadjuvant chemotherapy (pre-NAC) and first neoadjuvant chemotherapy (NAC1). The experiment used GUSpre, CDFIpre, GUS1 and CDFI1, as well as their combinations GUSpre+CDFIpre, GUS1+CDFI1, GUSpre+GUS1 and GUSpre+GUS1+CDFIpre+CDFI1 (GUSpre+1+CDFIpre+1) to predict the efficacy. In the GUSpre+1+CDFIpre+1 combination, the model achieved the best results in terms of accuracy, positive predictive value, negative predictive value and F1 score indicators, while sensitivity and specificity were close to the best results. At the same time, the experimental results of GUSpre, CDFIpre and GUSpre+CDFIpre, as well as the experimental results of GUS1, CDFI1 and GUS1+CDFI1, showed that the prediction effect of dual-modality data was better than that of a single modality at the same chemotherapy stage. It can be further observed that the results of GUSpre+GUS1 are better than those of GUSpre and GUS1, which indicates that the effectiveness of multi-stage data is better than that of a single chemotherapy stage and confirms the effectiveness of dual-modal data of different chemotherapy stages for predicting the efficacy of neoadjuvant chemotherapy.

[0078] Reference Figure 4, shows the performance comparison between the introduction of the bimodal layer-by-layer feature fusion module and other fusion methods. In this experiment, "Sum" and "Concat" are two commonly used feature fusion methods. "Sum" adds values ​​to the feature map element by element while keeping the number of channels unchanged. At the same time, "Concat" performs a merge operation on the number of channels. Neural Discriminative Dimensionality Reduction (NDDR)

[32] is a feature fusion method that can automatically learn each abstraction layer in different data streams. Specifically, features with the same spatial resolution in a single branch network are cascaded through channels and then reduced in dimension through a 1×1 convolution kernel. Finally, the fused features are fed to the next layer of the network. BLFFM is a layer-by-layer feature fusion method proposed in this paper. BLFFM1 and BLFFM2 represent two implementations to maintain the same spatial resolution of feature maps from different convolutional layer outputs, where BLFFM1 is a downsampling method of nearest neighbor interpolation, and BLFFM2 maintains the same spatial resolution by cropping edge information. Experimental results show that the BLFFM method achieves the best performance. Among them, the accuracy, specificity, positive predictive value and F1 score of BLFFM2 reached 83.93±1.22%, 74.62±7.44% and 73.98±1.4%, respectively. BLMFF1 has the best sensitivity (76.20±4.64%) and negative predictive value (89.14±1.75%). In addition, compared with the "Sum" fusion method, the accuracy of BLFFM2 increased by 2.70% (81.23% vs.83.93%), the specificity increased by 4.04% (83.89% vs.87.93%), and the F1 score increased by 3.04% (70.94% vs.73.98%), ranking third in the overall ranking. This shows that the proposed method can effectively integrate and make full use of the rich feature information between multimodal and unimodal data, thereby improving the prediction performance of the model.

[0079] Reference Figure 5 , shows the ablation experiment of different attention mechanisms. The purpose of this experiment is to prove that the temporal mixed attention module is effective. Figure 4 It can be clearly seen that the layer-by-layer feature fusion module is effective; therefore, Figure 5The experimental results in this paper all use ResNet50 with the introduction of the bimodal layer-by-layer feature fusion module BLFFM2 as the baseline model to verify the effectiveness of the temporal hybrid attention module (THAM). First, the influence of channel and spatial attention mechanisms is explored, including five classic attention mechanisms: Squeeze-and-Excitation (SE), Bottleneck Attention Module (BAM), Dual Attention Network (DANet), Coordinate Attention (CA), and Convolutional Block Attention Module (CBAM). The results show that among the five channel and spatial attention mechanisms, the CBAM attention mechanism achieves the best F1 score of 79.07±1.68%, which is 5.09% (79.07% vs. 73.98%), 3.68% (79.07% vs. 75.39%), 1.69% (79.07% vs. 77.38%), 4.27% (79.07% vs. 74.80%), and 2.35% (79.07% vs. 76.72%) higher than the no attention mechanism (baseline), SE, BAM, DANet, and CA, respectively. In this study, the THAM module is based on the CBAM attention mechanism. Adding the temporal attention (TA) module achieves the best prediction results in terms of accuracy, sensitivity, negative predictive value, and F1 score, with an F1 score improvement of 2.60% (81.67% vs. 79.07%) compared to the best performing CBAM attention module. This shows that the traditional attention mechanism method is extended to the time dimension to adapt to the characteristics of tracking image data, which can effectively extract multiple types of information, including global spatial, channel and temporal information, and enhance feature representation to improve network performance. These results show that paying attention to the time series information of ultrasound image data is helpful for the prediction of NAC response. The results of the ablation experiment show that removing any distillation method in the mixed distillation of the present invention will lead to a decrease in model performance. This proves that the combination of various distillation methods within the mixed distillation is beneficial, and they play an important role in improving model performance together.

[0080] Reference Figure 6, showing the performance comparison between the multimodal and multi-stage early prediction framework for the efficacy of neoadjuvant chemotherapy for breast cancer of the present invention and eight other popular classification methods. The purpose of this experiment is to test the performance of the proposed multimodal and multi-stage early prediction framework for the efficacy of neoadjuvant chemotherapy for breast cancer. In addition to the method of the present invention, other popular classification methods compared in the experiment include EfficientNet, DenseNet, ShuffleNetV2, Xception, MobileNetV2, InceptionV4, Inception_ResNetV2, ResNeXt. The experimental results show that the early prediction method for neoadjuvant chemotherapy for breast cancer proposed in the present invention is superior to popular prediction methods in all performance indicators.

[0081] In summary, the multimodal and multi-stage early prediction framework of the efficacy of neoadjuvant chemotherapy for breast cancer of the present invention introduces a dual-modal layer-by-layer fusion module and a temporal hybrid attention module to propose a high-accuracy early prediction method of the efficacy of neoadjuvant chemotherapy for breast cancer. This method can not only effectively mine the highly complex associations between different modal data and the complementary feature information between data, but also learn the associations between lesion features during chemotherapy, and enhance the network's ability to understand the key features of breast tumor progression. This helps to enhance the performance of the network in the efficacy prediction task.

[0082] The present invention is described in conjunction with the drawings and specific embodiments of the specification only to help understand the method and core idea of ​​the present invention. The method described in the present invention is not limited to the embodiments described in the specific embodiments. Other implementations derived by those skilled in the art based on the method and ideas of the present invention also belong to the technical innovation scope of the present invention. The content of this specification should not be understood as limiting the present invention.

Claims

1. A multi-modal and multi-stage framework for early prediction of the efficacy of neoadjuvant chemotherapy for breast cancer, characterized by: This framework uses a bimodal layer-by-layer feature fusion module to connect different modal data levels in the same stage. In addition to learning complementary information, it can capture the complex relationship between different modalities to achieve full fusion of multimodal data. It also uses a temporal mixed attention module to enable the network to not only focus on significant lesion areas, but also learn the correlation between lesion change characteristics during chemotherapy, thereby enhancing the model's ability to learn key features of breast tumors.

2. The early prediction framework of the efficacy of neoadjuvant chemotherapy for breast cancer based on multimodality and multistage according to claim 1 is characterized in that The specific steps are as follows: A. Data preprocessing: crop the non-imaging areas of the collected multi-modal and multi-stage ultrasound images and normalize the ultrasound images; B. Multimodal feature fusion: The outputs of corresponding layers from different modal data streams are connected through the bimodal layer-by-layer feature fusion module, so that the different modal data of each layer are related to each other, and the input of each layer in the same modal data stream corresponds to the output of all previous layers, which promotes the flow of information; C. Multi-stage feature fusion: The pre-NAC and NAC1 features obtained from the fusion of multimodal features are used through the temporal mixed attention module to capture the key features related to breast tumor-related properties and breast tumor lesion changes in the feature map in the three directions of channel, space, and time, reduce the impact of non-tumor lesion tissue information in ultrasound images, and enhance the network model's ability to learn the key features of breast tumor lesion changes; D. Efficacy evaluation prediction: The fused features are input into the fully connected layer to calculate whether the patient’s neoadjuvant chemotherapy is effective.

3. The early prediction framework of the efficacy of neoadjuvant chemotherapy for breast cancer based on multimodality and multistage according to claim 2 is characterized in that Step B: The framework receives grayscale ultrasound GUS and color Doppler blood flow imaging CDFI as input. Initially, convolution operations are performed on different modalities to obtain their respective initial feature maps, namely: and Subsequently, the i-th layer feature map is fused by performing convolution and nonlinear activation function operations on the (i-1) layer feature map of the modal path and combining it with the (i-1) layer feature map of another modal path. The fused feature map is calculated by combining all previous feature maps of the modal path with the (i-1) layer feature map of another modal path.

4. The early prediction framework for the efficacy of neoadjuvant chemotherapy for breast cancer based on multimodality and multistage according to claim 3 is characterized in that Step B: For grayscale ultrasound modality, the second layer feature map By calculating the feature map And the first layer fusion feature map To obtain, where the fusion feature map is calculated by calculating the characteristic graph of the GUS pathway and And the characteristic diagram of the CDFI path To obtain, the calculation method of this layer-by-layer feature fusion module is as follows: in, and They represent the fused feature maps obtained by the layer-by-layer feature fusion module, Conv represents the dimension reduction achieved by the 1×1 convolution operation, and […] represents the connection operation of the feature map; finally, The (i-1)th layer feature map of the GUS path Combined to calculate the i-th layer feature map Characteristic diagram of color Doppler blood flow imaging path The same is true for the feature map and The calculation is as follows: in, and They represent the mapping functions consisting of convolution operations and non-linear activation functions respectively.

5. The early prediction framework of the efficacy of neoadjuvant chemotherapy for breast cancer based on multimodality and multistage according to claim 3 is characterized in that In step C: The temporal hybrid attention module includes the convolutional block attention module (CBAM) and the temporal attention module (TA). CBAM combines the attention of the channel domain and the spatial domain. By adding the spatial attention module on the basis of the channel attention module, the performance of the target area is enhanced. The output feature maps of pre-neoadjuvant chemotherapy pre-NAC and the first neoadjuvant chemotherapy NAC1 represent GUS and CDFI, respectively, where g and c represent GUS and CDFI modal data, p and n represent pre-NAC and NAC1 data, and the corresponding feature maps are generated by the CBAM attention module These feature maps are calculated as follows: Among them, M ca represents the weighting factor of channel attention, M sa represents the weighting factor of spatial attention, Represents element-wise multiplication; dual-modal fusion feature map of two chemotherapy stages It is obtained by fusing different modal data at the same stage. These feature maps are calculated as follows: After obtaining the CBAM features between each stage, the feature map are concatenated together to obtain the CBAM feature sequence which can be represented by X, calculated as follows: Among them, stack means splicing feature maps according to the time dimension Then input feature X∈R T×C×H×W A 1×1×1 convolution is used to transform into two feature spaces q(X) and k(X), where C, H and W represent channel, height and width respectively, and q(X) = W q X and k(X) = W k XW q and W k is a trainable weight matrix. Subsequently, we reshape both q(X) and k(X) into R M×C , where M = T × H × W, to compute the attention map between any pair of regions through the time dimension.

6. The early prediction framework for the efficacy of neoadjuvant chemotherapy for breast cancer based on multimodality and multistage according to claim 5 is characterized in that Attention map F T (X) is as follows: among them F T (X) j,i Represents X i With X j The correlation degree between them, T represents the time dimension or the length of the sequence, i and j represent the indexes of the two regions or feature positions involved in the attention calculation, and the output feature map of the time attention is in represents the feature vectors at different positions in the sequence, As shown below: Where v(X) = W v X(W v is a learnable matrix), and X j Was added back to hold more information.

Images