Breast cancer neoadjuvant chemotherapy efficacy prediction device based on multi-feature fusion

Abstract

The present invention discloses a device for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on multi-feature fusion. The device comprises: an image acquisition module for acquiring multimodal breast images of the patient's lesion area; an image preprocessing module for cropping each modality of breast image to obtain a 3D region of interest (ROI) of that modality; a lesion region segmentation module for obtaining the lesion region based on the 3D ROI of each modality of breast image; and an efficacy diagnosis module for calculating and splicing the radiometric feature vector of the lesion region with the deep high-level features of each 3D ROI, and predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on the spliced vectors of each modality of breast image. The present invention can accurately predict the efficacy of NAC in an efficient and non-invasive manner, allowing for the development of personalized and precise treatment plans for patients.

Description

Technical Field

[0001] The present invention relates to the fields of medical image processing and deep learning, and in particular to a device for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on multi-feature fusion. Background Art

[0002] Breast cancer is one of the most common malignant tumors in women, with over 2 million cases diagnosed annually. It is the leading cause of cancer-related death in women and poses a serious threat to their health. For patients with locally advanced breast cancer, lumpectomy or mastectomy is typically used as the standard treatment, effectively reducing the rate of local recurrence. However, radical surgery may result in numerous sequelae and permanent stomas, which in severe cases can accelerate the patient's death. Some malignant tumors with severe symptoms cannot be directly cured by surgery or do not meet the surgical criteria. Studies have shown that neoadjuvant chemotherapy (NAC) can reduce the size of the primary tumor, lower the tumor stage, and increase the possibility of adopting a breast-conserving treatment strategy instead of a complete mastectomy. It is currently a very important approach in the comprehensive treatment of breast cancer.

[0003] However, breast cancer is a highly heterogeneous disease, and the treatment outcome for each patient is affected by multiple factors, including tumor molecular subtype and treatment regimen. Clinical trials have found that some patients are insensitive to NAC, and the efficacy varies greatly from patient to patient. Accurately predicting the efficacy of NAC can help select patients who may benefit from NAC and avoid excessive preoperative chemotherapy for patients who do not respond to chemotherapy, which can lead to missed surgery opportunities. Another advantage is that it allows for early observation of treatment response, allowing for rapid modification of treatment plans in the event of a poor response. Therefore, timely and accurate prediction of NAC efficacy and adjustment of treatment plans are key clinical steps.

[0004] The current gold standard for evaluating NAC efficacy is based on histopathology results. However, pathological assessment cannot monitor tumor response to chemotherapy drugs in real time during treatment, much less predict NAC response early. Furthermore, this invasive histological biopsy has limitations such as poor timeliness, secondary infection, difficulty in repeat testing, and high requirements for pathology departments. Therefore, finding noninvasive methods that can effectively predict efficacy is an urgent issue in NAC. Noninvasive imaging examinations can be performed multiple times during treatment. Magnetic resonance imaging (MRI) is one of the most sensitive imaging methods for detecting breast cancer, and its diverse imaging modalities can provide meaningful information from multiple perspectives.

[0005] Currently, most studies using MRI to evaluate the efficacy of breast NAC rely on pre- and post-NAC comparison data, which fails to address the issue of efficacy prediction. The vast majority of efficacy evaluation methods utilize only a single imaging sequence within the MRI imaging modality, failing to capitalize on the advantages of multi-parameter MRI sequences. Furthermore, feature extraction relies solely on radiomics or convolutional neural network methods, resulting in limited accuracy and a lack of deep feature extraction and high-level data feature fusion. To address these issues, a method and system for predicting pathological complete response to NAC in breast cancer based on multi-parameter breast MRI medical imaging sequences is needed to develop personalized, precise treatment plans for patients. Summary of the Invention

[0006] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a device for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on multi-feature fusion, which can accurately predict the efficacy of NAC efficiently and non-invasively, and formulate personalized and precise treatment plans for patients.

[0007] The technical contents of the present invention include:

[0008] A device for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on multi-feature fusion, comprising:

[0009] An image acquisition module is used to acquire a multimodal breast image of the patient's lesion area, wherein the modalities of the multimodal breast image include: T2W1 modality, DCE-MRI modality and DWI modality;

[0010] An image preprocessing module is used to crop each modality breast image to obtain a 3D region of interest of the modality breast image;

[0011] The lesion region segmentation module is used to obtain the lesion region based on the 3D region of interest of each modality breast image;

[0012] The efficacy diagnosis module is used to calculate and splice the radiation feature vector of the lesion area and the deep advanced features of each 3D region of interest, and predict the efficacy of neoadjuvant chemotherapy for breast cancer based on the spliced vectors of breast images of each modality.

[0013] Optionally, the image preprocessing module is further configured to:

[0014] Based on a heterogeneous image registration unit, performing heterogeneous image registration on the multimodal breast images;

[0015] Performing image denoising on the multimodal breast image based on an image denoising unit;

[0016] performing image standardization on the multimodal breast image based on an image standardization unit;

[0017] Based on the image enhancement unit, image enhancement is performed on the multimodal breast image.

[0018] Optionally, the heterogeneous image registration unit is configured to:

[0019] A rigid body registration algorithm based on standard mutual information is used, with the image of the DCE-MRI modality as a fixed image, and the images of the T2W1 modality and the DWI modality as floating images, respectively, for rigid body transformation, so that points corresponding to the same position in space in the image correspond one to one.

[0020] Optionally, the image denoising unit is configured to:

[0021] The N4 bias field correction tool in the Python toolkit SimpleITK was used to remove the low-frequency intensity uneven signal in the breast image.

[0022] Optionally, the image normalization unit is configured to:

[0023] The images of the T2W1 modality, the DCE-MRI modality, and the DWI modality are respectively normalized using a Z-score method.

[0024] Normalize the grayscale value range of each pixel in the normalized image.

[0025] Optionally, the lesion area segmentation module is used to:

[0026] Construct a semantic segmentation network based on 3D-UNet, where the structure of the semantic segmentation network is P layers of 3D convolutional layers and P′ sub-pooling layers in the downsampling path, and Q layers of 3D convolutional layers and Q′ sub-pooling layers in the upsampling path. The submodules in the downsampling path are replaced with submodules with residual connections.

[0027] Obtain multimodal sample breast images, use the sample breast images of T2W1 modality, DCE-MRI modality, and DWI modality as the input images of the semantic segmentation network, and train the semantic segmentation network with the labeled mask segmentation image as the training target to obtain a semantic segmentation model, wherein the loss function of the training process is Dice Loss = 1-Dice, y true represents the actual segmentation label, y pred Represents the predicted output graph, and smooth represents the smoothing term;

[0028] Breast images in T2W1 mode, DCE-MRI mode and DWI mode are respectively used as input images of the semantic segmentation model to obtain the lesion area.

[0029] Optionally, acquiring a multimodal sample breast image includes:

[0030] A data acquisition unit is used to obtain original imaging data and corresponding neoadjuvant chemotherapy efficacy category labels from a database on a patient-by-patient basis, wherein the neoadjuvant chemotherapy efficacy category labels include: pathological complete remission and pathological incomplete remission;

[0031] a data selection unit, configured to set exclusion criteria based on the efficacy prediction problem to select data, wherein the exclusion criteria include: patients who did not undergo bilateral breast MRI examination before treatment, patients who did not complete a full cycle of neoadjuvant chemotherapy or discontinued treatment for some reason, patients who did not undergo surgery after neoadjuvant chemotherapy, patients with incomplete postoperative pathological evaluation, and patients with distant metastasis;

[0032] The data annotation unit is used to obtain pixel-level annotations of the lesion area based on the annotation results of professional physicians on the medical annotation software ITK-SNAP;

[0033] The data collation unit is used to integrate data based on patients and convert the DICOM file format of the original data into NIfTI files.

[0034] Optionally, the efficacy diagnosis module is used to:

[0035] An initial radiomic feature extraction unit is used to select radiomic features to be extracted and extract radiomic features from the lesion area using open source radiomics analysis software to obtain initial radiomic features;

[0036] a radiation feature calculation unit, configured to input the autoencoder network encoding result of the initial radiation feature into a plurality of fully connected layers to obtain the radiation feature within the lesion area;

[0037] A deep high-level feature extraction unit is used to input the 3D region of interest of each modality breast image into a convolutional neural network based on ResNet50 to obtain deep high-level features of the modality;

[0038] a vector stitching unit, configured to stitch the radiological features with the deep high-level features of each modality to obtain a stitching vector of the corresponding modality breast image;

[0039] The prediction unit is configured to input the concatenated vectors of breast images of each modality into a multi-instance learning network, so that an attention layer in the multi-instance learning network assigns a weight to the concatenated vectors of each modality of breast images. The classifier in the multi-instance learning network obtains a predicted efficacy of neoadjuvant chemotherapy for breast cancer based on the weighted summation of the concatenated vectors of breast images of each modality.

[0040] Optionally, the initial radiation feature extraction unit is configured to:

[0041] For the sample lesion area, open source radiomics analysis software was used to obtain all radiomic features, including morphological features, first-order grayscale histogram features, second-order and high-order texture features, and filter-based features;

[0042] The importance of each radiological feature is scored using the random forest algorithm to determine the radiological features to be extracted;

[0043] Select the radiological features to be extracted.

[0044] Optionally, the deep high-level feature extraction unit is used to:

[0045] For breast images of the DCE-MRI modality, the images of stage one, stage three, and stage five are taken as the input images of the convolutional neural network to obtain deep and high-level features of the DCE-MRI modality;

[0046] and,

[0047] For breast images in the T2W1 modality, the single-channel image is copied three times and used as the input image of the convolutional neural network to obtain deep and advanced features of the T2W1 modality;

[0048] and,

[0049] For breast images of the DWI modality, the single-channel image is copied three times and used as the input image of the convolutional neural network to obtain the deep and advanced features of the DWI modality.

[0050] Optionally, the loss of training the efficacy prediction module is Loss=L MSE +αL CE , where L MSE represents the reconstruction loss for training the autoencoder network, L CE Describes the prediction loss of training the multi-instance learning network, and α represents the reconstruction loss L MSE and prediction loss L CE Hyperparameters that balance between.

[0051] The advantages of the present invention compared with the prior art are:

[0052] (1) The present invention predicts the efficacy of NAC based on breast MRI, which has the advantages of being non-invasive and timely compared to pathological examinations. Compared with other non-invasive imaging examinations such as CT and ultrasound, the MRI used in the present invention has unique advantages in detecting breast lesions due to its good tissue resolution and multiple imaging modes. Compared with other research methods that use MRI for efficacy prediction, the present invention can complete accurate NAC efficacy prediction using only pre-NAC data, and has broader application prospects.

[0053] (2) The present invention proposes a multi-feature fusion network that combines traditional radiomic features with deep features extracted using convolutional neural networks. Furthermore, compared to the 2D medical images used in many existing technologies, the present invention targets 3D medical imaging data, comprehensively considering the three-dimensional structural properties of the lesion area, and is more in line with doctors' thinking when reading medical images.

[0054] (3) The present invention proposes a therapeutic effect prediction model based on multi-instance learning to solve the multimodality problem in multi-parameter MRI sequences. It cleverly applies the multi-instance learning idea in the field of machine learning to medical multimodal images, and improves the recognition performance of therapeutic effect prediction by using different modalities of lesions.

[0055] (4) The present invention encapsulates the proposed efficacy prediction method and model into an auxiliary diagnosis system. This system eliminates the need for precise lesion segmentation during diagnosis, thus avoiding the tedious segmentation steps. This end-to-end approach is more automated and practical. The system also greatly improves the operability of clinicians and the intuitiveness of the results. Compared with manual reading, it is faster, more efficient, and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a general framework diagram of the method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer described in an embodiment of the present invention.

[0057] Figure 2 Schematic diagram of the structure of the multi-feature fusion network described in an embodiment of the present invention.

[0058] Figure 3 Schematic diagram of the structure of a multi-instance learning network based on multi-parameter MRI sequences according to an embodiment of the present invention.

[0059] Figure 4 Schematic diagram of the composition of the breast cancer neoadjuvant chemotherapy efficacy prediction and diagnosis system according to an embodiment of the present invention.

[0060] Figure 5 This is a diagram of the main interface of the breast cancer neoadjuvant chemotherapy efficacy prediction and diagnosis system according to an embodiment of the present invention.

[0061] Figure 6 This is an overall flow chart of the breast cancer neoadjuvant chemotherapy efficacy prediction and diagnosis system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0063] The technical solution adopted by the present invention is: a device for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on multi-feature fusion, such as Figure 1 As shown, the overall framework of the present invention includes:

[0064] 1. Image acquisition module

[0065] The image acquisition module is used to obtain breast images of the patient's lesion area using a multi-parameter MRI sequence (including multiple imaging modalities such as T2W1, DCE-MRI and DWI).

[0066] 2. Image Preprocessing Module

[0067] The image preprocessing module is used to perform preprocessing operations on each acquired multi-parameter MRI image data, and the preprocessing operations include: heterogeneous image registration, image denoising, image cropping, image standardization and image enhancement.

[0068] The present invention first uses a rigid registration algorithm based on standard mutual information to align DCE-MRI with T2W1 and DWI. Then, the N4BiasFieldCorrection method is used to correct the influence of the bias field during the MRI scan to complete image denoising. At the same time, a 3D region of interest block with a voxel size of W×H×D is obtained around the center of the lesion area. Each modality image of the regional block is first normalized, and then the grayscale value range of each pixel is normalized. Finally, random affine transformation, horizontal flip, vertical flip, and random image rotation (range 10°) are used for image enhancement.

[0069] Specifically, the image preprocessing module includes:

[0070] 1. Heterogeneous image registration unit: Because the three modalities of each multi-parameter MRI image data obtained, T2W1, DCE-MRI, and DWI, have different information such as image dimension, voxel spacing, and origin, a rigid body registration algorithm based on standard mutual information is used. DCE-MRI is used as a fixed image, and T2W1 and DWI are used as floating images. Rigid body transformation is performed to make the points corresponding to the same spatial position in the two images correspond one-to-one, thereby achieving the purpose of image registration.

[0071] Normalization Mutual Information (NMI) uses statistical data on image grayscale values to form a grayscale probability function for a single image and a joint probability function for the grayscale values of similar parts of two images to measure the degree of correlation between the two images. The formula for calculating normalization mutual information is as follows:

[0072]

[0073] Where H(R) is the entropy of a single image, and H(R,F) is the joint entropy of image R and image F.

[0074] The rigid body registration algorithm refers to the process of transformation involving only translation and rotation, and the interpolation method is linear interpolation. This process uses the greedy open source tool.

[0075] 2. Image denoising unit: Use the N4BiasFieldCorrection method to correct the influence of the bias field during the MRI scanning process. The bias field is an undesirable low-frequency intensity uneven signal present in MRI image data, which will destroy the MRI image. This process uses the N4 bias field correction tool in the Python toolkit SimpleITK.

[0076] 3. Image cropping unit: By removing tissue outside the breast tissue, the interference of irrelevant information on subsequent processes can be significantly reduced. A 3D volume of interest (VOI) with a voxel size of W × H × D is obtained by cropping around the center of the lesion area in the image, and a fixed input VOI size of 128 × 128 × 48 is obtained.

[0077] 4. Image normalization unit: Since the grayscale values of each modality to be processed are quite different, each modality image is first normalized using the Z-score method. The calculation formula is as follows:

[0078]

[0079] Where I represents the original image, Mean(I) represents the average pixel value of the original image, SD(I) represents the standard deviation of the pixel values of the original image, and O represents the normalized image.

[0080] Then the grayscale value range of each pixel is normalized and mapped to a floating point number between [0, 1].

[0081] 5. Image enhancement unit: Image enhancement is performed using random affine transformation, horizontal flip, vertical flip, and slight random rotation of the image (range 10°).

[0082] 3. Lesion Area Segmentation Module

[0083] The lesion region segmentation module is used to construct and train a lesion region segmentation model using preprocessed multi-parameter MRI sample image data, and to segment the lesion region based on the lesion segmentation model.

[0084] 1. Acquisition of multi-parameter MRI sample image data.

[0085] Obtain breast images of the patient's lesion area using a multi-parameter MRI sequence (including multiple imaging modalities such as T2W1, DCE-MRI, and DWI) and the corresponding efficacy category label (whether the pathology is completely relieved), and then perform data selection, data labeling, and data organization.

[0086] The present invention first exports the patient's multi-parameter MRI sequence (including multiple imaging modalities such as T2W1, DCE-MRI, and DWI) image data from the imaging device database as raw data. The corresponding pathological diagnosis results of NAC efficacy after this raw data are simultaneously obtained as the gold standard. Exclusion criteria are then formulated according to the standard NAC process to screen for high-quality data that meets the requirements, completing the data selection process. During the data annotation process, pixel-level annotation of the lesion area is performed on the DCE-MRI images of the raw data using commonly used medical annotation software. Finally, during the data collation process, the raw data is formatted, and the directory and file names are reorganized by patient.

[0087] Specifically, the image acquisition module includes:

[0088] 1. Data acquisition unit: Raw imaging data and corresponding NAC efficacy category labels are obtained from the database on a patient-by-patient basis from cooperating hospitals. The imaging data primarily includes breast images from multi-parameter MRI sequences (T2W1, DCE-MRI, and DWI modalities). Efficacy category diagnosis is achieved by further classifying the patient's pre- and post-NAC pathological diagnosis results into two categories: pathological complete response (pCR) and pathological incomplete response (non-pCR).

[0089] 2. Data selection unit: Based on the problem of NAC efficacy prediction concerned in the present invention, data selection is first performed according to the following exclusion criteria: patients who did not undergo bilateral breast MRI examination before treatment, patients who did not complete a full cycle of NAC or discontinued treatment for some reason, patients who did not undergo surgery after NAC, patients with incomplete postoperative pathological evaluation, and patients with distant metastasis.

[0090] In one example, according to the inclusion and exclusion criteria, 308 female breast cancer patients who visited a hospital unit from January to December 2018 were retrospectively collected in the experiment. The patients were aged 18 to 77 years, with an average of 50.0±10.2, including 98 pCR tumor cases and 210 non-pCR tumor cases.

[0091] 3. Data annotation unit: In order to accurately obtain the patient's lesion area, the annotation results of three professional physicians on the medical annotation software ITK-SNAP are combined to obtain pixel-level annotation of the lesion area.

[0092] 4. Data collation unit: Data integration is performed on a patient-by-patient basis, and the DICOM file format of the original data is converted into a NIfTI file.

[0093] In one example, the data integration is to name the directory according to the patient number (eg, Breast_Training_001), and to name the file according to the modality of the multi-parameter MRI data (eg, Breast_Training_001_DWI.nii).

[0094] 2. Lesion area segmentation

[0095] The present invention uses preprocessed 3D multi-parameter MRI image data as training data and the pixel-level annotation corresponding to the lesion area as the training target. It constructs a semantic segmentation network based on 3D-UNet, calculates the loss function according to the segmentation results of the model and the training target, and iteratively trains the model until the loss function converges or reaches the maximum number of iterations. The optimal model is saved as the lesion area segmentation model.

[0096] Specifically, the lesion area segmentation module includes:

[0097] S301: The pre-processed VOIs of the three modalities, each with a size of 128×128×48, are used as a channel of the input image. Therefore, the input of the segmentation network is 3×128×128×48. The labeled mask segmentation images are used as training targets to construct a semantic segmentation network. The backbone of the network is the well-known 3D-UNet network, which consists of four 3D convolutional layers (filter sizes [32, 64, 128, 256, 512]) and three pooling layers in the downsampling path, and three 3D convolutional layers (filter sizes [512, 256, 128, 64]) and three upsampling layers in the upsampling path, as well as skip connections.

[0098] S302: The present invention makes a minor adjustment to the submodule in the downsampling path, replacing it with a submodule structure with residual connections, which deepens the network while avoiding overfitting. During the training process, Dice Loss is used, and the calculation formula of Dice Loss is:

[0099] Dice loss = 1-Dice

[0100]

[0101] Among them, y true represents the actual segmentation label, i.e. the gold standard drawn manually, y pred Represents the predicted output graph of the model. The smooth term is added to prevent the denominator from being zero.

[0102] In one example, the 3D-UNet semantic segmentation network uses the IoU performance evaluation index, an initial learning rate of 0.001, a callback function to dynamically adjust the learning rate, and a decay coefficient of 0.5. The total number of training cycles is 300, and the training batch size is 16.

[0103] IV. Efficacy Diagnosis Module

[0104] The present invention proposes a multi-feature fusion network to fuse features from imaging omics and features from convolutional neural networks. The first branch first inputs the traditional radiomic features extracted from the image into an autoencoder to reconstruct the features. The second branch directly inputs the image into the convolutional neural network to extract deep features. Finally, these two feature vectors are spliced together to perform multi-feature fusion.

[0105] The present invention also proposes a multi-instance learning network based on multi-parameter MRI sequences. First, the three modal data of T2W1, DCE-MRI and DWI are respectively input as single instances into the multi-feature fusion network described in step 4 for feature extraction and fusion. Then, the feature instances corresponding to the three modalities are input into a multi-instance learning network to aggregate the tumor lesion information of all modalities of the patient for efficacy prediction.

[0106] 1. Multi-feature fusion

[0107] S401: Construct a multi-feature fusion network to fuse the features from radiomics and the features from convolutional neural networks. The specific method of the multi-feature fusion network is as follows: Figure 2 As shown in the figure, the network inputs are the VOIs and mask segmentation images of the three modalities. The VOI size is fixed to 128×128×48 after cropping. Because the DCE-MRI modality includes five time-series images from stages one to five, the images of stages one, three, and five are used as the image input channels, so the DCE-MRI input is 3×128×128×48. The T2W1 and DWI modalities are single-channel, with a size of 1×128×128×48. To ensure the consistency of the multi-feature fusion network structure, the T2W1 and DWI modalities are converted from single-channel to multi-channel. The single-channel image is copied three times to obtain an input of 3×128×128×48.

[0108] S402: Before building a multi-feature fusion network, it is necessary to use open source imaging omics analysis software (Pyradiomics) to extract radiomic features within the lesion area and save the extracted features in the form of an Excel file.

[0109] Specifically, feature extraction in Pyradiomics mainly includes four types of features: 1) Morphological features: describing features such as the location, shape, and size of the region of interest (ROI). 2) First-order grayscale histogram features: obtaining relevant statistical features by calculating the frequency distribution of different gray levels in the ROI. 3) Second-order and high-order texture features: describing the grayscale value distribution of pixels and their neighborhoods in the image to characterize texture features. 4) Filter-based features: The Laplacian of Gaussian (LoG) operator is often used for edge detection processing of original images. Different filter widths are used to highlight the texture features of the image based on the use of Gaussian function to reduce noise.

[0110] The aforementioned features were extracted for each case's multimodal MRI images and segmented images. A total of 293 initial features were extracted for each modality of each case. Feature dimensionality reduction was then performed to obtain the final 100 features. The feature importance was scored using the random forest algorithm in machine learning. The top 100 features, ranked by score, were selected as the initial radiomic features. These 100 features were subsequently extracted for each image during testing.

[0111] S403: The multi-feature fusion network is generally divided into two parallel branches. The first branch is through the autoencoder network f ae Reconstruct radiomic features from radiomics. The autoencoder network f ae The function of is to characterize the initial radiation features by taking the initial radiation features as the learning target. ae It consists of 4 layers of encoders and 4 layers of decoders, such as Figure 2 As shown, each layer is a fully connected layer with ReLU activation.

[0112] In order to fuse multiple features, the autoencoder network f ae The last layer of encoder results are input to 3 fully connected layers, and the output result is the feature vector m bi .

[0113] Specifically, in the process of training the autoencoder network of the first branch, the mean square error loss (MSE Loss) is used. The calculation formula of the MSE Loss is:

[0114]

[0115] Among them, x recon is the image feature reconstructed by the autoencoder, x is the original feature, and N is the total number of training samples.

[0116] S404: The multi-feature fusion network is generally divided into two parallel branches. The second branch uses the convolutional neural network f cnn As a feature extractor to extract deep features. The basic network framework is ResNet50, such as Figure 2 As shown in Figure 3, the network consists of 49 convolutional layers, 1 fully connected layer, and skip connections within the residual blocks.

[0117] Specifically, this step is done by f cnn Feature extraction obtains deep high-level feature vector n bi =f cnn (x bi ), by splicing it with the traditional radiation feature vector m obtained in step S403 bi Perform multi-feature fusion to obtain the fused feature vector z bi =(m bi ,n bi ).

[0118] 2. Prediction and diagnosis of therapeutic efficacy.

[0119] S405: Construct a multi-instance learning network based on multi-parameter MRI sequences. The specific method of the multi-instance learning network is as follows: Figure 3 shown.

[0120] Specifically, multiple instance learning (MIL), a weakly supervised learning method, has recently been applied to deep learning, particularly in pathology. Here, the present invention considers pCR efficacy prediction as an MIL problem. This work differs from previous work in that, in multi-instance learning, the present invention uses three modalities—T2W1, DCE-MRI, and DWI—in multi-parametric MRI data as "instances," and multi-parametric MRI data from a single case as a "bag." By training a multi-instance learning network, the present invention aggregates multimodal information from all lesions in the patient for efficacy prediction.

[0121] like Figure 3 As shown, the present invention converts the multi-modal data {x b1 ,x b2 ,x b3} is defined as an instance, and multi-parameter MRI data is defined as package X b , the label of the package is Y b ∈(0,1). First, the three modal data of T2W1, DCE-MRI and DWI are input as single instances into the multi-feature fusion network described in step 4 for feature extraction and fusion, and then the feature instance set Z corresponding to the three modalities is obtained. b ={z b1 ,z b2 ,z b3}, where z bi The size of the feature vector is M.

[0122] In order to assign different weights to the feature vectors corresponding to each modality, the present invention adds an attention layer f in the multi-instance learning network. att ,This attention layer consists of two layers of neural networks. Attention layer f att is defined as each eigenvector Z bi Assign an attention weight α bi , weight α bi The sum is equal to 1, and the weight is determined by the model, where the weight α bi The formula is as follows:

[0123]

[0124] Where w is of L×1 dimension and V is of L×M dimension, both of which are trainable parameters. M represents the size of the feature vector, L=50.

[0125] S406: Based on the weight α obtained in the previous step bi Through the classifier g c (i.e., an average pooling layer and a fully connected layer with a sigmoid function) to predict the pCR efficacy label:

[0126] p(Y b |X b )=g c (f att (Z b ))

[0127] S407: In the process of training the multi-instance learning network, the cross entropy loss (CE Loss) is used. The calculation formula of the CE Loss is:

[0128]

[0129] S408: Use joint loss Loss to autoencoder f ae , feature extractor f cnn , attention layer f att and classifier g c Perform joint training until convergence.

[0130]

[0131] where α is the reconstruction error L MSE and the prediction error L CE In our experiments, α=1 provides robust results.

[0132] S409: During model training, the Adam optimizer is used to optimize the model parameters. The initial learning rate is set to 0.001 and is halved every 2000 iterations. The maximum number of training epochs is 300, and the batch size is 16.

[0133] Experimental verification

[0134] The model of the present invention uses the area under the receiver operating characteristic (ROC) curve (AUC) and accuracy (Acc) as evaluation metrics, and predicts pCR efficacy through five replicates of three-fold cross-validation. The extracted radiomic features were trained using two known machine learning algorithms: support vector machine (SVM) and random forest (RF). Table 1 compares the classification performance of the present invention and various other models:

[0135] Network Model AUC (95% CI) Accuracy (%) Support Vector Machine 0.70 0.729 RF 0.725 0.74 3D-ResNet 0.62 0.64 The present invention 0.856 0.87

[0136] Table 1 Comparison of classification performance of the present invention and other models

[0137] See Figure 4 This embodiment provides a breast cancer neoadjuvant chemotherapy efficacy prediction system based on multi-feature fusion. The main interface of the software is shown in Figure 5 , which includes:

[0138] The first module, the image reading and saving module, mainly provides the functions of loading MRI image data, displaying images and saving images. On this basis, it expands the advanced functions of viewing images such as three-view display, slice conversion, coordinate display, etc.

[0139] The second module, the image preprocessing module, mainly provides preprocessing operations such as image denoising, image standardization, and image cropping for the read MRI image data.

[0140] The third module, the lesion area segmentation module, integrates the lesion area segmentation model trained during the experiment into the system, which can achieve high-precision segmentation results for the lesion area.

[0141] The fourth module, the efficacy diagnosis module, integrates the efficacy prediction model based on multi-instance learning trained during the experiment into the system, which can achieve accurate NAC efficacy prediction for the lesion area.

[0142] The following combination Figure 6 This paper introduces a method for predicting the efficacy of NAC in breast cancer lesions using a breast cancer neoadjuvant chemotherapy efficacy prediction system, which includes the following steps:

[0143] S1: Click [Load MRI Image Data] and select the breast MRI image to be displayed. The supported image data format is DICOM.

[0144] S2: After loading is complete, the DICOM sequence information will be displayed in the [Sequence Information Display Area] in the upper right corner, and the MRI image will be displayed in the [Image Data Display Area] below in the axial, sagittal, and coronal views. Other image viewing functions such as slice conversion and coordinate display are also supported.

[0145] S3: Determine whether the read image needs preprocessing, including image denoising, image standardization, image cropping, etc. If necessary, call step S4 and select the preprocessing method provided by the software; otherwise, go to step S5.

[0146] S4: According to the preprocessing method selected by the user, the corresponding interface is called to complete the operation and display it.

[0147] S5: Click [NAC Efficacy Prediction], and the software background will read the image and input it into the lesion area segmentation model to complete the lesion area segmentation.

[0148] S6: The background inputs the obtained lesion area and image into the efficacy prediction model based on multi-instance learning to extract multiple features and fuse them to generate the final efficacy prediction result.

[0149] S7: Output and display the final diagnosis results, click [Image Save], select the image view to be saved (axial, sagittal, coronal), and the selected image view in the current display area will be saved in .jpg format and stored in the system.

[0150] The above-described embodiments merely illustrate the implementation methods of the present invention, and their descriptions are relatively specific, but they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be based on the appended claims.

Claims

1. A device for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on multi-feature fusion, comprising: An image acquisition module is used to acquire a multimodal breast image of the patient's lesion area, wherein the modalities of the multimodal breast image include: T2W1 modality, DCE-MRI modality and DWI modality; An image preprocessing module is used to crop each modality breast image to obtain a 3D region of interest of the modality breast image; The lesion region segmentation module is used to obtain the lesion region based on the 3D region of interest of each modality breast image; wherein, obtaining the lesion region based on the 3D region of interest of each modality breast image includes: Construct a semantic segmentation network based on 3D-UNet, where the structure of the semantic segmentation network is P layers of 3D convolutional layers and P′ sub-pooling layers in the downsampling path, and Q layers of 3D convolutional layers and Q′ sub-pooling layers in the upsampling path. The submodules in the downsampling path are replaced with submodules with residual connections. Obtain multimodal sample breast images, use the sample breast images of T2W1 modality, DCE-MRI modality, and DWI modality as the input images of the semantic segmentation network, and train the semantic segmentation network with the labeled mask segmentation image as the training target to obtain a semantic segmentation model, wherein the loss function of the training process is Dice Loss = 1-Dice, y true represents the actual segmentation label, y pred Represents the predicted output graph, and smooth represents the smoothing term; Using breast images of T2W1 modality, DCE-MRI modality, and DWI modality as input images of the semantic segmentation model to obtain the lesion area; an efficacy diagnosis module, configured to calculate and stitch the radiation feature vectors of the lesion area with the deep advanced features of each 3D region of interest, and predict the efficacy of neoadjuvant chemotherapy for breast cancer based on the stitched vectors of breast images of each modality; wherein the efficacy diagnosis module comprises: an initial radiation feature extraction unit, a radiation feature calculation unit, a deep advanced feature extraction unit, a vector stitching unit, and a prediction unit; The initial radiomics feature extraction unit is used to select the radiomics features to be extracted and extract the radiomics features of the lesion area using open source radiomics analysis software to obtain the initial radiomics features; The radiation feature calculation unit is used to input the autoencoder network encoding results of the initial radiation features into several fully connected layers to obtain the radiation features in the lesion area; The deep high-level feature extraction unit is configured to input the 3D region of interest of each modality breast image into a convolutional neural network based on ResNet50 to obtain the deep high-level features of the modality; wherein the inputting the 3D region of interest of each modality breast image into the convolutional neural network based on ResNet50 to obtain the deep high-level features of the modality includes: For breast images of the DCE-MRI modality, the images of stage one, stage three, and stage five are taken as the input images of the convolutional neural network to obtain deep and high-level features of the DCE-MRI modality; and, For breast images in the T2W1 modality, the single-channel image is copied three times and used as the input image of the convolutional neural network to obtain deep and advanced features of the T2W1 modality; and, For breast images of the DWI modality, the single-channel image is copied three times and used as the input image of the convolutional neural network to obtain deep and advanced features of the DWI modality; The vector splicing unit is used to splice the radiological features with the deep high-level features of each modality to obtain a splicing vector of the corresponding modality breast image; The prediction unit is configured to input the splicing vectors of the breast images of each modality into a multi-instance learning network, so that an attention layer in the multi-instance learning network assigns a weight to the splicing vector of each modality of the breast images, and a classifier in the multi-instance learning network obtains a predicted efficacy of neoadjuvant chemotherapy for breast cancer based on a weighted summation result of the splicing vectors of the breast images of each modality; Among them, the loss of training the efficacy diagnosis module Loss = L MSE +αL CE , where L MSE represents the reconstruction loss for training the autoencoder network, L CE Describes the prediction loss of training the multi-instance learning network, and α represents the reconstruction loss L MSE and prediction loss L CE Hyperparameters that balance between.

2. The device according to claim 1, wherein The image preprocessing module is further used for: Based on a heterogeneous image registration unit, performing heterogeneous image registration on the multimodal breast images; Performing image denoising on the multimodal breast image based on an image denoising unit; performing image standardization on the multimodal breast image based on an image standardization unit; Based on the image enhancement unit, image enhancement is performed on the multimodal breast image.

3. The device according to claim 2, wherein The heterogeneous image registration unit is configured to: A rigid body registration algorithm based on standard mutual information is used, with the image of the DCE-MRI modality as a fixed image, and the images of the T2W1 modality and the DWI modality as floating images, respectively, for rigid body transformation, so that points corresponding to the same position in space in the image correspond one to one.

4. The device according to claim 2, wherein The image denoising unit is used to: The image normalization unit is used to: The images of the T2W1 modality, the DCE-MRI modality, and the DWI modality are respectively normalized using a Z-score method; Normalize the grayscale value range of each pixel in the normalized image.

5. The device according to claim 1, wherein The obtaining of multimodal sample breast images includes: A data acquisition unit is used to obtain original imaging data and corresponding neoadjuvant chemotherapy efficacy category labels from a database on a patient-by-patient basis, wherein the neoadjuvant chemotherapy efficacy category labels include: pathological complete remission and pathological incomplete remission; a data selection unit, configured to set exclusion criteria based on the efficacy prediction problem to select data, wherein the exclusion criteria include: patients who did not undergo bilateral breast MRI examination before treatment, patients who did not complete a full cycle of neoadjuvant chemotherapy or discontinued treatment for some reason, patients who did not undergo surgery after neoadjuvant chemotherapy, patients with incomplete postoperative pathological evaluation, and patients with distant metastasis; The data annotation unit is used to obtain pixel-level annotations of the lesion area based on the annotation results of professional physicians on the medical annotation software ITK-SNAP; The data collation unit is used to integrate data based on patients and convert the DICOM file format of the original data into NIfTI files.

6. The device according to claim 1, wherein Initial radiation feature extraction unit, used for: For the sample lesion area, open source radiomics analysis software was used to obtain all radiomic features, including morphological features, first-order grayscale histogram features, second-order and higher-order texture features, and filter-based features; The importance of each radiological feature is scored using the random forest algorithm to determine the radiological features to be extracted; Select the radiological features to be extracted.

Images