Deep learning of pretreatment quantitative multiparameter ultrasound images to predict breast cancer response to chemotherapy

Abstract

A system for predicting breast cancer response to neoadjuvant chemotherapy (NAC) using quantitative ultrasound (QUS) parameter images and / or B-mode images, the system comprising: an imaging system for acquiring at least one ultrasound data frame including raw RF signals and / or images; and a computer system for receiving the at least one ultrasound data frame and identifying a region of interest (ROI) in each of the at least one ultrasound data frame using one or more predetermined rules; generating at least one quantitative ultrasound (QUS) parameter map and / or processed B-mode image of the tumor; extracting an optimal feature map from the QUS parameter images and / or B-mode image using a feature network of a machine learning architecture; and classifying tumor subjects into responders or non-responders to NAC.

Description

[Technical field]

[0001] The present disclosure relates to a novel deep learning-based method for predicting breast cancer response to neoadjuvant chemotherapy (NAC) using pre-treatment quantitative ultrasound (QUS) multi-parameter imaging. [Background technology]

[0002] Breast cancer is the most common type of cancer and the leading cause of cancer-related deaths in women [1], [2]. In 2020, approximately 2.3 million cases of breast cancer were diagnosed worldwide, resulting in approximately 700,000 women's deaths [1]. Locally advanced breast cancer (LABC) is an aggressive subtype of breast cancer that comprises up to 20% of new cases each year [3]. LABC is often identified with tumors greater than 5 cm in size, occasionally with involvement of the skin and / or chest wall. Additionally, LABC includes patients diagnosed with inflammatory breast cancer or multiple positive axillary lymph nodes [3][4].

[0003] Patients diagnosed with LABC suffer a high risk of relapse and metastasis with a local recurrence rate of about 48% at 5 years [5]. With different systematic and targeted regimens available, neoadjuvant chemotherapy (NAC) followed by surgery is currently considered the standard treatment for patients with LABC [6]-[9]. In some cases, surgery is performed after adjuvant radiotherapy and / or hormonal therapy to reduce the risk of cancer recurrence [4][6]. Although response to NAC demonstrated a high correlation with patient survival, complete pathological response is limited to less than 30% of patients, and about 30% of patients do not respond even partially to NAC [3], [4], [6],

[10] -

[15] . To determine tumor pathological response to NAC, postoperative histopathology is considered the standard technique [6]-[9]. However, postoperative evaluation cannot be used to adjust NAC or switch to salvage treatment.

[0004] Currently, monitoring of tumor response to NAC relies largely on physical examination or standard anatomical imaging to assess changes in tumor size. The main limitation of these methods is that detectable changes in tumor dimensions are typically evident several months after treatment, and in some cases, measurable changes may not be evident on imaging, regardless of the pathological response to NAC

[16] . Early prediction of tumor response to NAC would allow for treatment adjustments by altering the regimen, dose, and / or sequence of treatment options, switching to more effective or even salvage treatments, before it is potentially too late for individual patients

[17] ,

[18] . A personalized strategy of LABC treatment is expected to improve tumor response to neoadjuvant treatment, spare patients from unnecessary side effects of ineffective treatment, and improve patients' overall survival and quality of life.

[0005] Ultrasound is a portable, rapid, and cost-effective imaging modality that can be applied to characterize the physical properties of tissues without the injection of exogenous contrast agents. In particular, quantitative ultrasound (QUS) techniques have been introduced to derive quantitative measures of tissue biophysical properties independent of instrument settings with a low level of operator dependency

[19] . Quantitative ultrasound spectral techniques examine the frequency dependence of ultrasound radio frequency (RF) signals backscattered from underlying tissues, which can be used to characterize tissue microstructure

[19] . QUS parameters derived from the analysis of the normalized power spectrum of RF signals, including midband fit (MBF), spectral slope (SS), spectral 0 MHz interference (SI), effective scattering diameter (ESD), and effective acoustic concentration (EAC), have shown promise in the detection and characterization of malignancies, examination of liver tissue, and detection of cardiovascular diseases

[20] -

[26] .

[0006] It has been shown that QUS spectral parameters can detect tumor cell death resulting from various anticancer drug treatments

[27] -

[30] . Also, several studies have demonstrated that hand-crafted features derived from QUS parameter maps can be used to predict and monitor breast cancer response to neoadjuvant chemotherapy before or within a few weeks of treatment initiation, with high correlation to clinical and pathological responses identified at the end of treatment

[31] -

[34] . For example, it has been demonstrated that text features of QUS spectral parameter maps have a higher correlation to histological tumor cell death in response to chemotherapy compared to QUS mean parameters

[35] . Furthermore, several studies have revealed the potential of text features of QUS parameter images to predict LABC tumor response to NAC as early as one week after the initiation of treatment

[36] -

[38] . In a recent study, Tadayyon et al. demonstrated that hand-crafted QUS features derived from both tumor nuclei and their boundaries can improve the performance of tumor response prediction before the initiation of treatment

[39] .

[0007] Deep learning techniques have been recently explored in various applications of medical image analysis

[40] ,

[41] . Such methods can potentially remove the process of extracting carefully designed, handcrafted features from images, which is required for traditional machine learning techniques. Instead, deep learning frameworks optimize data-driven feature maps during an iterative training procedure

[42] . In this context, several studies have been conducted on adapting deep convolutional neural networks (DCNNs) for NAC treatment response prediction in breast cancer patients using magnetic resonance imaging (MRI)

[43] -

[45] . Furthermore, several studies have investigated the potential of DCNNs in analyzing ultrasound images of breast tumors for cancer classification. For example, Byra et al. demonstrated the potential of convolutional neural networks for breast lesion classification using Nakagami parameter images

[46] . None of the previous studies have investigated the effectiveness of deep learning techniques using QUS multi-parameter images for treatment response prediction. Summary of the Invention

[0008] In one embodiment of the present disclosure, there is provided a system for predicting breast cancer response to neoadjuvant chemotherapy (NAC) using quantitative ultrasound (QUS) parameter images and / or B-mode images, the system comprising: an imaging system for acquiring at least one frame of ultrasound data including raw RF signals and / or an image; 1. A computer system comprising a hardware processor and a memory device encoded with instructions, the instructions causing the hardware processor to: an operation of receiving at least one frame of ultrasound data and identifying a region of interest (ROI) in each of the at least one frame of ultrasound data using one or more predetermined rules, the ROI including a tumor, the ROI being identified optionally automatically; generating at least one quantitative ultrasound (QUS) parameter map and / or processed B-mode image of the tumor from the at least one frame of ultrasound data; extracting optimal feature maps from the QUS parameter images and / or B-mode images using a feature network of a machine learning architecture; inputting the optimal feature map into a predictive network of a machine learning architecture, where the optimized feature maps obtained from the feature network are combined or averaged across all images associated with each tumor, and the features and predictive network may be integrated in an end-to-end system; pre-processing and formatting the QUS parameter images and / or B-mode images to generate a training dataset for a deep learning model; training a feature network to generate an optimal feature map for a single QUS parameter image and / or B-mode image; For the end-to-end system described above, the method includes the steps of: training a predictive network using a combined or averaged feature vector associated with each patient in the training dataset, where the combined features and the predictive network are jointly trained; classifying the tumor subject as a responder or non-responder to NAC and classifying the response as at least one of a complete response, a partial response, stable disease, or progressive disease; A computer system for executing the A system is provided, comprising:

[0009] In another aspect of the present disclosure, there is provided a method for predicting response to neoadjuvant chemotherapy (NAC) using quantitative ultrasound (QUS) parametric images, the method comprising: acquiring at least one ultrasound frame of data and / or image comprising raw RF signals using an imaging system; A computer system having a hardware processor and a memory device having instructions encoded thereon, the computer system being configured to cause the hardware processor to: receiving at least one frame of ultrasound data and identifying a region of interest (ROI) in each of the at least one frame of ultrasound data using one or more predetermined rules, the ROI including a tumor, the ROI may be identified automatically; generating at least one quantitative ultrasound (QUS) parameter map and / or processed B-mode image of the tumor from the at least one frame of ultrasound data; extracting optimal feature maps from the QUS parameter images and / or B-mode images using a feature network of a machine learning architecture; inputting the optimal feature map into a predictive network of a machine learning architecture, where the optimized feature maps obtained from the feature network are combined or averaged across all images associated with each tumor, and the features and predictive network may be integrated in an end-to-end system; pre-processing and formatting the QUS parameter images and / or B-mode images to generate a training data set for the deep learning model; training a feature network to generate an optimal feature map for a single QUS parameter image and / or B-mode image; For the end-to-end system described above, the method includes the steps of: training a predictive network using a combined or averaged feature vector associated with each patient in the training dataset, where the combined features and the predictive network are jointly trained; classifying the tumor subject as a responder or non-responder to NAC and classifying the response as at least one of a complete response, a partial response, stable disease, or progressive disease; and A method is provided, comprising:

[0010] Advantageously, a novel deep learning-based method is provided for predicting breast cancer response to neoadjuvant chemotherapy (NAC) using pre-treatment quantitative ultrasound (QUS) multi-parameter imaging. The machine learning architecture includes at least one of a deep convolutional neural network (DCNN) architecture, a deep learning architecture, and a transformer architecture. [Brief description of the drawings]

[0011] [Figure 1] A high-level view of the overall system architecture 10 for predicting breast cancer response to neoadjuvant chemotherapy (NAC) using pre-treatment quantitative ultrasound (QUS) multi-parameter imaging is shown. [Diagram 2] FIG. 1 shows a flow chart outlining exemplary steps for breast cancer response to neoadjuvant chemotherapy (NAC) using pre-treatment quantitative ultrasound (QUS) multi-parameter imaging. [Figure 3a-3c] Fig. 3a shows a scheme of a deep learning framework for response prediction, demonstrating a feature and prediction network; Fig. 3b shows a scheme of a deep learning framework for response prediction, demonstrating a residual module; Fig. 3c shows a scheme of a deep learning framework for response prediction, demonstrating an attention module. [Figure 4a-4r] Figures 4a and 4b show ultrasound B-mode images. Figures 4c-f show parameter overlays of MBF on B-mode images acquired pre-treatment from representative responders and non-responders to NAC, with associated PDA maps visualizing the level of influence of various regions in each parameter image on the network's decision (model 4 in Table 2), with the tumor nucleus outlined by a white dashed line. Figures 4g-j show parameter overlays of SI on B-mode images acquired pre-treatment from representative responders and non-responders to NAC, with associated PDA maps visualizing the level of influence of various regions in each parameter image on the network's decision (model 4 in Table 2), with the tumor nucleus outlined by a white dashed line. Figures 4k-n show parameter overlays of ESD on B-mode images acquired pre-treatment from representative responders and non-responders to NAC, with associated PDA maps visualizing the level of influence of various regions in each parameter image on the network's decision (model 4 in Table 2), and tumor nuclei outlined by dashed white lines. Figures 4o-r show parameter overlays of AC on B-mode images acquired pre-treatment from representative responders and non-responders to NAC, with associated PDA maps visualizing the level of influence of various regions in each parameter image on the network's decision (model 4 in Table 2), and tumor nuclei outlined by dashed white lines. [Figure 5a-5b] Histopathological images of surgical specimens obtained from representative patients are shown. [Figure 6a-6d] ROC curves generated for responding and non-responding patients in the validation and independent test sets identified prior to treatment using predictive models 1 to 4 in Table 2 are shown. [Figure 7a-7e] The 10-year recurrence-free survival curves for responders and non-responders identified after treatment based on clinical and histological criteria, and before treatment using the four predictive models presented in Table 2 are shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The following detailed description will be made with reference to the accompanying drawings. In the drawings and the following description, the same reference numerals are used to refer to the same or similar elements wherever possible. Although embodiments according to the present disclosure may be described, modifications, adaptations, and other implementations are possible. For example, the elements illustrated in the drawings may be substituted, added, or modified, and the methods described herein may be modified to the disclosed methods by substituting, reordering, or adding steps. Therefore, the following detailed description is not intended to limit the present disclosure. The appropriate scope of the present disclosure is defined by the appended claims.

[0013] Moreover, the specific embodiment shown and described herein is an example of the present invention and is not intended to limit the scope of the present invention in any way. Indeed, for the sake of brevity, certain subcomponents of individual operating components and other functional aspects of the system may not be described in detail herein. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent example functional relationships and / or physical couplings between the various elements. It should be noted that many alternative forms or additional functional relationships or physical connections may exist in an actual system.

[0014] Referring to FIG. 1, a high-level view of an overall system architecture 10 for predicting breast cancer response to neoadjuvant chemotherapy (NAC) using pre-treatment quantitative ultrasound (QUS) multi-parameter imaging is shown. Images may be acquired from one or more imaging systems 14, which may include medical imaging equipment such as X-ray imaging systems, CT scan imaging systems, ultrasound imaging systems, MRI imaging systems, nuclear medicine imaging systems, etc. Images 12 captured by the one or more imaging systems 14 are rendered as digital representations and stored on a computing device 16. The computing device 16 may include one or more processing units, such as a graphics processing unit (GPU). In one example, the computing device 16 implements a model for predicting breast cancer response to neoadjuvant chemotherapy (NAC), and outputs the results via a graphical user interface 20 on a peripheral screen or a customized plug-in.

[0015] The computing device 16 includes an image repository 30 for storage of images 12. The image repository 30 may be a computer readable medium, such as a hard disk. Alternatively, the acquired images 14 may be stored on a storage server 18 or a computing cloud. The image repository 30 may also include images of patients for analysis and / or training images that have been previously analyzed and / or annotated.

[0016] The term "computing device" refers to data processing hardware and encompasses any kind of apparatus, device, and machine for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus can also be or further include dedicated logic circuitry, such as a central processing unit (CPU) 40, a GPU 41 (graphics processing unit), an FPGA (field programmable gate array), or an ASIC (application specific integrated circuit). In some embodiments, the data processing apparatus and / or the dedicated logic circuitry can be hardware-based and / or software-based. The apparatus can optionally include code that creates an execution environment for a computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof. Although a single CPU 40 is shown in FIG. 1, two or more processing units may be used according to the particular needs, desires, or one particular implementation of the computing device 16. Typically, the GPU 41 executes instructions and manipulates data to perform the operations of the computing device 16. For example, the GPU 41 is implemented to accelerate calculations related to deep learning methods.

[0017] The memory 42 stores data for the computing device 16 and / or other components of the system 10. Although a single memory 42 is shown in FIG. 1, two or more memories may be used according to the particular needs, desires, or a particular implementation of the computing device 16. Although the memory 42 is illustrated as an integral component of the computing device 16, in an alternative embodiment, the memory 42 may be external to the computing device 16 and / or the system. For example, the memory 42 may include computer-readable media (temporary or non-temporary, as appropriate) suitable for storing computer program instructions and data, including, by way of example, semiconductor memory devices, such as erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and all forms of non-volatile memory, media, and memory devices, including CD-ROMs, DVD+ / -R, DVD-RAM, DVD-ROM disks, and Blu-ray disks. Memory 42 can store various objects or data, including caches, classes, frameworks, applications, backup data, jobs, web pages, web page templates, database tables, repositories that store business and / or dynamic information, as well as any other suitable information, including any parameters, variables, algorithms, models, instructions, rules, constraints, or references thereto. In addition, memory may include any other suitable data, such as logs, policies, security or access data, report files, and others. Processor 40 and memory 42 can be supplemented by, or incorporated in, special purpose logic circuitry.

[0018] In one example, the applications in memory 42 include algorithmic instructions that provide functionality in accordance with the particular needs, requirements or one particular implementation of computing device 16, particularly with respect to functionality required to process modeling calculations for predicting breast cancer response to neoadjuvant chemotherapy (NAC).

[0019] Computing device 16 may include an input / output module 43 to which input devices, such as a keypad, keyboard, touch screen, microphone, voice recognition device, or other device capable of accepting user information, and / or output devices that communicate information associated with the operation of computing device 16, including digital data, visual and / or audio information, or GUI 20.

[0020] The computing device 16 includes interfaces as part of the I / O module 43 that are used according to the particular needs, requirements or one particular implementation of the computing device 16. The interfaces are used by the computing device 16 to communicate with other systems in a distributed environment connected to a network 44. In general, the interfaces include logic that is coded in software and / or hardware in an appropriate combination and operable to communicate with the network 44. More specifically, the interfaces may include software that supports one or more communication protocols associated with the communications. The various components of the computing system are connected by interconnection means 45, such as an address bus, a data bus, a control bus and a peripheral bus.

[0021] A client terminal 46 (e.g., a remotely located radiology workstation) can request services regarding breast cancer response to neoadjuvant chemotherapy (NAC) and access results via the communication network 44. Thus, the computing device 16 can provide software as a service (SaaS) to the client terminal 46, or provide applications to the client terminal 46 for local download and / or provide functionality to the client terminal 46 with a remote access session via a web browser or the like. Thus, the annotated medical data can be accessible to a physician on an on-demand basis from the client terminal 46. The system 10 can also include a data storage 47 configured to maintain one or more data sets, including data structures that store linkages and other data, such as medical images, libraries, models, and rules. The data storage 47 can be a relational database, a flat data storage, a non-relational database, among others. EXAMPLES

[0022] In one study, the efficacy of the DCNN method on QUS spectral multi-parameter images was investigated to predict LABC response to NAC before the start of treatment. QUS spectral parametric images were generated using ultrasound data acquired from 181 LABC patients before treatment. Patients' responses to NAC were identified after surgery using standard clinical and pathological criteria and used as ground truth to evaluate the performance of the predictive model. The dataset was randomly split into a training set and an independent test set. Different DCNN architectures, including RAN

[47] and ResNet

[48] , were investigated for feature extraction in the deployed framework. In the experimental set, feature maps were extracted from the tumor nucleus as well as from the nucleus and its boundary. After averaging the features in different tumor cross sections, a fully connected network was utilized for response prediction.

[0023] [Research Protocol] The study was conducted in accordance with the guidelines and regulations as approved by the Institutional Research Ethics Board at Sunnybrook Health Sciences Centre (SHSC), Toronto, Canada. The study was open to all women aged 18–85 years who were diagnosed with LABC and were planned for NAC followed by surgery. One hundred and eighty-one eligible patients were recruited into the study after obtaining written informed consent. A core needle biopsy was performed for each patient to confirm the cancer diagnosis and grade the tumor. For each patient, the initial tumor size was determined using magnetic resonance (MR) images of the affected breast. Before the initiation of NAC, ultrasound B-mode images and radio frequency (RF) data were obtained from the patients (supine position with arms above the head) following a standardized protocol. Three experienced sonographers were responsible for acquiring the ultrasound data. For NAC, 62.9% of patients received doxorubicin, cyclophosphamide, followed by paclitaxel / docetaxel (AC-T / D), 32.6% were treated with 5-fluorouracil epirubicin, cyclophosphamide, followed by docetaxel (FEC-D), and 4.5% were treated with paclitaxel and cyclophosphamide (TC). Patients with HER2+ tumors also received trastuzumab. Patients were followed up for up to 10 years after treatment, and their clinical data were recorded for survival analysis. Of the 181 patients, approximately 30% (n=50) were randomly selected through stratified sampling and kept unobserved as an independent test set, and the remaining patients (n=131) were considered as a training set and used to develop and optimize the predictive model.

[0024] [Clinical and pathological response evaluation] All patients underwent breast surgery after completion of neoadjuvant chemotherapy, according to institutional guidelines. Before surgery, the size of the residual tumor was determined using MRI. Standard histopathology was performed on surgical specimens to evaluate the pathological response of the tumor to NAC. Specimens were stained with hematoxylin and eosin (H&E) and prepared in whole-mount 5'' × 7'' pathology slides, when possible. Mounted slides were digitized using a confocal scanner (TISSUEscope, Huron Technologies, Waterloo, Ontario). A board-certified pathologist who was kept blinded to the study outcomes examined all pathology specimens. A modified response (MR) grading system based on Response Evaluation Criteria in Solid Tumors (RECIST)

[49] and histopathological criteria

[39]

[50] was used to classify patients into two groups, responders and non-responders, as previously described

[51] . In the MR grading system, the MR score is defined as follows: MR1: no reduction in tumor size, MR2: less than 30% reduction in tumor size, MR3: 30%-90% reduction in tumor size or histopathological determination of very low residual tumor cellularity, MR4: more than 90% reduction in tumor, MR5: no evident tumor and no identifiable malignant cells in sections from the site of the tumor (ductal intraepithelial carcinoma may be present). In this study, patients with MR scores of 1-2 (less than 30% reduction in tumor size) were considered non-responders, and patients with MR scores of 3-5 (more than 30% reduction in tumor size or very low residual tumor cellularity) were determined as responders. Accordingly, 138 and 43 patients were identified as responders and non-responders, respectively.

[0025] FIG. 2 shows a flow chart 200 outlining exemplary steps for predicting breast cancer response to neoadjuvant chemotherapy (NAC) using pre-treatment quantitative ultrasound (QUS) multi-parameter imaging.

[0026] [Data Acquisition] In step 202, ultrasound RF data was acquired using the imaging system 14. For example, the imaging system 14 is an RF enabled Sonix RP System™ (Ultrasonix™, Vancouver, Canada) and an L14-5 / 60 transducer. In one example, the transducer was operated at a center frequency of about 6 MHz with a -6 dB bandwidth of 3-8 MHz, and the RF data was acquired using a sampling frequency of 40 MHz and digitized at 16-bit resolution. For each tumor, ultrasound data was acquired in 4-7 image planes across the breast, spaced approximately 1 cm apart. The focal depth was set to the center of the tumor depending on the individual patient situation. The breast area for ultrasound scanning may be specified by the oncologist, who determined the acquisition scan planes by physical examination of the patient (step 204). In one example, the image sizes along the lateral and axial directions were 6 cm and 4-6 cm, respectively.

[0027] [Generate QUS parameter map] In step 206, a quantitative ultrasound (QUS) technique was used to generate parameter maps for predicting and monitoring breast cancer response to neoadjuvant chemotherapy. Parameter images were generated and QUS spectral analysis was performed in conjunction with sliding window analysis (described below) to derive MBF, SI, ESD, and parameters

[23] ,

[24] . The average power spectrum was obtained by averaging over the Fourier transform of the Hanning-gated RF data calculated for all scan lines of the analysis window. The average power spectrum was normalized to remove the effects of the system transfer function and transducer beamforming using the reference phantom method

[52] ,

[53] . The reference phantom consisted of 5-30 μm diameter glass beads embedded in a uniform background of microscopic oil droplets in gelatin (Medical Physics Department, University of Wisconsin, USA). The reference phantom had an attenuation coefficient of 0.576 dB / MHz.cm and sound speed parameters of 1488 m / s. MBF and SI parameters were estimated using linear regression analysis within the -6 dB bandwidth of the transducer

[23] ,

[54] ,

[55] . ESD and EAC parameters were derived by fitting a spherical Gaussian form factor model to the estimated backscatter coefficients

[56] ,

[57] .

[0028] To generate QUS parameter maps for each tumor, tumor nuclei were manually contoured by trained staff under the supervision of an expert oncologist in each scanning plane using the associated B-mode images. Tumor border contours were automatically generated around the nuclei with a thickness of 5 mm based on the observations of a previous study

[39] . Then, using a sliding window analysis over the entire region of interest (tumor nuclei and borders), parameter maps were generated for all imaging planes of the tumor with a window size of 2 mm × 2 mm and an overlap of 95% in both the lateral and axial directions. The parameters calculated for each window were assigned to its center. The sliding window size was selected with an overlap size to obtain isotropic pixels such that the sliding window covered enough ultrasound wavelengths in the axial direction for spectral analysis while preserving texture in the generated parameter maps

[58]

[59] .

[0029] [Deep learning model] In step 208, the parameter maps were received by a deep learning framework for response prediction. An exemplary deep learning framework scheme is shown in FIG. 3a-3c. In an exemplary embodiment, the deep learning framework includes two cascaded networks including a residual network (ResNet) and a residual attention network (RAN) for extracting optimal feature maps from parameter images using a fully connected network for response prediction. The feature maps were derived from tumor nuclei only, as well as from nuclei and their boundaries. The first network is a deep convolutional neural network (DCNN) adapted to extract optimal feature maps from QUS parameter images, with several convolutional layers as the backbone, and is referred to as the feature network in this document. Two main architectures including a modified residual network version 101 (ResNet)

[48] and a modified residual attention network version 56 (RAN)

[47] were investigated as the backbone of the feature network in this study. 3a-3c show a fully connected layer after the convolutional layer (backbone) in the feature network, which is applied in training this network on a single parameter image to extract an optimal feature map for response prediction. Also shown in Fig. 3a-3c are adapted ResNet and RAN architectures with residual and attention modules. In the residual module applied in the ResNet architecture, the convolutional layer can be skipped through the identity branch. This strategy allows ultra-deep networks such as ResNet to avoid overfitting and achieve improved performance for unseen samples. In the attention module applied in the RAN architecture, the trunk branch determines the information that can be passed through the network, while the mask branch determines the amount of information from the trunk branch that should be passed through. Thus, the module can pass important information with higher weight and reduce the influence of less important information in the output of the network.

[0030] As shown in Figures 3a-c, the optimized feature maps obtained from the feature network are averaged over all parameter images associated with each tumor and then used in a second fully connected network (prediction network) adapted for response prediction at the patient level (described further below) (step 210). The prediction network consists of two fully connected layers with an input layer having the same size as the flattened feature vector (256), a central layer with 100 neurons, and a softmax layer at the end with an output size of 2, which predicts for each patient the probability of the response category (responder vs. non-responder). Dropout layers are added after the first and second layers of this network to avoid overfitting and improve its generalization performance.

[0031] [Preprocessing and model training] Before training the model, the parameter images were preprocessed and adjusted for the convolutional model (step 212). Approximately 25% of the training set (31 patients) was randomly selected as a validation set for optimizing the network hyperparameters. The parameter images were resampled to a size of 512x512 pixels. Then, the pixel values ​​in the parameter images of the training set were normalized to (0 1) to promote training convergence. A training set normalization parameter was used to normalize the validation and test sets. Data augmentation was applied to the training set to improve network training and avoid the problem of having a relatively small training data set. To augment the training data, horizontal flips and both horizontal and vertical shifts (maximum shift in image size: 30%) were stochastically applied.

[0032] Then, in step 214, a feature network was trained to generate an optimal feature map for a single QUS parameter image in the first step of model training. To train the feature network, the parameter images of the training set were fed into the network, while the different imaging planes of each tumor were considered as independent inputs with the tumor response as output. The optimal feature map of each imaging plane was obtained by feeding its corresponding parameter image into the trained feature network. For each patient, the optimal feature map was calculated for all 2D imaging planes of the tumor, which were flattened into a 1D vector, and then averaged over the entire tumor volume, resulting in an average feature vector with a size of 256×1, which was used in the prediction network. This strategy was applied to standardize the input size of the network for different tumors with various sizes and therefore different numbers of QUS parameter images. Then, in step 216, the prediction network was trained with the averaged feature vectors associated with the patients in the training set and evaluated over an independent test set for response prediction. To train the network, cross-entropy was used as the loss function, with a cost weight ratio of C:1 (C ≥ 1; optimized as described below) for non-responders versus responders to account for imbalances in the dataset. Network hyperparameters, including dropout rate (range: 0.3–0.7), hidden fully connected layer width (range: 50–300), learning rate (range: 0.1–0.00001), cost weight (range: 1 ≤ C ≤ 10) and batch size (range: 4–16), were optimized using the validation set. To select the network training optimizer between Adam and stochastic gradient descent (SGD) methods, preliminary experiments were conducted using the validation set, where the Adam optimizer was selected and applied

[60] . The optimal hyperparameters for training the model were as follows: dropout rate = 0.5, learning rate = 0.0001, cost weight = 5, batch size = 8.Early stopping was used to avoid overfitting by monitoring the network's performance on the validation set during the training process.

[0033] [Response prediction and risk assessment] In different experiments, QUS multi-parameter images of tumor nuclei (MBF, SI, ESD and EAC), as well as nuclei and their boundaries, were investigated when inputs to the DCNN framework and their performances were compared in response prediction. Deep learning models with different feature networks were trained and optimized using the training set. The performance of the optimized models was evaluated on an independent test set using accuracy, sensitivity, specificity and ROC analysis (step 218). In this study, sensitivity refers to the proportion of non-responders predicted as non-responders, and specificity refers to the proportion of responsive patients correctly predicted as responders by the model. Prediction Difference Analysis (PDA) was performed to visualize the importance of different regions of the input QUS parameter images to the network's decision

[61] . In each of the modified PDA procedures applied in this study, one small patch (8 × 8 pixels with 50% overlap between adjacent patches) of the input parameter images was occluded (pixel values ​​were set to zero). The absolute change in the model's predictions (output probabilities) was then calculated compared to the original parameter image input and considered as the influence of the occluded patch on the network's decisions (step 220). A PDA map was generated for each input parameter image by sliding the occluded patch across the image and assigning the estimated influence to the center.

[0034] The efficacy of the deployed predictive models in differentiating LABC patients with different recurrence-free survival was evaluated through Kaplan-Meier survival analysis. Survival curves were generated for responders and non-responders, identified based on the predictions of each model before treatment and based on clinical and histopathological criteria after treatment. The long rank test was applied to evaluate for statistically significant differences between the survival curves of the two responding cohorts obtained in each experiment.

[0035] [result] The clinical and histopathological characteristics of the patients involved are presented in Table 1. Patients had a mean initial tumor size of 5.2 cm and a mean residual tumor size of 2.5 cm at the end of treatment. Using the MR grading system, 76.2% and 23.8% of patients were identified as responders and non-responders, respectively, at the end of treatment.

[0036] [Table 1]

[0037] Figures 4a-r show QUS parameter maps of MBF, SI, ESD, and EAC superimposed on ultrasound B-mode images obtained from representative responsive and non-responsive patients, respectively. As observed in these representative images, the QUS parameter maps associated with responsive and non-responsive patients demonstrated different average and spatial patterns of pixel values ​​within the tumor nuclei and borders. Also shown in the figures are PDA maps associated with these parameter images that visualize the relative influence of different regions within each image on the network's decision for response prediction. Figures 5a and 5b show H&E stained histopathology images of surgical specimens obtained from representative responsive and non-responsive patients. In the responsive patients, minimal tumor cellularity remained within the tumor bed after chemotherapy, as evident in the histopathology slides. In contrast, the histopathology images of non-responsive patients typically showed large residual disease areas with minimal chemotherapy effect.

[0038] Table 2 shows the response prediction results in different experiments for the validation and independent test sets. Figures 6a-d show the ROC associated with the validation and test sets for different prediction models. Using a ResNet architecture as the backbone of the model to extract feature maps from parameter images of tumor nuclei, an AUC of 0.77 was obtained on the independent test set. Extending the input parameter images to include tumor nuclei and their boundaries improved the AUC of this model to 0.83.

[0039] [Table 2]

[0040] Applying features extracted from parameter images of tumor nuclei using the RAN architecture resulted in an accuracy of 80% and an AUC of 0.82 on the independent test set. Similar to the model with the ResNet architecture as the feature extractor, the overall performance of this model was improved by extending the input parameter image to include the tumor boundary. Notably, this model resulted in the best predictive performance on the independent test set, with an accuracy and AUC of 88% and 0.86, respectively. All models demonstrated relatively similar performance on the validation and test sets, implying good generalizability of models trained on unseen samples.

[0041] Figures 7a-e show the 10-year recurrence-free survival curves of responders and non-responders identified after treatment, based on clinical and histological criteria, and before treatment, using the four predictive models presented in Table 2. Survival analysis demonstrated a statistically significant difference (p-value = 0.030) between the survival curves of responders and non-responders identified after treatment. Among the predicted responder cohorts before treatment, those identified using the predictive model with RAN as the feature network demonstrated a statistically significant difference or approached significance. In particular, the model inputting parameter images of the tumor nuclei approached a significant difference (p-value = 0.058), whereas the model with input parameter images extending to the tumor border demonstrated a statistically significant difference (p-value = 0.040) between the survival of the two predicted cohorts. The responses identified by the other two models before treatment did not show a significant difference in survival.

[0042] In yet another embodiment, a tensor processing unit (TPU) may be used as an alternative to the GPU 41, allowing the model to run in a substantially faster and smoother manner. The TPU is an AI accelerator application specific integrated circuit (ASIC) specifically for neural network machine learning.

[0043] Implementations of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or by one or more combinations thereof. Implementations of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory computer storage medium for execution by or control of the operation of a data processing device. Alternatively, or in addition, the program instructions can be encoded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a receiver device suitable for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0044] A computer program, which may also be referred to or described as a program, software, software application, module, software module, script, or code, may be written in any type of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be arranged in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may correspond to a file in a file system, but this is not required. A program may be stored in part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple organized files, e.g., files that store one or more modules, subprograms, or code portions. A computer program may be arranged to be executed on one computer, or on multiple computers that are located in one location or distributed across multiple locations and interconnected by a communication network. Portions of the program shown in the various figures are shown as individual modules that implement various structures and functions through various objects, methods, or other processes, but the program may instead include several sub-modules, third party services, components, libraries, etc., as appropriate. Conversely, the structure and functionality of the various components may be combined into a single component where appropriate.

[0045] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, e.g., a CPU, GPU, FPGA, or ASIC.

[0046] To provide interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display), LED (light emitting diode), or plasma monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse, trackball, or trackpad, by which the user can provide input to the computer. Input can also be provided to the computer using a touch screen, such as a tablet computer surface with pressure sensitivity, a multi-touch screen using capacitive or electrical sensing, or other types of touch screens. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0047] The term "graphical user interface" or "GUI", in the singular or plural, can be used to describe one or more graphical user interfaces and each display of a particular graphical user interface. A GUI can refer to any graphical user interface, including but not limited to a web browser, a touch screen, or a command line interface (CLI), that processes information and efficiently presents the information results to a user. In general, a GUI can include multiple user interface (UI) elements, some or all of which are associated with a web browser, such as interactive fields, pull-down lists, buttons, etc. that can be acted upon by a user. These and other UI elements can relate to or represent the functionality of a web browser.

[0048] Implementations of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., as a data server, a middleware component, e.g., an application server, a front-end component, e.g., a client computer having a graphical user interface or web browser through which a user can interact with implementations of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system 10 may be interconnected by any form or medium of wired and / or wireless digital data communication, e.g., a communications network 44. Examples of communications networks include local area networks (LANs), radio access networks (RANs), metropolitan area networks (MANs), wide area networks (WANs), Worldwide Interoperability for Microwave Access (WIMAX), wireless local area networks (WLANs), e.g., using 802.11a / b / g / n or 802.20, all or a portion of the Internet, and / or any other communications system or systems in one or more locations, as well as free space optical networks. Networks may communicate using, for example, Internet Protocol (IP) packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells, voice, video, data, and / or other suitable information between network addresses.

[0049] A computing system may include clients and servers and / or Internet of Things (IOT) devices running publisher / subscriber applications. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of clients and servers arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0050] There may be any number of computers associated with system 10 or external to system 10 communicating via network 44. Additionally, the terms "client," "user," and other suitable terms may be used interchangeably where appropriate without departing from the scope of this disclosure.

[0051] In another embodiment, system 10 follows the cloud computing model by providing on-demand network access to a shared pool of configurable computing resources (e.g., servers, storage, applications and / or services) that can be rapidly provisioned and released with minimal or no resource management effort, including interaction with a service provider by a user (operator of a thin client).

[0052] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. It should be noted that each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions described in the blocks may be performed in a different order than that described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that executes the specified functions or operations, or a combination of dedicated hardware and computer instructions.

[0053] Advantages, other benefits, and solutions to problems have been described above with respect to a particular embodiment. However, benefits, advantages, solutions to problems, and any elements that can cause any benefit, advantage, or solution to occur or become more pronounced should not be construed as critical, necessary, or essential features or elements of any or all claims. As used herein, the terms "comprises," "comprising," or any variation thereof, are intended to include a non-exclusive inclusion, such that a process, method, article, or device that includes a list of elements does not include only those elements, but may include other elements not expressly listed or other elements inherent to such process, method, article, or device. Furthermore, any element described herein is not necessarily required for the practice of the invention unless expressly described as "essential" or "essential."

[0054] The above description of exemplary embodiments of the present invention refers to the accompanying drawings, which show exemplary embodiments for illustration purposes. These exemplary embodiments are described in sufficient detail to enable one skilled in the art to practice the present invention, but it should be understood that other embodiments may be realized and that logical and mechanical changes may be made without departing from the spirit and scope of the present invention. For example, the steps recited in any method or process claim may be performed in any order and are not limited to the order presented. The above detailed description is for illustrative purposes only and not for limiting purposes, and the scope of the present invention is defined by the above description and with reference to the appended claims.

[0055] [References] [1] H. Sung et al., “Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” CA. Cancer J. Clin., Feb. 2021, doi: 10.3322 / caac.21660. [2] F. Bray, J. Ferlay, I. Soerjomataram, R. L. Siegel, L. A. Torre, and A. Jemal, “Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” CA. Cancer J. Clin., vol. 68, no. 6, pp. 394-424, Nov. 2018, doi: 10.3322 / caac.21492. [3] S. H. Giordano, “Update on Locally Advanced Breast Cancer,” Oncologist, vol. 8, no. 6, pp. 521-530, Dec. 2003, doi: 10.1634 / theoncologist.8-6-521. [4] G. N. Hortobagyi, “Comprehensive management of locally advanced breast cancer.,” Cancer, vol. 66, no. 6 Suppl, pp. 1387-91, Sep. 1990, doi: 10.1002 / 1097-0142(19900915)66:14+<1387::aid-cncr2820661414>3.0.co;2-i. [5] G. G et al., “Locally advanced non-metastatic breast cancer: analysis of prognostic factors in 125 patients homogeneously treated with a combined modality approach,” Eur. J. Cancer, vol. 31A, no. 9, pp. 1428-1433, 1995, doi: 10.1016 / 0959-8049(95)00199-S. [6] S. J. Cleator, A. Makris, S. E. Ashley, R. Lal, and T. J. Powles, “Good clinical response of breast cancers to neoadjuvant chemoendocrine therapy is associated with improved overall survival.,” Ann. Oncol. Off. J. Eur. Soc. Med. Oncol., vol. 16, no. 2, pp. 267-72, Feb. 2005, doi: 10.1093 / annonc / mdi049. [7] V. Guarneri et al., “Prognostic value of pathologic complete response after primary chemotherapy in relation to hormone receptor status and other factors,” J. Clin. Oncol., vol. 24, no. 7, pp. 1037-1044, Mar. 2006, doi: 10.1200 / JCO.2005.02.6914. [8] B. T. Hennessy et al., “Outcome after pathologic complete eradication of cytologically proven breast cancer axillary node metastases following primary chemotherapy,” J. Clin. Oncol., vol. 23, no. 36, pp. 9304-9311, 2005, doi: 10.1200 / JCO.2005.02.5023. [9] J. B. Nikas, W. C. Low, and P. A. Burgio, “Prognosis of treatment response (pathological complete response) in breast cancer,” Biomark. Insights, vol. 7, pp. 59-70, May 2012, doi: 10.4137 / BMI.S9387.

[10] B. Fisher et al., “Effect of preoperative chemotherapy on the outcome of women with operable breast cancer,” J. Clin. Oncol., vol. 16, no. 8, pp. 2672-2685, 1998, doi: 10.1200 / JCO.1998.16.8.2672.

[11] S. H. Giordano, “Update on Locally Advanced Breast Cancer,” Oncologist, vol. 8, no. 6, pp. 521-530, 2003, doi: 10.1634 / theoncologist.8-6-521.

[12] D. Sethi, R. Sen, J. Sen, S. Parshad, S. Khetarpal, and M. Garg, “Histopathologic changes following neoadjuvant chemotherapy in various malignancies,” Int. J. Appl. Basic Med. Res., vol. 2, no. 2, p. 111, 2012, doi: 10.4103 / 2229-516x.106353.

[13] S. Chuthapisith, J. M. Eremin, M. El-Sheemy, and O. Eremin, “Neoadjuvant chemotherapy in women with large and locally advanced breast cancer: Chemoresistance and prediction of response to drug therapy,” Surgeon, vol. 4, no. 4. Edinburgh University Press, pp. 211-219, 01-Aug-2006, doi: 10.1016 / S1479-666X(06)80062-4.

[14] W. Haque, V. Verma, S. Hatch, V. Suzanne Klimberg, E. Brian Butler, and B. S. Teh, “Response rates and pathologic complete response by breast cancer molecular subtype following neoadjuvant chemotherapy,” Breast Cancer Res. Treat., vol. 170, no. 3, pp. 559-567, Aug. 2018, doi: 10.1007 / s10549-018-4801-3.

[15] T. Byrski et al., “Pathologic complete response rates in young women with BRCA1-positive breast cancers after neoadjuvant chemotherapy,” J. Clin. Oncol., vol. 28, no. 3, pp. 375-379, Jan. 2010, doi: 10.1200 / JCO.2008.20.7019.

[16] K. Brindle, “New approaches for imaging tumour responses to treatment,” Nature Reviews Cancer, vol. 8, no. 2. pp. 94-107, Feb-2008, doi: 10.1038 / nrc2289.

[17] G. von Minckwitz et al., “S3-2: Neoadjuvant Chemotherapy Adapted by Interim Response Improves Overall Survival of Primary Breast Cancer Patients - Results of the GeparTrio Trial.,” in Cancer Research, 2011, vol. 71, no. 24, doi: 10.1158 / 0008-5472.sabcs11-s3-2.

[18] G. Von Minckwitz et al., “Neoadjuvant vinorelbine-capecitabine versus docetaxel-doxorubicin- cyclophosphamide in early nonresponsive breast cancer: Phase III randomized gepartrio trial,” J. Natl. Cancer Inst., vol. 100, no. 8, pp. 542-551, Apr. 2008, doi: 10.1093 / jnci / djn085.

[19] J. Mamou and M. L. Oelze, Quantitative ultrasound in soft tissues. Springer Netherlands, 2013.

[20] E. J. Feleppa et al., “Typing of prostate tissue by ultrasonic spectrum analysis,” IEEE Trans. Ultrason. Ferroelectr. Freq. Control, vol. 43, no. 4, pp. 609-619, Jul. 1996, doi: 10.1109 / 58.503779.

[21] M. Yang, T. M. Krueger, J. G. Miller, and M. R. Holland, “Characterization of anisotropic myocardial backscatter using spectral slope, intercept and midband fit parameters,” Ultrason. Imaging, vol. 29, no. 2, pp. 122-134, Apr. 2007, doi: 10.1177 / 016173460702900204.

[22] D. J. Coleman, F. L. Lizzi, R. H. Silverman, L. Helson, J. H. Torpey, and M. J. Rondeau, “A model for acoustic characterization of intraocular tumors.,” Invest. Ophthalmol. Vis. Sci., vol. 26, no. 4, pp. 545-50, Apr. 1985.

[23] H. Tadayyon, A. Sadeghi-Naini, L. Wirtzfeld, F. C. Wright, and G. Czarnota, “Quantitative ultrasound characterization of locally advanced breast cancer by estimation of its scatterer properties,” Med. Phys., vol. 41, no. 012903, Jan. 2014, doi: 10.1118 / 1.4852875.

[24] H. Tadayyon, A. Sadeghi-Naini, and G. J. Czarnota, “Noninvasive characterization of locally advanced breast cancer using textural analysis of quantitative ultrasound parametric images,” Transl. Oncol., vol. 7, no. 6, pp. 759-767, 2014, doi: 10.1016 / j.tranon.2014.10.007.

[25] A. Sadeghi-Naini et al., “Quantitative ultrasound spectroscopic imaging for characterization of disease extent in prostate cancer patients,” Transl. Oncol., vol. 8, no. 1, pp. 25-34, Feb. 2015, doi: 10.1016 / j.tranon.2014.11.005.

[26] A. Guimond et al., “Quantitative ultrasonic tissue characterization as a new tool for continuous monitoring of chronic liver remodelling in mice,” Liver Int., vol. 27, no. 6, pp. 854-64, Aug. 2007, doi: 10.1111 / j.1478-3231.2007.01493.x.

[27] B. Banihashemi, R. Vlad, B. Debeljevic, A. Giles, M. C. Kolios, and G. J. Czarnota, “Ultrasound imaging of apoptosis in tumor response: Novel preclinical monitoring of photodynamic therapy effects,” Cancer Res., vol. 68, no. 20, pp. 8590-8596, Oct. 2008, doi: 10.1158 / 0008-5472.CAN-08-0006.

[28] R. M. Vlad, S. Brand, A. Giles, M. C. Kolios, and G. J. Czarnota, “Quantitative ultrasound characterization of responses to radiotherapy in cancer mouse models,” Clin. Cancer Res., vol. 15, no. 6, pp. 2067-2075, Mar. 2009, doi: 10.1158 / 1078-0432.CCR-08-1970.

[29] G. J. Czarnota et al., “Tumor radiation response enhancement by acoustical stimulation of the vasculature,” Proc. Natl. Acad. Sci. U. S. A., vol. 109, no. 30, pp. E2033-E2041, Jul. 2012, doi: 10.1073 / pnas.1200053109.

[30] A. Sadeghi-Naini et al., “Low-frequency quantitative ultrasound imaging of cell death in vivo,” Med. Phys., vol. 40, no. 8, p. 082901, Jul. 2013, doi: 10.1118 / 1.4812683.

[31] H. Tadayyon et al., “Quantitative ultrasound assessment of breast tumor response to chemotherapy using a multi-parameter approach,” Oncotarget, vol. 7, no. 29, pp. 45094-45111, Jul. 2016, doi: 10.18632 / oncotarget.8862.

[32] L. Sannachi et al., “Non-invasive evaluation of breast cancer response to chemotherapy using quantitative ultrasonic backscatter parameters,” Med. Image Anal., vol. 20, no. 1, pp. 224-236, Feb. 2015, doi: 10.1016 / j.media.2014.11.009.

[33] A. Sadeghi-Naini et al., “Quantitative ultrasound evaluation of tumor cell death response in locally advanced breast cancer patients receiving chemotherapy,” Clin. Cancer Res., vol. 19, no. 8, pp. 2163-2173, Apr. 2013, doi: 10.1158 / 1078-0432.CCR-12-2965.

[34] D. DiCenzo et al., “Quantitative ultrasound radiomics in predicting response to neoadjuvant chemotherapy in patients with locally advanced breast cancer: Results from multi‐institutional study,” Cancer Med., vol. 9, no. 16, pp. 5798-5806, Aug. 2020, doi: 10.1002 / cam4.3255.

[35] A. Sadeghi-Naini et al., “Conventional frequency ultrasonic biomarkers of cancer treatment response in vivo,” Transl. Oncol., vol. 6, no. 3, pp. 234-243, 2013, doi: 10.1593 / tlo.12385.

[36] A. Sadeghi-Naini et al., “Chemotherapy-Response Monitoring of Breast Cancer Patients Using Quantitative Ultrasound-Based Intra-Tumour Heterogeneities,” Sci. Rep., vol. 7, no. 10352, Dec. 2017, doi: 10.1038 / s41598-017-09678-0.

[37] A. Sadeghi-Naini et al., “Early prediction of therapy responses and outcomes in breast cancer patients using quantitative ultrasound spectral texture,” Oncotarget, vol. 5, no. 11, pp. 3497-3511, 2014, doi: 10.18632 / oncotarget.1950.

[38] A. Dasgupta et al., “Quantitative ultrasound radiomics using texture derivatives in prediction of treatment response to neo-adjuvant chemotherapy for locally advanced breast cancer,” Oncotarget, vol. 11, no. 42, pp. 3782-3792, Oct. 2020, doi: 10.18632 / oncotarget.27742.

[39] H. Tadayyon et al., “A priori Prediction of Neoadjuvant Chemotherapy Response and Survival in Breast Cancer Patients using Quantitative Ultrasound,” Sci. Rep., vol. 7, no. 45733 (2017), Apr. 2017, doi: 10.1038 / srep45733.

[40] D. Shen, G. Wu, and H.-I. Suk, Deep Learning in Medical Image Analysis. Elsevier, 2017.

[41] G. Litjens et al., “A survey on deep learning in medical image analysis,” Medical Image Analysis, vol. 42. pp. 60-88, 01-Dec-2017, doi: 10.1016 / j.media.2017.07.005.

[42] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, “Deep learning,” 2016, doi: 10.4258 / hir.2016.22.4.351.

[43] K. Ravichandran, N. Braman, A. Janowczyk, and A. Madabhushi, “A deep learning classifier for prediction of pathological complete response to neoadjuvant chemotherapy from baseline breast DCE-MRI,” in Medical Imaging 2018: Computer-Aided Diagnosis, 2018, vol. 10575, p. 11, doi: 10.1117 / 12.2294056.

[44] Y. Qu, H. Zhu, K. Cao, X. Li, M. Ye, and Y. Sun, “Prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer using a deep learning (DL) method,” Thorac. Cancer, vol. 11, no. 3, pp. 651-658, Mar. 2020, doi: 10.1111 / 1759-7714.13309.

[45] R. Ha et al., “Prior to Initiation of Chemotherapy, Can We Predict Breast Tumor Response? Deep Learning Convolutional Neural Networks Approach Using a Breast MRI Tumor Dataset,” J. Digit. Imaging, vol. 32, no. 5, pp. 693-701, Oct. 2019, doi: 10.1007 / s10278-018-0144-1.

[46] M. Byra, H. Piotrzkowska-Wroblewska, K. Dobruch-Sobczak, and A. Nowicki, “Combining Nakagami imaging and convolutional neural network for breast lesion classification,” in IEEE International Ultrasonics Symposium, IUS, 2017, pp. 1-4, doi: 10.1109 / ULTSYM.2017.8092154.

[47] F. Wang et al., “Residual Attention Network for Image Classification,” Proc. - 30th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR 2017, vol. 2017-January, pp. 6450-6458, Apr. 2017.

[48] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016, vol. 2016-December, pp. 770-778, doi: 10.1109 / CVPR.2016.90.

[49] E. A. A. Eisenhauer et al., “New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1),” Eur J Cancer, no. 14865, pp. 228-247, Jan. 2009, doi: 10.1016 / j.ejca.2008.10.026.

[50] K. N. Ogston et al., “A new histological grading system to assess response of breast cancers to primary chemotherapy: prognostic significance and survival,” The Breast, vol. 12, no. 5, pp. 320-327, Oct. 2003, doi: 10.1016 / S0960-9776(03)00106-1.

[51] H. Taleghamar, H. Moghadas-Dastjerdi, G. J. Czarnota, and A. Sadeghi-Naini, “Characterizing intra-tumor regions on quantitative ultrasound parametric images to predict breast cancer response to chemotherapy at pre-treatment,” Sci. Rep., vol. 11, no. 14865, Jul. 2021, doi: 10.1038 / s41598-021-94004-y.

[52] Y. Labyed, T. A. Bigelow, and B. L. McFarlin, “Estimate of the attenuation coefficient using a clinical array transducer for the detection of cervical ripening in human pregnancy,” Ultrasonics, vol. 51, no. 1, pp. 34-39, Jan. 2011, doi: 10.1016 / j.ultras.2010.05.005.

[53] L. X. Yao, J. A. Zagzebski, and E. L. Madsen, “Backscatter coefficient measurements using a reference phantom to extract depth-dependent instrumentation factors,” Ultrason. Imaging, vol. 12, no. 1, pp. 58-70, Jan. 1990, doi: 10.1016 / 0161-7346(90)90221-i.

[54] F. L. Lizzi et al., “Comparison of theoretical scattering results and ultrasonic data from clinical liver examinations,” Ultrasound Med. Biol., vol. 14, no. 5, pp. 377-385, 1988.

[55] M. F. Insana, R. F. Wagner, D. G. Brown, and T. J. Hall, “Describing small-scale structure in random media using pulse-echo ultrasound,” J. Acoust. Soc. Am., vol. 87, no. 1, pp. 179-92, Jan. 1990.

[56] M. L. Oelze, W. D. O’Brien, J. P. Blue, and J. F. Zachary, “Differentiation and characterization of rat mammary fibroadenomas and 4T1 mouse carcinomas using quantitative ultrasound imaging,” IEEE Trans. Med. Imaging, vol. 23, no. 6, pp. 764-771, Jun. 2004, doi: 10.1109 / TMI.2004.826953.

[57] M. F. Insana and T. J. Hall, “Parametric ultrasound imaging from backscatter coefficient measurements: image formation and interpretation.,” Ultrason. Imaging, vol. 12, no. 4, pp. 245-67, Oct. 1990.

[58] A. Sadeghi-Naini et al., “Breast-Lesion Characterization using Textural Features of Quantitative Ultrasound Parametric Maps,” Sci. Rep., vol. 7, no. 13638, Dec. 2017, doi: 10.1038 / S41598-017-13977-X.

[59] K. A. Topp, J. F. Zachary, and J. O’Brien, “Quantifying B-mode images of in vivo rat mammary tumors by the frequency dependence of backscatter,” J. Ultrasound Med., vol. 20, no. 6, pp. 605-612, 2001, doi: 10.7863 / JUM.2001.20.6.605.

[60] D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, 2015.

[61] L. M. Zintgraf, T. S. Cohen, T. Adel, and M. Welling, “Visualizing Deep Neural Network Decisions: Prediction Difference Analysis,” 5th Int. Conf. Learn. Represent. ICLR 2017 - Conf. Track Proc., Feb. 2017.

[62] L. Sannachi et al., “Response monitoring of breast cancer patients receiving neoadjuvant chemotherapy using quantitative ultrasound, texture, and molecular features,” PLoS One, vol. 13, no. 1, p. e0189634, Jan. 2018, doi: 10.1371 / journal.pone.0189634.

[63] M. Byra, A. Nowicki, H. Wroblewska-Piotrzkowska, and K. Dobruch-Sobczak, “Classification of breast lesions using segmented quantitative ultrasound maps of homodyned K distribution parameters,” Med. Phys., vol. 43, no. 10, pp. 5561-5569, Oct. 2016, doi: 10.1118 / 1.4962928.

[64] J. Wu, G. Gong, Y. Cui, and R. Li, “Intratumor partitioning and texture analysis of dynamic contrast-enhanced (DCE)-MRI identifies relevant tumor subregions to predict pathological response of breast cancer to neoadjuvant chemotherapy,” J. Magn. Reson. Imaging, vol. 44, no. 5, pp. 1107-1115, Nov. 2016, doi: 10.1002 / jmri.25279.

[65] A. Sadeghi-Naini et al., “Early detection of chemotherapy-refractory patients by monitoring textural alterations in diffuse optical spectroscopic images,” Med. Phys., vol. 42, no. 11, pp. 6130-6146, Nov. 2015, doi: 10.1118 / 1.4931603.

[66] E. Karami et al., “Quantitative MRI Biomarkers of Stereotactic Radiotherapy Outcome in Brain Metastasis,” Sci. Rep., vol. 9, no. 19830, Dec. 2019, doi: 10.1038 / s41598-019-56185-5.

[67] P. M. Lamb, N. M. Perry, S. J. Vinnicombe, and C. A. Wells, “Correlation between ultrasound characteristics, mammographic findings and histological grade in patients with invasive ductal carcinoma of the breast,” Clin. Radiol., vol. 55, no. 1, pp. 40-44, 2000, doi: 10.1053 / crad.1999.0333.

[68] A. M. Fowler, D. A. Mankoff, and B. N. Joe, “Imaging Neoadjuvant Therapy Response in Breast Cancer,” Radiology, vol. 285, no. 2, pp. 358-375, Nov. 2017, doi: 10.1148 / radiol.2017170180.

Claims

1. 1. A system for predicting breast cancer response to neoadjuvant chemotherapy (NAC) using quantitative ultrasound (QUS) parametric images and / or B-mode images, the system comprising: an imaging system for acquiring at least one ultrasound frame of data including raw RF signals and / or an image; a computer system including a hardware processor and a memory device encoded with instructions; wherein the instructions cause the hardware processor to: receiving the at least one ultrasound frame of data and identifying a region of interest (ROI) in each of the at least one ultrasound frame of data using one or more predetermined rules, the ROI including a tumor, and the ROI may be identified automatically; generating at least one quantitative ultrasound (QUS) parameter map and / or processed B-mode image of the tumor from the at least one frame of ultrasound data; extracting optimal feature maps from said QUS parameter images and / or B-mode images using a feature network of a machine learning architecture; inputting the optimized feature map into a predictive network of the machine learning architecture, wherein the optimized feature maps obtained from the feature network are combined or averaged across all images associated with each tumor, and the feature and predictive networks may be integrated in an end-to-end system; pre-processing and formatting the QUS parametric images and / or B-mode images to generate a training data set for a deep learning model; training the feature network to generate the optimal feature map for a single QUS parametric image and / or B-mode image; For the end-to-end system, training the prediction network using a combined or averaged feature vector associated with each patient in the training dataset, wherein the combined feature and prediction network are jointly trained; classifying the tumor subject as a responder or non-responder to said NAC, and classifying the response as at least one of a complete response, a partial response, stable disease, or progressive disease; A system that allows the execution of the above.

2. The system of claim 1 , wherein the at least one ultrasound frame of data comprises at least one of an ultrasound B-mode image and radio frequency (RF) data.

3. The system of claim 1 , wherein the parameter maps are generated for all imaging planes of the tumor using a sliding window analysis throughout the region of interest.

4. 4. The system of claim 3, wherein the sliding window size is selected with an overlap size to obtain isotropic pixels such that the sliding window covers sufficient ultrasound wavelengths in the axial direction for spectral analysis while preserving texture in the generated parameter map.

5. 10. The system of claim 1, wherein the machine learning architecture comprises at least one of a deep convolutional neural network (CNN) architecture, a deep learning architecture, and a transformer architecture.

6. 6. The system of claim 5, wherein the deep convolutional neural network (DCNN) architecture comprises a residual network (ResNet) architecture.

7. 6. The system of claim 5, wherein the deep convolutional neural network (DCNN) architecture comprises a residual attention network (RAN) architecture.

8. The system according to any one of claims 1 to 7, wherein the imaging system is an ultrasound imaging system.

9. 9. The system of claim 8, wherein the optimized feature map obtained from the feature network is averaged across all parametric images associated with each tumor.

10. 10. The system of claim 9, wherein the averaged, optimized feature map is then input into the prediction network adapted for response prediction at the patient level.

11. 11. The system of claim 10, wherein the predictive network includes a fully connected layer having an input layer, a middle layer, and a softmax layer at the end with an output size of 2 to predict the probability of a response category (responder vs. non-responder) for each patient.

12. 12. The system of claim 11, wherein the input layer comprises the same size as a flattened feature vector (256) and the middle layer comprises 100 neurons.

13. 13. The system of claim 12, wherein a dropout layer is added after each layer to minimize overfitting and improve its generalization performance.

14. 2. The system of claim 1, wherein the parameter images of the training set are preprocessed by resampling the parameter images to a predetermined pixel size, and pixel values ​​in the parameter images of the training set are normalized to (0 1) to promote training convergence.

15. 15. The system of claim 14, wherein a training set normalization parameter is used for normalization and data augmentation is applied to the training data set.

16. 16. The system of claim 15, wherein the training data set is augmented by at least horizontally flipping and shifting the parametric images both horizontally and vertically.

17. 16. The system of claim 15, wherein the feature network is trained to generate the optimal feature map for a single QUS parameter image by inputting the parameter image into the feature network.

18. 20. The system of claim 17, wherein the optimal feature map is obtained by feeding, for each imaging plane, its corresponding parametric image into the trained feature network.

19. 20. The system of claim 18, wherein the optimal features are calculated for all 2D imaging planes of the tumor, flattened into a 1D vector, and then averaged over the entire tumor volume to obtain a mean feature vector used in the prediction network.

20. 20. The system of claim 19, wherein the predictive network is trained using the averaged feature vectors associated with the patients in the training dataset and evaluated across an independent test set for response prediction.

21. 1. A method for predicting response to neoadjuvant chemotherapy (NAC) using quantitative ultrasound (QUS) parametric images, the method comprising: acquiring at least one ultrasound frame of data and / or image comprising raw RF signals using an imaging system; A computer system including a hardware processor and a memory device encoded with instructions is used to cause the hardware processor to: receiving the at least one ultrasound frame of data and identifying a region of interest (ROI) in each of the at least one ultrasound frame of data using one or more predetermined rules, the ROI including a tumor, and the ROI may be identified automatically; generating at least one quantitative ultrasound (QUS) parameter map and / or processed B-mode image of the tumor from the at least one frame of ultrasound data; extracting optimal feature maps from said QUS parameter images and / or B-mode images using a feature network of a machine learning architecture; inputting the optimized feature map into a predictive network of the machine learning architecture, wherein the optimized feature maps obtained from the feature network are combined or averaged across all images associated with each tumor, and the feature and predictive networks may be integrated in an end-to-end system; pre-processing and formatting the QUS parametric images and / or B-mode images to generate a training data set for a deep learning model; training the feature network to generate the optimal feature map for a single QUS parametric image and / or B-mode image; For the end-to-end system described above, the operation includes training the prediction network using a combined or averaged feature vector associated with each patient in the training dataset, wherein the combined feature and prediction network are trained together; classifying the tumor subject as a responder or non-responder to said NAC, and classifying the response as at least one of a complete response, a partial response, stable disease, or progressive disease; and A method comprising:

Images