Improving magnetic resonance imaging annotation

Abstract

Embodiments herein disclose computer-implemented methods, computer program products, and computer systems for annotating magnetic resonance imaging (MRI) images. The methods can include receiving mammogram (MG) image data representing an annotated MG image of a patient breast, the annotated MG image being one of a cranio-caudal view or a mediolateral oblique view. The method can include identifying an annotation representing an abnormality at a first location in the annotated MG image; receiving MRI image data representing an MRI image of the patient breast; generating annotated MRI image data using the MRI image data and the annotation identified in the annotated MG image, the annotated MRI image data including an MRI annotation at a second location in the annotated MRI image representing the abnormality based at least in part on the first location; and storing the annotated MRI image data in a database.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to the field of medical image annotation, and more specifically to projecting mammogram (MG) image annotations onto magnetic resonance imaging (MRI) images. BACKGROUND

[0002] In performing breast cancer screening examinations on patients, medical professionals capture and examine multiple images taken from different viewpoints and from different time points to make a diagnosis. Medical professionals use mammograms to help identify early signs of breast cancer. Mammograms are X-ray pictures of the breast and can be used as screening mammograms to examine women without signs or symptoms of disease for breast cancer. To help make a diagnosis, computer-aided diagnosis (CAD) algorithms are often used to make a decision based on a single image. Images taken during a mammogram can include different viewpoints or angles of the left and right breasts. For example, standard mammogram image view types can include bilateral craniocaudal (CC), bilateral mediolateral oblique (MLO), where bilateral refers to the left and right versions of the view.

[0003] Mammography is a more common and cost-effective image diagnosis for making a diagnosis of breast cancer, but is limited in its ability to identify all types of abnormalities, especially hard tissue identification. However, MRI is able to identify abnormalities that would be less evident in an MG examination, although MRI is more computationally expensive, time consuming, and requires a higher skill level. SUMMARY

[0004] The present invention is described in disclosing various embodiments of a method, computer program product, and computer system for annotating magnetic resonance imaging (MRI) images using mammogram annotations. One embodiment of the disclosure is a computer-implemented method for annotating an MRI image, the computer-implemented method executable by one or more processors configured for receiving mammogram (MG) image data representing one or more annotated MG images of a patient breast; identifying one or more annotations in the one or more annotated MG images, the one or more annotations representing an abnormality at a first location in the one or more annotated MG images; receiving MRI image data representing one or more MRI images of the patient breast; generating annotated MRI image data using the MRI image data and the one or more annotations identified in the one or more annotated MG images, the annotated MRI image data including one or more annotations representing the abnormality at a second location based at least in part on the first location, and storing the annotated MRI image data in a database.

[0005] In embodiments, each of the one or more annotated MG images is one or either of a craniocaudal (CC) view or a mediolateral oblique (MLO) view. The one or more MRI images can include a plurality of image slices, including one or more image slices having an abnormality.

[0006] Some embodiments can include one or more processors configured to determine a breast side and quadrant of a patient’s breast based on a first location in the one or more annotated MG images, and locate a corresponding breast side and quadrant in a plurality of image slices of the one or more MRI images. Further, the one or more processors configured to generate annotated MRI image data can also be configured to map the one or more annotations to a second location of a corresponding breast side and quadrant of an MRI image represented in the MRI image data. Additionally, the one or more processors can be configured to provide the annotated MRI image data to a supervised machine learning model, and process the annotated MRI image data with the one or more annotations using the supervised machine learning model to generate output data corresponding to a diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 A block diagram depicting a distributed data processing environment for annotating MRI images according to embodiments of the present application is depicted.

[0008] Figure 2 An architectural diagram depicting a research grade image model for annotating MRI images to perform diagnostic assessments according to embodiments of the present application is depicted.

[0009] Figure 3 A set of annotated images according to embodiments of the present application is depicted.

[0010] Figure 4 Another set of annotated images according to embodiments of the present application is depicted.

[0011] Figure 5 A flowchart depicting a machine learning model for performing diagnostic assessments using annotated images according to embodiments of the present application is depicted.

[0012] Figure 6 A flowchart depicting another machine learning model for performing diagnostic assessments using annotated images according to embodiments of the present application is depicted.

[0013] Figure 7 A flowchart depicting steps of a computer-implemented method for annotating MRI images according to embodiments of the present application is depicted.

[0014] Figure 8 A flowchart depicting steps of a computer-implemented method for annotating MRI images according to embodiments of the present application is depicted. Figure 1Block diagram of a computing device of a distributed data processing environment. DETAILED DESCRIPTION

[0015] The present invention addresses the problem of efficiently annotating MRI images. When mammographers screen patients for breast cancer, they examine multiple images taken from different viewpoints in one or more prior exams before they get their diagnosis. The medical specialist performing the breast cancer screening exam can capture an X-ray image referred to as a screening MG. The screening MG image can be annotated by the medical specialist to identify relevant findings (e.g., abnormalities, lesions, tumors) in the image or to show useful information about the relevant findings. The annotations can be saved within the MG image or saved elsewhere and associated with the MG image.

[0016] The present invention therefore provides a computer-implemented method of efficiently and automatically annotating MRI images using annotated MG images. During implementation, a computing device comprising one or more processors can be configured to receive, from a database, MG image data representing one or more annotated MG images of a patient's breast.

[0017] Each of the one or more annotated MG images is one of a craniocaudal (CC) view or a mediolateral oblique (MLO) view. Each image taken during the MG includes a respective type of view. For example, the type of view for an image can include a standard type of view or a non-standard type of view for a mammogram, where the standard type of view is typically captured during a screening exam and the non-standard type of view is typically captured in a subsequent diagnostic exam. The standard type of view can be captured during a screening exam and can include a bilateral CC view or a bilateral MLO view, where the bilateral type of view can include a location variant representing a left side (L) or a right side (R) indicated as a CC L view, a CC R view, an MLO L view, and an MLO R view. The non-standard type of view can be captured during a diagnostic exam and can include one of an intermediate lateral (ML) view, a lateral-medial (LM) view, a lateral-medial oblique (LMO) view, a late intermediate lateral (late ML) view, a stepped oblique (SO) view, a spot view, a spot compression view, a double spot compression view, a magnification view, a magnified craniocaudal view (XCC), an axillary view, a cut view, a tangential view, a CC view, a large eye CC view, a rolled CC view, a lifted CC view, a 20° oblique view, an inferior-medial supero-lateral oblique projection view, and an Eklund technique view. The non-standard image can correspond to a standard image such that the non-standard image can be a variation of one or more of the standard images.

[0018] The one or more MRI images can include a plurality of image slices, including one or more image slices having an abnormality. In one embodiment, the one or more processors can be configured to apply a lesion segmentation algorithm using only the discovered relevant image slices and the regions identified as having an abnormality.

[0019] The one or more processors can be configured to identify one or more annotations representing an abnormality at a first location in the one or more annotated MG images, and receive MRI image data representing an MRI image of the patient breast.

[0020] The one or more processors can be configured to generate annotated MRI image data using the MRI image data and the one or more annotations identified in the one or more annotated MG images. The annotated MRI image data can include one or more annotations representing an abnormality at a second location based at least in part on the first location.

[0021] The one or more processors configured to generate annotated MRI image data can be further configured to map the one or more annotations to a second location of a corresponding breast side and quadrant of the MRI image represented in the MRI image data. For example, the patient breast can be identified by breast side and quadrant in which an abnormality can be found.

[0022] The one or more processors can be configured to store the annotated MRI image data in a database according to methods known to those of ordinary skill in the art for storing data in a database.

[0023] The one or more processors can be configured to provide the annotated MRI image data to a supervised machine learning model, and process the annotated MRI image data with the one or more annotations using the supervised machine learning model to generate output data corresponding to a diagnosis.

[0024] Embodiments of the present invention provide a system and method for processing MG image data from a standard image set and / or a diagnostic image set to determine a diagnostic assessment using a research-level image classifier and a diagnostic model.

[0025] Embodiments of the present invention provide a research-level image classifier for processing a standard set and / or a diagnostic set of mammograms to determine a classification (e.g., benign, malignant, normal) for each image in the study and produce model output data as a feature vector for each image. The research-level image classifier can include a deep convolutional neural network (“CNN”) for image classification that can be a rule-based machine learning model that combines a discovery component (e.g., genetic algorithm) with a learning component (e.g., supervised or unsupervised learning). Generally, an image classifier system seeks to identify a set of context-dependent rules that collectively store and apply knowledge in a piecewise fashion in order to make predictions (e.g., behavior modeling, classification, data mining, regression, function approximation, or game strategy). The research-level image classifier can also include a combination network component configured to concatenate the model output data feature vectors to produce a research-level classification through a series of fully connected layers. The research-level image classifier can be trained using images labeled with medical labels or otherwise associated with annotations and / or diagnostic reports to improve the functionality and accuracy of the image classifier.

[0026] In another embodiment, the research-level classifier can include two separate image-level classifiers, where a first image-level classifier receives the two CC (i.e., CC L and CC R) MG images and a second image-level classifier receives the two MLO (i.e., MLO L and MLO R) MG images. Further, the research-level image classifier can include an image-level classifier for each of the four input images.

[0027] Embodiments of the present invention can be configured to output a prediction for the entire study, where the prediction can represent a benign or malignant diagnosis. The system can also be configured to output a prediction for the left or right breast. The system can also be configured to classify the results according to the Breast Imaging Reporting and Data System (BI-RADS), which classifies results into categories numbered zero (0) through six (6). The system can also be configured to process the image data for each side separately and then combine the results through an averaging or maximum operation. The image classifier of the system can be pre-trained to produce a per-image diagnosis to improve training speed.

[0028] Further, embodiments of the present invention provide a diagnostic model that produces an overall diagnosis for a patient based on the output data from the research-level image classifier.

[0029] The present invention will now be described in detail with reference to the accompanying drawings.

[0030] Figure 1 A block diagram illustrating a distributed data processing environment 100 for annotating MG images according to embodiments of the present invention is shown. Figure 1Only one embodiment of the application is provided in the figures, and no implication arises that any particular implementation is limiting as to the environments in which different embodiments of the application can be implemented. In the depicted embodiment, the distributed data processing environment 100 includes computing devices 120, servers 125, databases 124, and image sensors 130 interconnected through a network 110. The network 110 operates as a computing network, which can be, for example, a local area network (LAN), a wide area network (WAN), or a combination of both, and can include wired, wireless, or fiber optic connections. In general, the network 110 can be any combination of connections and protocols that will support communications between the computing devices 120, servers 125, databases 124, and image sensors 130. The distributed data processing environment 100 can also include additional servers, computers, sensors, or other devices not shown.

[0031] The computing device 120 operates to execute at least a portion of a computer program for projecting MG image annotations onto MRI images and performing diagnostic assessments. In embodiments, the computing device 120 can be communicatively coupled with the image sensor 130, or the image sensor 130 can be one of the computing device 120 components. The computing device 120 is configured to send and / or receive data from the network 110 and the image sensor 130. In some embodiments, the computing device 120 can be a management server, a web server, or any other electronic device or computing system capable of receiving and transmitting data. In some embodiments, the computing device 120 can be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a smart phone, or any programmable electronic device capable of communicating with the databases 124, servers 125 via the network 110. The computing device 120 can include components as further described in detail in the Figure 8

[0032] The computing device 120 can also be configured to receive, store, and process images captured on the image sensor 130. For example, the computing device 120 can be communicatively coupled to the image sensor 130 and receive data corresponding to images captured by the image sensor 130 via a communication link. The computing device 120 can be configured to store the image data in a memory of the computing device 120 or transmit the image data to the databases 124 or servers 125 via the network 110. The image data can be processed by one or more processors of the computing device 120 or by one or more processors associated with the servers 125 in a cloud computing network.

[0033] ​Database 124 operates as a repository for data flowing to and from network 110. Examples of data include image data, annotated image data, user data, device data, network data, and data corresponding to images captured by image sensor 130. A database is an organized collection of data. Database 124 can be implemented with any type of storage device (e.g., a database server, a hard drive, or a flash memory) capable of storing data and configuration files that can be accessed and utilized by computing device 120. In one embodiment, database 124 is accessed by computing device 120 to store data corresponding to images captured by image sensor. In another embodiment, database 124 is accessed by computing device 120 to access annotated image data, user data, device data, network data, and data corresponding to images captured by image sensor 130. In another embodiment, database 124 can reside elsewhere within distributed network environment 100, so long as database 124 is accessible to network 110.

[0034] Server 125 can be a standalone computing device, a management server, a web server, or any other electronic device or computing system capable of receiving, sending, and processing data and capable of communicating with computing device 120 via network 110. In other embodiments, server 125 represents a server computing system that utilizes multiple computers as a server system, such as a cloud computing environment. In other embodiments, server 125 represents a computing system that utilizes clustered computers and components (e.g., database server computers, application server computers, etc.) that act as a single seamless resource pool when accessed within distributed data processing environment 100. Server 125 can include components as described in further detail below. Figure 8

[0035] While the above description, and Figure 2 ​The research-level image model 200 is shown, although the present disclosure is not so limited. In at least some embodiments, the model 200 can implement a trained component or trained model configured to perform the processes described above with respect to the research-level image model 200. The trained component can include one or more machine learning models 240, including but not limited to one or more classifiers, one or more neural networks, one or more probability maps, one or more decision trees, and the like. In other embodiments, the trained component can include a rules-based engine, one or more statistical-based algorithms, one or more mapping functions or other types of functions / algorithms to determine whether a natural language input is a complex or non-complex natural language input. In some embodiments, the trained component can be configured to perform binary classification, where a natural language input can be classified into one of two classes / categories. In some embodiments, the trained component can be configured to perform multi-class or multinomial classification, where a natural language input can be classified into one of three or more classes / categories. In some embodiments, the trained component can be configured to perform multi-label classification, where a natural language input can be associated with multiple classes / categories.

[0036] Different machine learning techniques can be used to train and operate the trained component to perform the different processes described herein. Models can be trained and operated according to different machine learning techniques. Such techniques can include, for example, neural networks such as deep neural networks and / or recurrent neural networks, inference engines, trained classifiers, and the like. Examples of trained classifiers include support vector machines (SVMs), neural networks, decision trees, AdaBoost (short for “Adaptive Boosting”) in combination with decision trees, and random forests. Focusing on SVMs as an example, SVMs are supervised learning models with an associated learning algorithm that analyzes data and identifies patterns in the data, and are commonly used for classification and regression analysis. Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that assigns new examples into one category or the other, making it a non-probabilistic binary linear classifier. More complex SVM models can be built with training sets that identify more than two categories, where the SVM determines which category is most similar to the input data. SVM models can be mapped such that examples of individual categories are separated by clear gaps. New examples are then mapped into the same space and predicted to belong to a category based on which side of the gap they fall on. A classifier can issue a “score” that indicates which category the data most closely matches. The score can provide an indication of how close the data matches the category.

[0037] To apply machine learning techniques, the machine learning process itself needs to be trained. Training a machine learning component requires establishing a“ground truth” for training examples. In machine learning, the term“ground truth” refers to the accuracy of a classification for a training set of supervised learning techniques. Different techniques can be used to train a model, including backpropagation, statistical learning, supervised learning, semi-supervised learning, stochastic learning, or other known techniques.

[0038] Figure 2 An architectural diagram of a research-grade image model 200 for annotating MRI images for diagnostic evaluation according to embodiments of the present application is described. The model 200 can include a machine learning model 240 configured to receive annotated MRI image data 208 and / or annotated MG image data 202 from an image database 224. The annotated MRI image data 202 can be generated by a computing device 220 including one or more processors configured to identify and extract annotations (e.g., using an annotation identifier and extractor 212). The computing device 220 (e.g., described in Figure 1 the middle as a computing device 120) can include one or more processors configured to project or map (e.g., using an annotation projector 216) one or more annotations 214 to an MRI image represented by MRI image data 206. Further, the computing device 220 can include one or more processors configured to generate annotated MRI image data 208 based on the MRI image data 206 and the one or more annotations 214 identified and extracted from the annotated MG image represented by the annotated MG image data 202.

[0039] In an embodiment, the annotation projector 216 can be configured to apply a lesion segmentation algorithm to generate the annotated MRI image data 208. For example, once a breast side with an abnormality or lesion is identified in the MG image, the one or more processors (e.g., the annotation projector 216) can be configured to automatically locate the most important image slice in the MRI image data representing the MRI image with the identified breast side. Further, the one or more processors can be configured to use the breast quadrant found in the MG image to locate the relevant quadrant region in the relevant MRI image slice. The one or more processors (e.g., the annotation projector 216) can be further configured to apply the lesion segmentation algorithm using only the relevant MRI image slice and region found in the MRI image data representing the MRI image. As a result, the annotation projector 216 can generate annotated MRI image data including the MRI image and annotations corresponding to the abnormalities identified in the MG image data.

[0040] The machine learning model 240 can be configured to receive a plurality of input images and produce model output data as a feature vector corresponding to each image, and transmit the model output data to a series of fully connected layers. The output data from the fully connected layers can represent a classification for the full study. In other words, if the study is a breast cancer diagnosis study for a particular patient, the model output data for the study can be a classification for the patient of one of a normal, benign, or malignant diagnosis. The model output data can be in the form of a confidence score or probability for each classification, representing the likelihood of the diagnosis corresponding to the normal, benign, and malignant classifications.

[0041] In an embodiment, the model 200 can include an image database 224 configured to store, receive, and provide image data to and from components within the model 200. For example, the image database 224 can include annotated MG image data 202 and MRI image data 206 captured by the image sensor 130 via the computing device 120, where the image database 224 is associated with the database 124 in the computing device 120. Figure 1

[0042] In an embodiment, the machine learning model 240 can be modified to include two separate image-level classifiers (e.g., one taking two bilateral CC images and the other taking two bilateral MLO images, or left and right images) or a separate image-level classifier for each input image.

[0043] In an embodiment, the annotations 214 can be provided to an annotation projector 216 to generate annotated MRI image data 208. The annotations 214 can include data corresponding to information collected about the annotated MG image data 208, which can be provided to the machine learning model 240 during training. The information about the annotations 214 is made or provided by a medical professional or computer program by appending relevant diagnostic information associated with the MG images.

[0044] ​In an embodiment, the model 200 can include a diagnostic model 230 configured to receive model output data from the machine learning model 240 to produce an overall diagnosis for the study. For example, model output data from the machine learning model 240 can be received at the diagnostic model 230 to generate diagnostic data associated with a confidence score. In an embodiment, the machine learning model 240 can be configured to receive annotated MG image data 202 to produce first model output data that, when processed by the diagnostic model 230, produces first diagnostic data corresponding to a first confidence score. In another embodiment, the machine learning model 240 can be configured to receive annotated MRI image data 208 to produce second model output data that, when processed by the diagnostic model 230, produces second diagnostic data corresponding to a second confidence score, where the second confidence score is greater than the first confidence score, indicating a more accurate diagnosis. Alternatively, depending on the parameters of the study, a second confidence score that is greater than a first confidence score can indicate a less accurate diagnosis.

[0045] In another embodiment, the machine learning model 240 can be pre-trained to produce per-image diagnoses to improve training speed.

[0046] The diagnostic model 230 can be implemented as any type of neural network that accepts time-series input of arbitrary length, such as a recurrent neural network (“RNN”), long short-term memory (“LSTM”), or a temporal convolutional network (“TCN”). The diagnostic model 230 can also be implemented as an ensemble classifier, such as a random forest, that takes multiple predictions produced by the machine learning model 240 described above and combines the predictions into a final diagnosis for the patient.

[0047] Figure 3 A set of annotated images according to an embodiment of the present application is depicted.

[0048] In an embodiment, one or more processors associated with the computing device 120 can be configured to receive MG image data representing one or more annotated images 310 of a patient’s breast. The one or more processors can be configured to identify one or more MG annotations 312, 314 in the one or more annotated MG images, where the one or more MG annotations 312, 314 can represent an abnormality at a first location in the one or more annotated MG images 310. The one or more annotated MG images 310 can be one of one or more view types that can be identified by a view type annotation. In this example, the view type annotation 316 is LCC, which corresponds to a cranio-caudal view of the left breast. The view type for MG images can be a 2-dimensional (2D) view, and the view type for MRI images can be a 3-dimensional (3D) view.

[0049] As described above, the one or more processors can be configured to determine a breast side and a quadrant of the patient's breast based on the first location in the one or more annotated MG images 310. The breast side (e.g., left "L" side or right "R" side) and the quadrant (e.g., upper-outer "UO", lower-outer "LO", upper-inner "UI", and lower-inner "LI") can be used to describe the location of the lesion or abnormality. The one or more annotations shown in the one or more MRI images 330 can be generated by the one or more processors by mapping 320 the one or more MG annotations 312, 314 to second locations of the corresponding breast side and quadrant of the one or more MRI images 330 represented in the MRI image data. Mapping 320 the one or more MG annotations 312, 314 to the one or more MRI images 330 at the corresponding locations of the abnormality results in generating the one or more MRI annotations 332, 334.

[0050] Further, the one or more processors associated with the computing device 120 can be configured to receive MRI image data representing one or more MRI images 330 of the patient's breast. The one or more processors can be configured to generate annotated MRI image data using the MRI image data and the one or more MG annotations 312, 314 identified in the one or more annotated MG images 310. The annotated MRI image data can include one or more MRI annotations 332, 334 representing the abnormality at the second location based at least in part on the first location. For example, the one or more MRI images 330 belong to a second view type identified by the view type annotation 336, which corresponds to an AX 3DDYN 0.9mm SUB with an axial 3D dynamic view scaled to 0.9 mm.

[0051] The second location of the visually identifiable abnormality by the MRI annotation 334 can be determined by converting the 2D coordinate information of the first location of the abnormality identified by the MG annotation 314 to 3D coordinate information.

[0052] MG images can be annotated using other annotation modes to describe the mass, calcification, and other imaging observations of the abnormality. For example, the mass of the abnormality can be annotated to describe the density, margin, shape, size, and count / number of the abnormality. The calcification of the abnormality can be annotated to describe the count / number, distribution, and stability of the abnormality. Other imaging observations can include architecture, distortion, related findings, overall breast appearance, composition, or special cases. The imaging observations can be determined based on the mass, calcification, and other imaging observations, while the anatomical entity can be determined as a result of also considering the location and laterality of the abnormality and its related features.

[0053] Figure 4 Another set of annotated images according to embodiments of the application are depicted.

[0054] In an embodiment, one or more processors associated with the computing device 120 can be configured to receive MG image data representing one or more annotated images 410 of a patient breast. The one or more processors can be configured to identify one or more MG annotations 412, 414 in the one or more annotated MG images, where the one or more MG annotations 412, 414 can represent an abnormality at a first location in the one or more annotated MG images 410. The one or more annotated MG images 410 can be one of one or more view types, which can be identified by a view type annotation (not shown in this example).

[0055] Further, one or more processors associated with the computing device 120 can be configured to receive MRI image data representing one or more MRI images 430 of a patient breast. The one or more processors can be configured to generate annotated MRI image data using the MRI image data and the one or more MG annotations 412, 414 identified in the one or more annotated MG images 410. The one or more processors can generate the annotated MRI image data by mapping 420 the one or more MG annotations 412, 414 to the one or more MRI images 430. Mapping annotations can include: identifying the one or more annotations; extracting the one or more annotations; and applying the one or more annotations to a location consistent with a location from which the one or more annotations were extracted. The annotated MRI image data can include one or more MRI annotations 432, 434 representing an abnormality at a second location based at least in part on the first location.

[0056] Figure 5 A flowchart depicting a machine learning model 500 for performing diagnostic assessments using annotated images in accordance with embodiments of the present application is depicted.

[0057] In an embodiment, the machine learning model 500 can receive MRI image data 510 corresponding to MRI images and / or annotated MRI images captured during a patient diagnostic exam. The machine learning model 500 can include an image classifier (e.g., a custom inception ResNetV2) 520 configured to process the MRI image data 510 and generate model output data 530 representing features of the MRI image data 510. The model output data 530 can also be a 2D feature map with 62x30x384 for each image.

[0058] In an embodiment, the result combination network 540 can be configured to combine the model output data 530 to produce binary probabilities. Further, the result combination network 540 can be configured to perform global average pooling by taking the 3-D feature vector and reducing the 3-D feature vector to one dimension by averaging the 62x30x384 dimensional feature vector for each image and concatenating the feature vectors to a 1536 one-dimensional feature vector. The result combination network 540 can also include a dense layer as a fully connected layer, where each input from the concatenation layer will be connected to the fully connected layer. A Softmax takes the output from the fully connected layer and generates binary probabilities. The binary probabilities can be determined using a Softmax function, as described herein. The machine learning model 500 can also be configured to output a prediction for each image view provided in the MRI image data 510. Alternatively, the Softmax function can be configured to generate a 6-dimensional vector consistent with the BIRAD protocol, where the 6-dimensional vector can include a ranking between zero (0) and six (6), depending on the severity of the diagnosis consistent with the BIRADS protocol.

[0059] Figure 6 A process flow for a machine learning model 600 for performing diagnostic evaluations using annotated images, according to embodiments of the present application, is depicted.

[0060] In an embodiment, the machine learning model 600 can include an image-level classifier (e.g., a custom inception ResNetV2) 620 configured to process the MRI image data 610 and generate model output data 630 representing features of the MRI image data 610. The model output data 630 can also be a 2-D feature map providing a classification for each image, where the classification can be one of various diagnoses. For example, the model output data 630 can provide a classification for each image corresponding to a benign diagnosis or a malignant diagnosis. A result combination network 640 can be configured to receive the model output data 630 and can then be configured to concatenate and reduce the model output data 630 to a single feature vector by a dense function, which can then be provided to a Softmax function as described above.

[0061] Figure 7 A flowchart of a computer-implemented method 700 for annotating MRI images, according to embodiments of the present application, is depicted.

[0062] In an embodiment, the computer-implemented method 700 can include one or more processors configured to receive 702 annotated MG image data 202 representing one or more annotated MG images of a patient breast. The one or more processors can be associated with a computing device 120, 220 and configured to receive 702 the MG image data representing one or more annotated MG images of a patient breast from a database 124 via a network 110.

[0063] Further, the method 700 can include one or more processors configured to identify 704 one or more annotations in the one or more annotated MG images, the one or more annotations representing an abnormality at a first location in the one or more annotated MG images. The annotations can be identified as textual or numerical characters displayed on the one or more annotated MG images. The annotations can represent an abnormality or lesion identified within the one or more MG images. The annotations can include geometric information describing the location or characteristics of the abnormality or lesion. The annotations can also identify a view type of the one or more MG images or identify a specific region of the patient breast in which the abnormality or lesion is located. For example, the one or more annotations can include a breast side of the patient breast and a view type (e.g., L CC - left craniocaudal view). Further, the one or more annotations can include quadrants (e.g., upper-outer “UO,” lower-outer “LO,” upper-inner “UI,” and lower-inner “LI”) used to describe a region of the patient breast that includes the lesion or abnormality.

[0064] Further, the method 700 can further include one or more processors configured to receive 706 MRI image data representing one or more MRI images of the patient breast. The one or more processors can be configured to generate annotated MRI image data using the MRI image data and the one or more MG annotations identified in the one or more annotated MG images. The annotated MRI image data can include one or more MRI annotations representing an abnormality at a second location based at least in part on the first location. By converting 2D coordinate information of the first location of the abnormality identified by the MG annotations 314 to 3D coordinate information, a second location of the abnormality identified by the MRI annotations 334 can be determined. For example, if the first location is determined to be an X coordinate and a Y coordinate in a 2D image mapping, then the second location can be determined to be an X’ coordinate, a Y’ coordinate, and a Z coordinate in a 3D image mapping, where X’ is a 3D representation of the X coordinate, the Y’ coordinate is a 3D representation of the Y coordinate, and the Z coordinate represents a dimension coordinate added from the 2D coordinates of the first location to the 3D coordinates of the second location in the mapping.

[0065] The one or more MRI images can be of a second view type corresponding to a first view type that includes information representative of image scaling.

[0066] Further, the method 700 can further include one or more processors configured to generate 708 annotated MRI image data using the MRI image data and the one or more annotations identified in the one or more annotated MG images, the annotated MRI image data including one or more annotations representative of the abnormality at a second location based at least in part on the first location.

[0067] Further, the method 700 can further include one or more processors configured to store 710 the annotated MRI image data in a database.

[0068] Figure 8 A block diagram of a computing device of a distributed computing environment in accordance with an embodiment of the present application is depicted. Figure 8 A block diagram of a computing device 800 suitable for use with the server(s) 125 and computing device 120 in accordance with exemplary embodiments of the present application is depicted. It should be understood that Figure 8 Only one implementation is provided for illustration and no inference is to be drawn concerning any implied limitations on the environment in which different embodiments can be implemented. Numerous modifications can be made to the depicted environments.

[0069] The computing device 800 includes a communication structure 802 that provides communication between a cache 816, a memory 806, a persistent storage 808, a communication unit 810, and input / output (I / O) interface(s) 812. The communication structure 802 can be implemented with any architecture designed for passing data and / or control information between the processor(s), such as microprocessors, communication and network processors, system memory, peripheral devices, and any other hardware components within a system. For example, the communication structure 802 can be implemented with one or more buses or cross-over switches.

[0070] The memory 806 and the persistent storage 808 are computer readable storage media. In this embodiment, the memory 806 includes a random access memory (RAM). Generally, the memory 806 can include any suitable volatile or non-volatile computer readable storage media. The cache 816 is a fast memory that enhances the performance of the computer processor(s) 804 by providing rapid access to frequently accessed data and instructions.

[0071] The programs can be stored in the permanent storage device 808 and memory 806 for execution by one or more of the respective computer processors 804 via cache 816. In an embodiment, the permanent storage device 808 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, the permanent storage device 808 can include a solid state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer readable storage media that is capable of storing program instructions or digital information.

[0072] The media used by the permanent storage device 808 can also be removable. For example, a removable hard drive can be used for permanent storage device 808. Other examples include optical and magnetic disks, thumb drives, and smart cards, which are inserted into a drive for transfer onto another computer readable storage media, also a portion of permanent storage 808.

[0073] In these examples, communication unit 810 provides

[0074] I / O interface(s) 812 allow for input and output of data with other devices utilizing I / O interface(s) 812. For example, I / O interface 812 can provide a connection to external device(s) 818 such as a image sensor 130, a keyboard, a keypad, a touch screen, and / or some other suitable input device. External device(s) 818 can also include portable computer readable storage media such as, for example, a thumb drive, a portable optical or magnetic disk, and a memory card. Software and data 814 used to practice embodiments of the present application can be stored on such portable computer readable storage media and can be loaded onto permanent storage device 808 via I / O interface(s) 812. I / O interface(s) 812 also connect to display 820.

[0075] Display 820 provides a mechanism to display data to a user and can be, for example, a computer monitor.

[0076] Software and data 814 described herein is identified based on the application for which the embodiments of the present application are implemented. However, it is to be understood that any particular

[0077] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0078] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or raised structures in grooves of a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0079] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for execution by the computing device. The network adapter can buffer the computer readable program instructions during the time the computer readable program instructions are being received from the network and being forwarded to the computer readable storage medium within the respective computing / processing device.

[0080] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0081] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0082] These computer readable program instructions can be provided to a processor of a computer, or other programmable data processing apparatus, to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including

[0083] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0084] The computer program product of the different embodiments of the application can have the form of a non-transitory computer readable medium having stored thereon the computer program. The computer readable medium can include a solid-state storage media or storage device, such as a memory or memories in the electronic device. Furthermore, the computer readable medium can include a transitory computer readable medium, such as signals on a modulated light carrier or signals on a modulated radio frequency carrier, or the like.

[0085] The description of the different embodiments of the application has been presented for purposes of illustration and description but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the application. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application, or technical improvements found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for annotating magnetic resonance imaging (MRI) images, comprising: receiving, by one or more processors, mammogram (MG) image data representing one or more annotated MG images of a patient breast; identifying, by the one or more processors, one or more MG annotations in the one or more annotated MG images, the one or more MG annotations representing an abnormality at a first location in the one or more annotated MG images; receiving, by the one or more processors, MRI image data representing one or more MRI images of the patient breast; generating, by the one or more processors, annotated MRI image data using the MRI image data and the one or more MG annotations identified in the one or more annotated MG images, the annotated MRI image data including one or more MRI annotations representing the abnormality at a second location based at least in part on the first location, the first and second locations corresponding to a breast side and a quadrant of the patient breast; and storing, by the one or more processors, the annotated MRI image data in a database.

2. The computer-implemented method of claim 1, wherein each of the one or more annotated MG images is one or either of a craniocaudal (CC) view or a mediolateral oblique (MLO) view.

3. The computer-implemented method of claim 1, wherein the one or more MRI images include a plurality of image slices, the plurality of image slices including one or more image slices having the abnormality.

4. The computer-implemented method of claim 3, further comprising: determining, by the one or more processors, a breast side and a quadrant of the patient breast based on the first location in the one or more annotated MG images.

5. The computer-implemented method of claim 4, further comprising: locating, by the one or more processors, a corresponding breast side and quadrant in the plurality of image slices of the one or more MRI images.

6. The computer-implemented method of claim 5, wherein generating the annotated MRI image data includes: mapping, by the one or more processors, the one or more MG annotations to the second location of the corresponding breast side and quadrant of the MRI images represented in the MRI image data.

7. The computer-implemented method of claim 1, further comprising: providing, by the one or more processors, the annotated MRI image data to a supervised machine learning model; and processing, by the one or more processors, the annotated MRI image data with the one or more MRI annotations using the supervised machine learning model to generate output data corresponding to a diagnosis.

8. A computer program product for annotating magnetic resonance imaging (MRI) images, the computer program product comprising: ​ program instructions executable by a processor to cause the processor to perform the method of any one of claims 1-7.

9. A computer system for annotating magnetic resonance imaging (MRI) images, the computer system comprising: one or more computer processors; program instructions for execution by at least one of the one or more processors, the program instructions executable by a processor to cause the processor to perform the method of any one of claims 1-7.

Images