Method, computer program, and computer system (improving MRI annotation)

Abstract

To provide a method of annotating a magnetic resonance imaging (MRI) image.SOLUTION: A method includes receiving mammogram (MG) image data representing annotated MG images of a patient breast. The annotated MG images being one of a craniocaudal view or of a mediolateral oblique view. The method includes: identifying annotations representing an abnormality at a first location in the annotated MG images; receiving MRI image data representing MRI images of the patient breast; generating annotated MRI image data representing an abnormality using the MRI image data and the annotations identified in the annotated MG images; and storing the annotated MRI image data in a database.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to the field of medical image annotation, and in particular to projecting annotations of mammogram (MG) images onto magnetic resonance imaging (MRI) images. [Background technology]

[0002] When screening a patient for breast cancer, medical professionals capture and review multiple images taken at different perspectives and time points to make a diagnosis. Medical professionals use mammograms to help identify early signs of breast cancer. A mammogram is an x-ray of the chest and can be used as a screening mammogram to check for breast cancer in women who have no signs or symptoms of the disease. To aid in diagnosis, computer-aided diagnosis (CAD) algorithms are often used to make decisions based on a single image. Images taken during a mammogram can include different perspectives or angles for the left and right breast. For example, standard mammogram image formats can include bilateral craniocaudal (CC), bilateral mediolateral oblique (MLO), where bilateral refers to the left and right versions of the shot.

[0003] Mammography has become a more common and cost-effective imaging modality for diagnosing breast cancer, but it is limited in that it cannot identify all types of abnormalities, especially in the case of hard tissues, whereas MRI can identify abnormalities that are less noticeable in MG examinations, but it is more computationally expensive, time-consuming, and requires a higher level of skill. Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to provide a technique for annotating Magnetic Resonance Imaging (MRI) images. [Means for solving the problem]

[0005] The present invention is described in various embodiments that disclose methods, computer program products, and computer systems for annotating Magnetic Resonance Imaging (MRI) images with mammogram annotations. One embodiment of the present disclosure is a computer-implemented method for annotating MRI images, the computer-implemented method may be executed by one or more processors configured to: receive mammogram (MG) image data representing one or more annotated MG images of a patient's chest; identify one or more MG annotations in the one or more annotated MG images, where the one or more MG annotations represent an abnormality at a first location in the one or more annotated MG images; receive MRI image data representing one or more MRI images of the patient's chest; generate annotated MRI image data using the MRI image data and the one or more MG annotations identified in the one or more annotated MG images, where the annotated MRI image data includes one or more MRI annotations representing an abnormality at a second location based at least in part on the first location; and store the annotated MRI image data in a database.

[0006] In one embodiment, each of the one or more annotated MG images is either a craniocaudal (CC) view or a mediolateral oblique (MLO) view. The one or more MRI images can include multiple image slices, including one or more image slices with an abnormality.

[0007] Some embodiments may include one or more processors configured to determine a chest left / right side and quadrant of the patient chest based on a first location in the one or more annotated MG images and identify a corresponding chest left / right side and quadrant in multiple image slices of the one or more MRI images. Additionally, the one or more processors configured to generate the annotated MRI image data may be further configured to map the one or more annotations to a second location of the corresponding chest left / right side and quadrant of the MRI image represented in the MRI image data. Additionally, the one or more processors may be configured to provide the annotated MRI image data to a supervised machine learning model and process the annotated MRI image data with the one or more annotations using the supervised machine learning model to generate output data corresponding to a diagnosis. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a distributed data processing environment for annotating MRI images, according to an embodiment of the present invention. [Diagram 2] FIG. 1 shows an architecture diagram of a research-level image model for annotating MRI images for diagnostic evaluation, according to an embodiment of the present invention. [Diagram 3] 1 illustrates a set of annotated images according to one embodiment of the present invention. [Figure 4] 1 illustrates another set of annotated images according to an embodiment of the present invention. [Diagram 5] 1 shows a flowchart of a machine learning model used to perform diagnostic assessment using annotated images, according to an embodiment of the present invention. [Figure 6] 1 shows a flowchart of another machine learning model used to perform diagnostic assessment using annotated images, according to an embodiment of the present invention. [Figure 7]1 shows a flowchart of steps of a computer-implemented method for annotating MRI images, according to one embodiment of the present invention. [Figure 8] 2 illustrates a block diagram of a computing device in the distributed data processing environment of FIG. 1, in accordance with one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention addresses the problem of efficiently annotating MRI images. When screening a patient for breast cancer, a mammographer reviews multiple images taken from different perspectives during one or more pre-exams before arriving at a diagnosis. A medical professional performing a breast cancer screening exam can capture an X-ray image, called a screening MG. The screening MG image can be annotated by the medical professional to identify relevant findings (such as abnormalities, lesions, tumors, etc.) in the image or to indicate useful information about the relevant findings. The annotations can be stored within the MG image or can be stored elsewhere and associated with the MG image.

[0010] Accordingly, the present invention provides a computer-implemented method for efficiently and automatically annotating MRI images with annotated MG images. In implementation, a computing device including one or more processors can be configured to receive MG image data representative of one or more annotated MG images of a patient's chest from a database.

[0011] Each of the one or more annotated MG images is either a craniocaudal (CC) view or a medial-lateral oblique (MLO) view. Each image taken during MG includes a corresponding view format. For example, the view format of an image can include a standard view format or a non-standard view format of a mammogram, where the standard view format is generally taken during a screening examination and the non-standard view format is generally taken during a follow-up diagnostic examination. The standard view format can be taken during a screening examination and can include a bilateral CC view or a bilateral MLO view, where the bilateral view format can include positional variants representing a left (L) position or a right (R) position, and is indicated as a CC L view, a CC R view, a MLO L view, and a MLO R view. Non-standard image formats may be captured during a diagnostic exam and may include one of a mediolateral (ML) view, a lateral medial (LM) view, a lateral medial oblique (LMO) view, a late mediolateral (ML) view, a step oblique (SO) view, a spot view, a spot compression view, a double spot compression view, a magnificent view, an exaggerated craniocaudal (XCC) view, an axillary view, a cleavage view, a tangential view, a CC view, a bull's eye CC view, a roll CC view, an elevated CC view, a 20° oblique view, an inferior medial superior lateral oblique view, and an Eklund technique view. A non-standard image may correspond to a standard image such that the non-standard image is a variation of one or more of the standard images.

[0012] The one or more MRI images can include multiple image slices, including one or more image slices having an abnormality. In one embodiment, the one or more processors can be configured to apply a lesion segmentation algorithm using only the associated found image slices and regions identified as having an abnormality.

[0013] The one or more processors can be configured to identify one or more annotations representative of an abnormality at a first location in the one or more annotated MG images, and to receive MRI image data representative of an MRI image of the patient's chest.

[0014] The one or more processors can be configured to generate annotated MRI image data using the MRI image data and the one or more annotations identified in the one or more annotated MG images. The annotated MRI image data can include one or more annotations representative of an anomaly at a second location based at least in part on the first location.

[0015] The one or more processors configured to generate the annotated MRI image data may be further configured to map the one or more annotations to a second location of a corresponding left or right chest and quadrant of the MRI image represented in the MRI image data. For example, the patient's chest may be identified by the left or right chest and quadrant in which an abnormality may be found.

[0016] The one or more processors may be configured to store the annotated MRI image data in a database according to methods for storing data in a database known to those of skill in the art.

[0017] The one or more processors can be configured to provide the annotated MRI image data to a supervised machine learning model and process the annotated MRI image data with the one or more annotations using the supervised machine learning model to generate output data corresponding to a diagnosis.

[0018] Embodiments of the present invention provide systems and methods that process MG image data from a standard set of images or a diagnostic set of images, or both, using study-level image classifiers and diagnostic models to determine a diagnostic assessment.

[0019] An embodiment of the present invention provides a study-level image classifier for processing a standard or diagnostic set of mammograms, or both, to determine a classification (e.g., benign, malignant, normal) for each image in the study and generate model output data as a feature vector for each image. The study-level image classifier may include a deep convolutional neural network ("CNN") for image classification, which may be a rule-based machine learning model that combines a discovery component (e.g., genetic algorithm) and a learning component (e.g., supervised or unsupervised learning). In general, image classifier systems attempt to identify a set of context-dependent rules that collectively store and apply knowledge in a piecewise manner to make predictions (e.g., behavioral modeling, classification, data mining, regression, function approximation, or game strategies). The study-level image classifier may also include a combinatorial network component configured to concatenate model output data feature vectors through a series of fully connected layers to generate study-level classifications. The research-level image classifier can be trained with images tagged with medical labels or other images associated with annotations or diagnostic reports or both to improve the capabilities and accuracy of the image classifier.

[0020] In another embodiment, the research level image classifier may include two separate image level classifiers, where a first image level classifier receives the two CC (i.e., CC L and CC R) MG images and a second image level classifier receives the two MLO (i.e., MLO L and MLO R) MG images. Additionally, the research level image classifier may include an image level classifier for each of the four input images.

[0021] An embodiment of the invention may be configured to output a prediction for the entire study, where the prediction may represent a benign or malignant diagnosis. The system may also be configured to output a prediction for the left or right breast. The system may also be configured to sort the results according to the Breast Imaging Reporting and Data System (BI-RADS), which sorts the results into categories numbered zero (0) to six. The system may also be configured to process the image data for each side separately and then combine the results by an average or maximum operation. The image classifier of the system may be pre-trained to generate a diagnosis for each image, improving the speed of training.

[0022] Additionally, embodiments of the present invention provide a diagnostic model that generates an overall diagnosis for a patient based on the output data from the study-level image classifier.

[0023] The invention will now be described in detail with reference to the figures.

[0024] FIG. 1 illustrates a block diagram of a distributed data processing environment 100 for annotating MG images, according to an embodiment of the present invention. FIG. 1 merely provides an illustration of an embodiment of the present invention and does not imply any limitation with respect to the environments in which different embodiments may be implemented. In the illustrated embodiment, the distributed data processing environment 100 includes a computing device 120, a server 125, a database 124, and an image sensor 130 interconnected over a network 110. The network 110 operates as a computing network, which may be, for example, a local area network (LAN), a wide area network (WAN), or a combination of the two, and may include wired, wireless, or fiber optic connections. In general, the network 110 may be any combination of connections and protocols that support communication between the computing device 120, the server 125, the database 124, and the image sensor 130. The distributed data processing environment 100 may also include additional servers, computers, sensors, or other devices not shown.

[0025] The computing device 120 is operable to execute at least a portion of a computer program for projecting the MG image annotations onto the MRI image and performing a diagnostic evaluation. In one embodiment, the computing device 120 may be communicatively coupled to the image sensor 130, or the image sensor 130 may be one of the components of the computing device 120. The computing device 120 is configured to transmit, receive, or transmit and receive data from the network 110 and the image sensor 130. In some embodiments, the computing device 120 may be an administrative server, a web server, or any other electronic device or computing system capable of transmitting and receiving data. In some embodiments, the computing device 120 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a smartphone, or any programmable electronic device capable of communicating with the database 124, the server 125 via the network 110. The computing device 120 may include components as described in more detail in FIG. 8.

[0026] Computing device 120 may also be configured to receive, store, and process images captured by image sensor 130. For example, computing device 120 may be communicatively coupled to image sensor 130 and receive data corresponding to images captured by image sensor 130 over a communications link. Computing device 120 may be configured to store the image data in a memory of computing device 120 or transmit the image data over network 110 to database 124 or server 125. The image data may be processed by one or more processors of computing device 120 or by one or more processors associated with server 125 in a cloud computing network.

[0027] The database 124 acts as a repository of data flowing to and from the network 110. Examples of data include image data, annotated image data, user data, device data, network data, and data corresponding to images captured by the image sensor 130. A database is an organized collection of data. The database 124 can be implemented in any type of storage device, such as a database server, hard disk drive, flash memory, etc., capable of storing data and configuration files that can be accessed and used by the computing device 120. In one embodiment, the database 124 is accessed by the computing device 120 to store data corresponding to images captured by the image sensor. In another embodiment, the database 124 is accessed by the computing device 120 to access the annotated image data, user data, device data, network data, and data corresponding to images captured by the image sensor 130. In another embodiment, the database 124 may reside elsewhere in the distributed network environment 100, provided that the database 124 is accessible to the network 110.

[0028] Server 125 may be a standalone computing device, an administration server, a web server, or any other electronic device or computing system capable of receiving, transmitting, and processing data and communicating with computing device 120 over network 110. In other embodiments, server 125 represents a server computing system that uses multiple computers as a server system, such as a cloud computing environment. In yet other embodiments, server 125 represents a computing system that uses clustered computers and components (e.g., database server computers, application server computers, etc.) that function as a single pool of seamless resources when accessed within distributed data processing environment 100. Server 125 may include components as described in further detail in FIG. 8.

[0029] Although the implementation of research-level image model 200 is described above and illustrated in FIG. 2, the disclosure is not limited thereto. In at least some embodiments, model 200 can implement a trained component or trained model configured to perform the processing described above with respect to research-level image model 200. The trained component can include one or more machine learning models 240, including but not limited to one or more classifiers, one or more neural networks, one or more probability graphs, one or more decision trees, etc. In other embodiments, the trained component can include a rule-based engine, one or more statistical-based algorithms, one or more mapping functions, or other forms of functions / algorithms to determine whether a natural language input is a compound natural language input or a non-compound natural language input. In some embodiments, the trained component can be configured to perform a binary classification that can classify a natural language input into one of two classes / categories. In some embodiments, the trained component can be configured to perform a multi-class or multinomial classification that can classify a natural language input into one of three or more classes / categories. In some embodiments, the trained component can be configured to perform multi-label classification that can associate natural language input with two or more classes / categories.

[0030] A variety of machine learning techniques can be used to train and operate the trained components to perform the various processes described herein. Models can be trained and operated according to a variety of machine learning techniques. Such techniques can include, for example, neural networks (such as deep neural networks and / or recurrent neural networks), inference engines, trained classifiers, and the like. Examples of trained classifiers include support vector machines (SVMs), neural networks, decision trees, AdaBoost (short for Adaptive Boosting) combined with decision trees, and random forests. Taking SVMs as an example, SVMs are supervised learning models with associated learning algorithms that analyze data to recognize patterns in the data, and are often used in classification and regression analysis. Given a set of training examples, each marked as belonging to one of two categories, the SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear classifier. More complex SVM models can be built using training sets that identify more than two categories, and the SVM determines which category is most similar to the input data. An SVM model can map examples of distinct categories so that they are separated by a distinct gap. A new example is then mapped into the same space and predicted to belong to a category based on which side of the gap it falls on. The classifier can emit a "score" that indicates which category the data best matches. The score provides an indication of how closely the data matches that category.

[0031] In order to apply machine learning techniques, the machine learning process itself needs to be trained. To train the machine learning component, a "ground truth" of training examples needs to be established. In machine learning, the term "ground truth" refers to the classification accuracy of a training set in supervised learning techniques. A variety of techniques can be used to train a model, including backpropagation, statistical learning, supervised learning, semi-supervised learning, probabilistic learning, or other known techniques.

[0032] 2 illustrates an architecture diagram of a research-level image model 200 for annotating MRI images for diagnostic evaluation, according to an embodiment of the present invention. The model 200 can include a machine learning model 240 configured to receive annotated MRI image data 208 or annotated MG image data 202 from an image database 224, or both. The annotated MRI image data 208 can be generated by a computing device 220 that includes one or more processors configured to identify and extract annotations (e.g., using annotation identifier and extractor 212). The computing device 220 (e.g., depicted as computing device 120 in FIG. 1) can include one or more processors configured to project or map one or more annotations 214 onto the MRI image represented by the MRI image data 206 (e.g., using annotation projector 216). Further, the computing device 220 may include one or more processors configured to generate annotated MRI image data 208 based on the MRI image data 206 and one or more annotations 214 identified and extracted from the annotated MG image represented by the annotated MG image data 202.

[0033] In one embodiment, the annotation projector 216 can be configured to apply a lesion segmentation algorithm to generate the annotated MRI image data 208. For example, once a left or right breast having an abnormality or lesion is identified in the MG image, the one or more processors (e.g., annotation projector 216) can be configured to automatically identify the most significant image slices of the identified left or right breast in the MRI image data representing the MRI image. Furthermore, the one or more processors can be configured to identify the relevant quadrant regions in the relevant MRI image slices using the breast quadrants found in the MG image. The one or more processors (e.g., annotation projector 216) can be further configured to apply the lesion segmentation algorithm using only the relevant MRI image slices and regions found in the MRI image data representing the MRI image. As a result, the annotation projector 216 can generate annotated MRI image data including the MRI image and annotations corresponding to the abnormalities identified in the MG image data.

[0034] The machine learning model 240 can be configured to receive a number of input images, generate model output data as feature vectors corresponding to each image, and transmit the model output data to a series of fully connected layers. The output data from the fully connected layers can represent a classification for the complete study. In other words, if the study is a breast cancer diagnostic study for a particular patient, the model output data for the study can be a classification for the patient: normal, benign, or malignant. The model output data can be in the form of a confidence score or probability for each classification, representing the likelihood of the diagnosis corresponding to the normal, benign, or malignant classification.

[0035] In one embodiment, model 200 can include an image database 224 configured to store, receive, and provide image data to and from components in model 200. For example, image database 224 can include annotated MG image data 202 and MRI image data 206 captured by image sensor 130 via computing device 120, where image database 224 correlates with database 124 of FIG.

[0036] In one embodiment, the machine learning model 240 may be modified to include two separate image level classifiers, e.g., one taking two bilateral CC images and the other taking two bilateral MLO images, or a classifier taking a left image and a right image, or a separate image level classifier for each input image.

[0037] In one embodiment, the annotations 214 can be provided to an annotation projector 216 to generate the annotated MRI image data 208. The annotations 214 can include data corresponding to information collected about the annotated MG image data 202 that can be provided to the machine learning model 240 during training. The information related to the annotations 214 can be created or provided by a medical professional or a computer program by appending relevant diagnostic information associated with the MG image.

[0038] In one embodiment, the model 200 can include a diagnostic model 230 configured to receive model output data from the machine learning model 240 and generate an overall diagnosis for the study. For example, the model output data from the machine learning model 240 can be received in the diagnostic model 230 and generate diagnostic data associated with a confidence score. In one embodiment, the machine learning model 240 can be configured to receive the annotated MG image data 202 and generate first model output data, which when processed by the diagnostic model 230 generates first diagnostic data corresponding to the first confidence score. In another embodiment, the machine learning model 240 can be configured to receive the annotated MRI image data 208 and generate second model output data, which when processed by the diagnostic model 230 generates second diagnostic data corresponding to a second confidence score, which is greater than the first confidence score and indicates a more accurate diagnosis. Alternatively, a second confidence score greater than the first confidence score may indicate a less accurate diagnosis depending on the parameters of the study.

[0039] In another embodiment, the machine learning model 240 can be pre-trained to generate a diagnosis for each image to improve training speed.

[0040] The diagnostic model 230 can be implemented as any form of neural network that accepts time series inputs of any length, such as a recurrent neural network ("RNN"), a long short-term memory ("LSTM"), or a temporal convolutional network ("TCN"). The diagnostic model 230 can also be implemented as an ensemble classifier, such as a random forest, that takes multiple predictions generated by the machine learning model 240 described above and combines the predictions into a final diagnosis for the patient.

[0041] FIG. 3 illustrates a set of annotated images according to one embodiment of the present invention.

[0042] In one embodiment, one or more processors associated with the computing device 120 may be configured to receive MG image data representing one or more annotated images 310 of a patient's chest. The one or more processors may be configured to identify one or more MG annotations 312, 314 in the one or more annotated MG images, which may represent an abnormality at a first location in the one or more annotated MG images 310. The one or more annotated MG images 310 may be one of one or more imaging modalities that may be identified by an imaging modality annotation. In this example, the imaging modality annotation 316 is L CC, which corresponds to a craniocaudal view of the left chest. The imaging modality of the MG images may be two-dimensional (2D) imaging, and the imaging modality of the MRI images may be three-dimensional (3D) imaging.

[0043] As described herein above, the one or more processors can be configured to determine a chest left / right side and quadrant of the patient's chest based on a first location in the one or more annotated MG images 310. The chest left / right side (e.g., left "L" side or right "R" side) and quadrant (e.g., superior lateral "UO", inferior lateral "LO", superior medial "UI", and inferior medial "LI") can be used to describe the location of the lesion or abnormality. The one or more annotations shown in the one or more MRI images 330 can be generated by the one or more processors mapping 320 the one or more MG annotations 312, 314 to a second location of the corresponding chest left / right side and quadrant of the one or more MRI images 330 represented in the MRI image data. One or more MG annotations 312, 314 are mapped 320 to one or more MRI images 330 at the locations of the corresponding abnormalities, resulting in one or more MRI annotations 332, 334.

[0044] Further, one or more processors associated with the computing device 120 may be configured to receive MRI image data representing one or more MRI images 330 of the patient's chest. The one or more processors may be configured to generate annotated MRI image data using the MRI image data and one or more MG annotations 312, 314 identified in the one or more annotated MG images 310. The annotated MRI image data may include one or more MRI annotations 332, 334 representing an abnormality at a second location based at least in part on the first location. For example, the one or more MRI images 330 are of a second imaging modality identified by an imaging modality annotation 336 of AX 3D DYN 0.9 mm_SUB corresponding to an axial 3D dynamic imaging with a scale of 0.9 mm.

[0045] A second location of the abnormality visually identifiable by the MRI annotations 334 can be determined by converting the 2D coordinate information of the first location of the abnormality identified by the MG annotations 314 into 3D coordinate information.

[0046] Other annotation schemata can be used to annotate MG images to describe the mass, calcification, and other imaging observations of the abnormality. For example, the mass of the abnormality can be annotated to describe the density, margin, shape, size, and count / number of the abnormality. The calcification of the abnormality can be annotated to describe the count / number, distribution, and stability of the abnormality. Other imaging observations include structure, distortion, associated findings, overall breast appearance, composition, or special cases. The imaging observations can be confirmed based on the mass, calcification, and other imaging observations, while the anatomical entity can be determined as a result of also considering the location and laterality of the abnormality and its associated characteristics.

[0047] FIG. 4 illustrates another set of annotated images according to one embodiment of the present invention.

[0048] In one embodiment, one or more processors associated with the computing device 120 can be configured to receive MG image data representing one or more annotated images 410 of a patient's chest. The one or more processors can be configured to identify one or more MG annotations 412, 414 in the one or more annotated MG images, where the one or more MG annotations 412, 414 can be representative of an abnormality at a first location in the one or more annotated MG images 410. The one or more annotated MG images 410 can be one of one or more imaging modalities, which can be identified by an imaging modality annotation, not shown in this example.

[0049] Further, one or more processors associated with the computing device 120 may be configured to receive MRI image data representing one or more MRI images 430 of the patient's chest. The one or more processors may be configured to generate annotated MRI image data using the MRI image data and one or more MG annotations 412, 414 identified in the one or more annotated MG images 410. The one or more processors may generate the annotated MRI image data by mapping 420 the one or more MG annotations 412, 414 to the one or more MRI images 430. Mapping the annotations may include identifying one or more annotations, extracting the one or more annotations, and applying the one or more annotations to a location that coincides with the location from which the one or more annotations were extracted. The annotated MRI image data may include one or more MRI annotations 432, 434 representing an abnormality at a second location based at least in part on the first location.

[0050] FIG. 5 shows a flow chart of a machine learning model 500 used to perform diagnostic assessment using annotated images, according to an embodiment of the present invention.

[0051] In one embodiment, the machine learning model 500 may receive MRI image data 510 corresponding to an MRI image captured during a diagnostic exam of a patient, or annotated MRI images, or both. The machine learning model 500 may include an image classifier (e.g., Custom Inception ResNetV2) 520 configured to process the MRI image data 510 and generate model output data 530 representing features of the MRI image data 510. The model output data 530 may be a 2D feature map having 62x30x384 for each image.

[0052] In one embodiment, the outcome combination network 540 can be configured to combine the model output data 530 to generate a binary probability. Additionally, the outcome combination network 540 can be configured to perform global average pooling by taking the 3D feature vectors, reducing the 3D feature vectors to one dimension by averaging the 62×30×384 dimensional feature vectors of each image, and concatenating the feature vectors into 1536 one-dimensional feature vectors. The outcome combination network 540 can also include a dense layer, which is a fully connected layer, where each input from the concatenation layer is connected to the fully connected layer. A Softmax takes the output from the fully connected layer and generates a binary probability. The binary probability can be determined using a softmax function as described herein. The machine learning model 500 can also be configured to output a prediction for each image shot provided in the MRI image data 510. Alternatively, the softmax function can be configured to generate a six-dimensional vector consistent with the BIRADS protocol, where the six-dimensional vector can include a ranking between zero (0) and six according to the severity of the diagnosis along the BIRADS protocol.

[0053] FIG. 6 illustrates a process flow of a machine learning model 600 used to perform diagnostic assessment using annotated images, according to an embodiment of the present invention.

[0054] In one embodiment, the machine learning model 600 can include an image level classifier (e.g., Custom Inception ResNetV2) 620 configured to process MRI image data 610 and generate model output data 630 representing features of the MRI image data 610. The model output data 630 can also be a 2D feature map providing a classification for each image, where the classification can be one of various diagnoses. For example, the model output data 630 can provide a classification for each image corresponding to a benign or malignant diagnosis. The result combination network 640 can be configured to receive the model output data 630 and then configured to concatenate and reduce the model output data 630 into a single feature vector by a dense function, which can then be provided to a softmax function as described above.

[0055] FIG. 7 shows a flowchart of a computer-implemented method 700 for annotating MRI images, according to one embodiment of the present invention.

[0056] In one embodiment, the computer-implemented method 700 can include one or more processors configured to receive 702 annotated MG image data 202 representing one or more annotated MG images of the patient's chest. The one or more processors can be associated with the computing device 120, 220 and configured to receive 702 the MG image data representing the one or more annotated MG images of the patient's chest from the database 124 via the network 110.

[0057] Further, the method 700 may include one or more processors configured to identify (704) one or more annotations in the one or more annotated MG images, the one or more annotations representing an abnormality at a first location in the one or more annotated MG images. The annotations may be identified as text or numbers displayed on the one or more annotated MG images. The annotations may represent an abnormality or lesion identified in the one or more MG images. The annotations may include geometric information describing a location or characteristics of the abnormality or lesion. The annotations may also identify an imaging type of the one or more MG images, or may identify a particular region of the patient's chest in which the abnormality or lesion is located. For example, the one or more annotations may include a left-right chest and imaging type (e.g., L CC, i.e., left craniocaudal imaging) of the patient's chest. Additionally, the one or more annotations may include a quadrant (e.g., superior lateral "UO", inferior lateral "LO", superior medial "UI", and inferior medial "LI") used to describe the area of ​​the patient's chest that contains the lesion or abnormality.

[0058] Additionally, the method 700 may further include one or more processors configured to receive MRI image data representative of one or more MRI images of the patient's chest (706). The one or more processors may be configured to generate annotated MRI image data using the MRI image data and one or more MG annotations identified in the one or more annotated MG images. The annotated MRI image data may include one or more MRI annotations representative of the anomaly at a second location based at least in part on the first location. The second location of the anomaly identified by the MRI annotations 334 may be determined by converting 2D coordinate information of the first location of the anomaly identified by the MG annotations 314 to 3D coordinate information. For example, if a first location is determined to be at an X coordinate and a Y coordinate of the 2D image map, then a second location can be determined to be at an X' coordinate, a Y' coordinate, and a Z coordinate of the 3D image map, where X' is a 3D representation of the X coordinate, the Y' coordinate is a 3D representation of the Y coordinate, and the Z coordinate represents an additional dimension of coordinates by mapping the 2D coordinates of the first location to the 3D coordinates of the second location.

[0059] The one or more MRI images may be of a second imaging modality corresponding to the first imaging modality that includes information representative of image scale.

[0060] Additionally, the method 700 may further include one or more processors configured to generate 708 annotated MRI image data using the MRI image data and the one or more annotations identified in the one or more annotated MG images, where the annotated MRI image data may include one or more annotations representative of an abnormality at a second location based at least in part on the first location.

[0061] Additionally, the method 700 may further include one or more processors configured to store 710 the annotated MRI image data in a database.

[0062] Figure 8 illustrates a block diagram of computing devices in a distributed computing environment in accordance with an embodiment of the present invention. Figure 8 illustrates a block diagram of a computing device 800 suitable for server 125 and computing device 120 in accordance with an exemplary embodiment of the present invention. It should be understood that Figure 8 is only provided as an illustration of one implementation and is not intended to imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the illustrated environment may be made.

[0063] Computing device 800 includes a communications fabric 802 that provides communications between cache 816, memory 806, persistent storage 808, communications units 810, and input / output (I / O) interfaces 812. Communications fabric 802 may be implemented with any architecture designed to pass data and / or control information between processors (such as microprocessors, communications and network processors), system memory, peripheral devices, and other hardware components in the system. For example, communications fabric 802 may be implemented with one or more buses or crossbar switches.

[0064] Memory 806 and persistent storage 808 are computer readable storage media. In this embodiment, memory 806 includes random access memory (RAM). In general, memory 806 may include any suitable volatile or non-volatile computer readable storage medium. Cache 816 is a high-speed memory that improves performance of computer processor 804 by retaining recently accessed and near-accessed data from memory 806.

[0065] Programs may be stored in persistent storage 808 and memory 806 for execution and / or access by one or more of the respective computer processors 804 via cache 816. In one embodiment, persistent storage 808 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage 808 may include a solid-state hard drive, a semiconductor memory device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or other computer-readable storage medium capable of storing program instructions or digital information.

[0066] The media used by persistent storage 808 may also be removable. For example, a removable hard drive may be used for persistent storage 808. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer readable storage medium that is also part of persistent storage 808.

[0067] Communications unit 810, in these examples, provides for communication with other data processing systems or devices. In these examples, communications unit 810 includes one or more network interface cards. Communications unit 810 may provide communications using either or both physical and wireless communications links. Programs described herein may be downloaded to persistent storage 808 through communications unit 810.

[0068] The I / O interface 812 allows for input and output of data to and from other devices that may be connected to the computing device 120. For example, the I / O interface 812 may provide a connection to external devices 818, such as the image sensor 130, a keyboard, a keypad, a touch screen, or other suitable input devices or combinations thereof. The external devices 818 may also include portable computer-readable storage media, such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data 814 used to implement embodiments of the present invention may be stored on such portable computer-readable storage media and loaded into the persistent storage 808 via the I / O interface 812. The I / O interface 812 also connects to a display 820.

[0069] Display 820 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0070] The software and data 814 described herein are identified based on the application for which it is implemented in a particular embodiment of the invention. However, it should be understood that any particular program nomenclature herein is used merely for convenience, and thus the invention should not be limited to use with any particular application identified or implied or identified and implied by such nomenclature.

[0071] The present invention may be a system, a computer-implemented method, or a computer program product, or combinations thereof, at any possible level of technical detail of integration. The computer program product may include one or more computer-readable storage media having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0072] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves having instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium is not to be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or an electrical signal sent through a wire.

[0073] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium in the respective computing / processing device.

[0074] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by individualizing the electronic circuitry using state information of the computer readable program instructions to implement aspects of the invention.

[0075] Aspects of the present invention are described with reference to flowchart illustrations and / or block diagrams of computer-implemented methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0076] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions executed by the processor of the computer or other programmable data processing apparatus create means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer program instructions may also be stored in a computer readable medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, whereby the instructions stored in the computer readable medium include an article of manufacture including instructions implementing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0077] The computer program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0078] The flowcharts and block diagrams in the figures illustrate the architecture, functionality and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts may represent a module, segment or portion of code, including one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be performed simultaneously, in a partially or fully overlapping manner in time, performed substantially simultaneously, accomplished as one step, or the blocks may sometimes be performed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0079] The description of various embodiments of the present invention has been presented for illustrative purposes, but they are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present invention. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over the art found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A method for annotating magnetic resonance imaging (MRI) images by computer processing, comprising: receiving, by one or more processors, mammogram (MG) image data representing one or more annotated MG images of a patient's breast; identifying, by one or more processors, one or more MG annotations in the one or more annotated MG images, the one or more MG annotations representing an anomaly at a first location in the one or more annotated MG images; receiving, by one or more processors, MRI image data representative of one or more MRI images of the patient's breast; generating, by one or more processors, annotated MRI image data using the MRI image data and the one or more MG annotations identified in the one or more annotated MG images, the annotated MRI image data including one or more MRI annotations representative of the anomaly at a second location based at least in part on the first location; storing, by one or more processors, the annotated MRI image data in a database; The method includes:

2. The method of claim 1 , wherein each of the one or more annotated MG images is one of a cranio-caudal (CC) view or a medio-lateral oblique (MLO) view.

3. The method of claim 1 or claim 2, wherein the one or more MRI images comprise a plurality of image slices, including one or more image slices having the abnormality. determining, by one or more processors, a left / right chest position and a quadrant of the patient's chest based on the first position within the one or more annotated MG images. The method of claim 3 further comprising:

5. Identifying, by one or more processors, corresponding left and right chest positions and quadrants in the plurality of image slices of the one or more MRI images. The method of claim 4 further comprising:

6. generating the annotated MRI image data, Mapping, by one or more processors, the one or more MG annotations to the second locations of the corresponding left and right chest quadrants of the one or more MRI images represented in the MRI image data. The method of claim 5 , comprising:

7. further comprising automatically selecting, by one or more processors, a subset of the MRI image data based on extracting from the one or more annotated MG images an identification of the left and right sides of the patient's chest and a quadrant of the abnormality at the first location; 7. The method of claim 1, wherein generating the annotated MRI image data using the MRI image data and the one or more MG annotations identified in the one or more annotated MG images comprises generating annotated MRI image data using the selected subset of the MRI image data and the one or more MG annotations identified in the one or more annotated MG images.

8. Providing, by one or more processors, the annotated MRI image data to a supervised machine learning model; processing, by one or more processors, the annotated MRI image data with the one or more MRI annotations using the supervised machine learning model to generate output data corresponding to a diagnosis; The method of any one of claims 1 to 7, further comprising:

9. A computer program comprising program instructions causing a computer to carry out a method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing the computer program of claim 9.

11. 1. A computer system for annotating Magnetic Resonance Imaging (MRI) images, the computer system comprising: one or more processors; one or more computer readable storage media; program instructions stored on said one or more computer readable storage media for execution by at least one of said one or more processors; wherein the program instructions include: program instructions for receiving mammogram (MG) image data representing one or more annotated MG images of a patient's breast; program instructions for identifying one or more MG annotations in the one or more annotated MG images, the one or more MG annotations representing an anomaly at a first location in the one or more annotated MG images; program instructions for receiving MRI image data representative of one or more MRI images of the patient's breast; program instructions for generating annotated MRI image data using the MRI image data and the one or more MG annotations identified in the one or more annotated MG images, the annotated MRI image data including one or more MRI annotations at a second location based at least in part on the first location, the one or more MRI annotations in the annotated MRI image data being indicative of the anomaly; program instructions for storing the annotated MRI image data in a database; 1. A computer system comprising:

12. The computer system of claim 11 , wherein each of the one or more annotated MG images is one of a craniocaudal (CC) view or a mediolateral oblique (MLO) view.

13. 13. The computer system of claim 11 or claim 12, wherein the one or more MRI images comprise a plurality of image slices including one or more image slices having the abnormality.

14. program instructions for determining a left / right chest and quadrant of the patient's chest based on the first location within the one or more annotated MG images; program instructions for identifying the corresponding left and right chest regions and quadrants in the plurality of image slices of the one or more MRI images; 14. The computer system of claim 13, further comprising:

15. The program instructions for generating the annotated MRI image data include: and program instructions for mapping the one or more MG annotations to the second locations of the corresponding left and right chest quadrants of the one or more MRI images represented in the MRI image data.

15. The computer system of claim 14, comprising:

16. The method of claim 15, further comprising: automatically selecting a subset of the MRI image data based on extracting from the one or more annotated MG images an identification of a left or right side of the patient's chest and a quadrant of the abnormality at the first location; The computer system of any one of claims 11 to 15, wherein the program instructions for generating the annotated MRI image data using the MRI image data and the one or more MG annotations identified in the one or more annotated MG images include program instructions for generating annotated MRI image data using the selected subset of the MRI image data and the one or more MG annotations identified in the one or more annotated MG images.

17. program instructions for providing the annotated MRI image data to a supervised machine learning model; program instructions for processing the annotated MRI image data with the one or more MRI annotations using the supervised machine learning model to generate output data corresponding to a diagnosis; 17. The computer system of claim 11, further comprising:

Images