SYSTEM AND METHOD FOR CORRELATING OBJECTS OF INTEREST - Patent application

An ensemble machine learning model correlates regions of interest in craniocaudal and mediolateral oblique images by combining similarity and geometric matching, addressing the challenge of accurate lesion correlation in mammography and improving diagnostic precision.

JP2024541551APending Publication Date: 2024-11-08HOLOGIC INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024531498
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-29
Filing Date
2022-11-23
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Current medical imaging techniques, such as mammography, struggle to accurately correlate regions of interest between different image types (craniocaudal and mediolateral oblique views) due to human error and lack of automated methods for identifying and correlating lesions across these views.

Method used

An ensemble machine learning model is employed to correlate regions of interest in craniocaudal and mediolateral oblique images by combining similarity and geometric matching models, providing a joint probability of correlation based on confidence levels to enhance accuracy.

Benefits of technology

The ensemble model significantly improves the accuracy of lesion correlation between different image types, reducing human error and enhancing diagnostic precision in breast imaging.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541551000001_ABST
    Figure 2024541551000001_ABST
Patent Text Reader

Abstract

A method for correlating regions in an image pair including a craniocaudal image and a mediolateral oblique image. Data from a similarity matching model is received by an ensemble model, the data including at least the regions of the matched pairs and a first confidence level indicator associated with the regions of the matched pairs. Data from a geometric matching model is received by the ensemble model, the data including at least the regions of the matched pairs and a second confidence level indicator. A joint probability of correlation is determined by the ensemble model based on an evaluation of each of the first and second confidence levels by the ensemble matching model, the joint probability of correlation providing a probability that a region in each image correlates to a corresponding region in the other image. The joint probability of correlation is provided to an output device.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application was filed as a PCT international patent application on November 23, 2022, and claims priority to and benefit of U.S. Provisional Application No. 63 / 283,866, filed November 29, 2021, which is incorporated by reference in its entirety herein. [Background technology]

[0002] Medical imaging provides a non-invasive method for visualizing the internal structures of a patient. Visualization methods can be used to screen for and diagnose cancer and other diseases in a patient. For example, early screening can detect lesions in the breast that may be cancerous so that treatment can occur at an early stage in the disease.

[0003] Mammography is a form of medical imaging that utilizes X-ray radiation to visualize breast tissue. These techniques are often used to screen patients for potential cancerous lesions and other abnormalities. A traditional mammogram involves obtaining two-dimensional (2D) images of the breast from various angles. A cranial-caudal (CC) image is one of the standard image types captured using X-ray radiation. A CC image is a visualization of the breast from above (e.g., a view from the top of the breast). Another standard image type is a medial-lateral oblique (MLO) image, in which the breast is imaged from the side at an angle. Typically, MLO images are captured at an angle or at an angle of 40-60 degrees. Reviewing and examining the information in the CC and MLO images together can increase the diagnostic power of breast imaging. Tomosynthesis is another method of taking a mammogram, in which multiple images are obtained. Each image is obtained at a separate thickness, and the images are taken at multiple angles.

[0004] It is against this background that the present disclosure is made. Summary of the Invention [Means for solving the problem]

[0005] In one aspect, the disclosure relates to a method for correlating regions of interest (ROIs) in an image pair including a cranio-caudal (CC) image and a medio-lateral oblique (MLO) image, the method including: receiving, by an ensemble matching machine learning (ML) model, data from a similarity matching ML model, the data from the similarity matching ML model including at least the ROIs of the matched pair and a first confidence level indicator associated with the ROIs of the matched pair; receiving, by the ensemble matching ML model, data from a geometric matching (GM) model, the data from the GM model including at least the ROIs of the matched pair and a second confidence level indicator; determining, by the ensemble matching ML model, a joint probability of correlation based on an evaluation of each of the first and second confidence levels by the ensemble matching ML model, the joint probability of correlation providing a probability that a ROI in the CC image is correlated to a corresponding ROI in the MLO image, and vice versa; and providing the joint probability of correlation to an output device. In one embodiment, the method further includes receiving data associated with the training CC-MLO image pairs and training an ensemble matching ML model using the data associated with the training CC-MLO image pairs.

[0006] In another embodiment of the above aspect, the method further includes determining a third confidence level indicator based on the assessment of each of the first and second confidence levels and the joint probability of the correlation by the ensemble matching ML model, the third confidence level indicator being a likelihood of confidence associated with the joint probability of the correlation. In another embodiment, the joint probability is a probability that the similarity correlation and the GM correlation adequately correlate the CC-ROI and the MLO-ROI. In yet another embodiment, providing the joint probability of the correlation to the output device includes a numerical value associated with the joint probability of the correlation.

[0007] In another embodiment, each image of the matched lesion pair is a full breast image. In yet another embodiment, each image of the matched lesion pair contains only an ROI for each of the CC image and the MLO image. In yet another embodiment, providing the joint probability of correlation to the output device includes a numerical display. In another embodiment of the above aspect, providing the joint probability of correlation to the output device includes receiving a set of CC-ROIs and presenting the MLO-ROI in response to receiving a selection of the CC-ROI. In a further embodiment, the method further includes determining that the joint probability of correlation exceeds a predetermined threshold in response to receiving a selection of the CC-ROI and presenting the MLO-ROI in response to determining that the joint probability of correlation exceeds a predetermined threshold.

[0008] In another embodiment of the above aspect, the matched pair of ROIs includes a similarity correlation between a CC-ROI in the CC-image and an MLO-ROI in the MLO-image, and the first confidence level indicator is a probability associated with the correlation between the CC-ROI and the MLO-ROI. In another embodiment, the second confidence level indicator is a probability associated with a GM correlation between the CC-ROI and the MLO-ROI. In yet another embodiment, the method further includes displaying a pair of symbols on the output display, each symbol of the pair of symbols marking an ROI of the matched pair of ROIs. In another embodiment, the data from the GM model comprises location data for each of the ROIs of the matched pair of ROIs. In a further embodiment, the second confidence level indicator indicates a probability that a first location of a first ROI of the matched pair of ROIs and a second location of a second ROI of the matched pair of ROIs are the same location. In a still further embodiment, the CC image and the MLO image each depict a breast, and the GM model logically divides the breast into quadrants.

[0009] In another embodiment of the above aspect, the data from the similarity model comprises characteristic data for each of the ROIs of the matched pair of ROIs. In another embodiment, the first confidence level indicator indicates a degree of similarity between a first set of characteristics associated with a first ROI of the matched pair of ROIs and a second set of characteristics associated with a second ROI of the matched pair of ROIs. In a further embodiment, the characteristic data includes one or more of size, shape, one or more perimeters, location, density, one or more colors, orientation, texture, pattern, and depth.

[0010] In another aspect, the disclosure relates to a system for ensemble matching of craniocaudal (CC) and mediolateral oblique (MLO) images, the system including at least one processor in communication with at least one memory, and an ensemble matching module executing on the at least one processor and configured during operation to receive matched CC-MLO image pairs and similarity confidence level indicators associated with the matched CC-MLO image pairs from a similarity matching model, receive matched CC-MLO image pairs and GM confidence level indicators associated with the matched CC-MLO image pairs from a geometric matching (GM) model, apply the ensemble matching model, determine an ensemble confidence level based on an ensemble machine learning (ML) algorithm trained on a plurality of matched CC-MLO pairs, associate the ensemble confidence level with the matched CC-MLO image pairs, and output the ensemble confidence level with the matched CC-MLO image pairs. In an embodiment of this aspect, the system further includes an image acquisition module.

[0011] In another embodiment of the above aspect, the matched CC-MLO image pair comprises a first region of interest (ROI) identified in the CC image and a second ROI identified in the MLO image, and either the similarity ML model or the GM model assigns correlation to the first and second ROI. In a further embodiment, the ensemble matching module further receives location data from the GM model, the location data being associated with the correlation between the first and second ROI. In another further embodiment, the ensemble matching module further receives shape data from the similarity ML model, the shape data being associated with the correlation between the first and second ROI. In a still further embodiment, the ensemble matching module further receives perimeter data from the similarity ML model, the shape data being associated with the correlation between the first and second ROI. In yet a further embodiment, the similarity matching model determines a matched CC-MLO image pair by identifying a CC region of interest (ROI) in a CC-image received from the image acquisition module, searching an MLO image received from the image acquisition module for an MLO-ROI, the MLO image including a plurality of regions, the similarity matching model examining each region of the plurality of regions, determining at least one similarity characteristic of each of the CC-ROI and the MLO-ROI, and correlating the CC-ROI and the MLO-ROI based on the at least one similarity characteristic. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an exemplary system for detecting a region of interest in a breast.

[0013] [Diagram 2] FIG. 2 illustrates an exemplary X-ray imaging system.

[0014] [Diagram 3] FIG. 3 illustrates a perspective view of the X-ray imaging system of FIG.

[0015] [Figure 4] FIG. 4 illustrates the x-ray imaging system of FIG. 2 in a breast positioning for a left lateral oblique (LMLO) imaging orientation.

[0016] [Diagram 5] FIG. 5 illustrates a technique for geometric matching that uses the distance between the nipple and the lesion in the CC image.

[0017] [Figure 6] FIG. 6 illustrates a quadrant technique that may be used by a geometric matching ML model.

[0018] [Figure 7] 7A-7C illustrate CC-MLO image pairs showing soft tissue lesions.

[0019] [Figure 8] 8A-8C illustrate a CC-MLO image pair showing a calcification cluster.

[0020] [Figure 9] FIG. 9 illustrates a process flow for processing CC-MLO image pairs.

[0021] [Figure 10] FIG. 10 illustrates a flow chart of a method for correlating a first lesion in a CC image to a second lesion in an MLO image.

[0022] [Figure 11] FIG. 11 illustrates a flow chart of another method for correlating a first lesion in a CC image to a second lesion in an MLO image.

[0023] [Figure 12] FIG. 12 illustrates a flow chart of an ensemble method for correlating a first lesion in a CC image to a second lesion in an MLO image.

[0024] [Figure 13]FIG. 13 illustrates a flowchart of a method for training an ensemble matching machine learning (ML) model.

[0025] [Figure 14] FIG. 14 illustrates a block diagram of example physical components of a computing system that can be used to implement one or more aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Detailed Description The present disclosure is directed to a system and method for locating a lesion in breast tissue using an imaging device. In particular, a computing system utilizes a machine learning (ML) model or an artificial intelligence (AI) model to navigate to a first lesion in an image of a first image type, navigate to a second lesion in an image of a second image type, and correlate the first and second lesions. In correlating the first and second lesions, the computing system determines that the first and second lesions are the same object in the tissue of the breast. In one embodiment, the first image type is a craniocaudal (CC) image and the second image type is a medial-lateral oblique (MLO) image. As discussed herein, an image type refers to an image viewpoint from which an image is taken in addition to other features, e.g., a modality with which the image is taken. The term "lesion" refers to any object of interest, such as a mass, one or more calcifications, and other suspicious areas.

[0027] In an embodiment, a system and method embodying the present disclosure identifies a first region of interest (ROI) in an image of a first image type, identifies a second ROI in an image of a second image type, and correlates the first ROI to the second ROI. The ROI may contain one or more lesions. The image of the first image type may include multiple lesions. Similarly, the image of the second image type may also include multiple lesions. Although the embodiments discussed in more detail herein are primarily directed to mammograms, it will be understood by those skilled in the art that the principles of the present disclosure are also applicable to the use of other forms of breast imaging, such as tomosynthesis, ultrasound, and magnetic resonance imaging, as well as medical imaging on other structures of the body.

[0028] In general, the computing system is operative to correlate a first lesion in an image of a first image type and a second lesion in an image of a second image type and provide a confidence level indicator. The correlation is a determination by the system that the first and second lesions respectively represent the same object in the breast. The confidence level indicator indicates a level of confidence in the correlation between the first and second lesions. The confidence level indicator represents a likelihood that the determination that the first and second lesions represent the same object in the breast has been reliably determined. The computing system uses one or more ML models to analyze the images of the first image type and the images of the second image type to determine whether the first lesion in the image of the first image type correlates with the second lesion in the image of the second image type. The confidence level indicator may also be generally understood as a confidence score.

[0029] In one example implementation, the confidence level indicator is presented on the display. The confidence level indicator is provided to a radiologist to assist the radiologist in determining whether a first lesion in a first image type is correlated with a second lesion in a second image type. When the system determines that a first lesion in a first image type is correlated with a second lesion in a second image type, this indicates that the first and second lesions are different images of the same object in the patient's breast. In some examples, one or more of a similarity matching ML model, a geometric matching ML model, and an ensemble matching ML model are used.

[0030] FIG. 1 illustrates an exemplary identification system 100 for locating a lesion in a breast using one or more types of imaging and for correlating a first lesion in a first image type with a second lesion in a second image type. The identification system 100 includes a computing system 102 and an X-ray imaging system 104. In some examples, the identification system 100 operates to guide a radiologist to a lesion in the breast based on data collected during an X-ray imaging procedure when the lesion is first identified. In some examples, the identification system 100 provides a confidence level indicator to the radiologist to assist the radiologist in verifying that a first lesion visible in an image of the first image type is correlated with a second lesion identified in an image of the second image type. In an embodiment, the confidence level indicator is presented on a display.

[0031] The X-ray imaging system 104 operates to capture images of breast tissue using X-ray radiation. The X-ray imaging system 104 includes an X-ray imaging device 124 and an X-ray computing device 112 in communication with the X-ray imaging device 124. The X-ray imaging device 124 is described in further detail in connection with FIGS. 2-4. The X-ray computing device 112 operates to receive input from a radiologist (R), operate the X-ray imaging device 124, and view images received from the X-ray imaging device 124.

[0032] A radiologist R operates the X-ray computing device 112 and captures X-ray images of the patient's (P) breasts using the X-ray imaging device 124. The X-ray images typically include CC and MLO images (LMLO, RMLO and RCC, LCC) for each breast, among others. The X-ray images may be taken as part of a routine medical examination or as part of a diagnostic test.

[0033] For each image type (CC, MLO, etc.), images may be acquired as multiple projections (Tp) at different angles and thicknesses. The multiple images may be processed or reconstructed to generate multiple reconstructed images or slices (Tr). The multiple Tr images may be combined into a single combined image (Ms) that shows the most relevant clinical information and ROI location. Each pixel in the combined image Ms may be mapped to a Tr image or slice. Each CC image and MLO image may be one or more tomosynthesis image slices or Tr image. In an embodiment, one or more of the CC image and MLO image may be a Tp projection image.

[0034] In an embodiment, each of the Ms, Tr, and Tp images may be stored in a data storage device and can be retrieved for review by a radiologist. The images are then presented to radiologist R, who re-identifies one or more lesions in patient P's breast that may require additional analysis to determine whether the lesions are potentially cancerous and require a biopsy.

[0035] The computing system 102 operates to process and store information received from the X-ray imaging system 104. In the embodiment of FIG. 1, the computing system 102 includes a matching engine 106 and a data storage device 108. In some embodiments, the matching engine 106 and the data storage device 108 are stored in the memory of the computing system 102 (e.g., memory 1408 in FIG. 14). In some embodiments, the computing system 102 accesses the matching engine 106 and the data storage device 108 from a remote computing system 110, such as a cloud computing environment. Although FIG. 1 illustrates the computing system 102 as being separate from other components of the identification system 100, the computing system 102 may also be incorporated into the X-ray computing device 112 or another computing device utilized in patient management. In some embodiments, the computing system 102 includes two or more computing devices.

[0036] A radiologist R would typically review multiple CC and MLO tomosynthesis images for each breast. In particular, the radiologist would visually review the images, locate a region of interest in one of the CC images, and attempt to find a similar region in one of the corresponding MLO images. However, this process is incomplete and subject to human error because there are no known techniques that can find related artifacts across different types of images.

[0037] The matching engine 106 is programmed to analyze the different types of images and determine whether a first lesion in a first image of a first image type is the same object as a second lesion in a second image of a second image type. The matching engine 106 includes one or more machine learning (ML) modules 116, 118, 120. The matching engine 106 represents logic or programming running on a processor in the computer system 102, such as the processing device 1402 in FIG. 14. In an embodiment, the matching engine further includes a correlation evaluator 122.

[0038] In the example of FIG. 1, the training data store 114 is utilized to train the ML models 116, 118, 120. The training data store 114 stores a plurality of training cases of correlated first and second lesions in images of a first and second image type, where the first and second lesions are correlated by a matching engine or by a human. In an embodiment, the training cases stored in the training data store 114 each include at least a pair of images (also referred to simply as an "image pair") of a first image type and a second image type. Each image in the training image pair contains one or more of an ROI, a confirmed lesion, or a potential lesion. A first lesion in a first image in the training image pair is correlated to a second lesion in a second image of the training image pair. The first image is of the first image type and the second image is of the second image type. The stored training cases include training image pairs with correctly correlated lesions, and in an embodiment also include training image pairs with incorrectly correlated lesions.

[0039] The computer system 102 uses training cases stored in the training data storage 114 to train the ML models 116, 118, and 120 to identify features that can be used to correlate first and second lesions in images of the first and second image types. In one embodiment, each ML model 116, 118, 120 is a machine learning classifier. Once trained, the ML models 116, 118, and 120 can be used by the matching engine 106 to correlate lesions in images of different types.

[0040] Various machine learning techniques can be utilized to generate machine learning classifiers for use as lesion classifiers. In some embodiments, the machine learning model is a supervised machine learning model. In other embodiments, one or more of the machine learning models are unsupervised machine learning models. In some embodiments, the machine learning model is based on an artificial neural network. In some embodiments, the neural network is a deep neural network (DNN). In some embodiments, the machine learning model is a convolutional deep neural network (CNN). In some embodiments, an ensemble or combination of two or more networks is utilized to generate one or more of the image classifiers. In some embodiments, two or more machine learning models are utilized to generate features or feature sets from training image pairs in the training data store 114.

[0041] The resulting trained machine learning classifiers, ML models 116, 118, 120, are applied by a correlation evaluator 122. The correlation evaluator 122 compares image pairs (e.g., CC-MLO image pairs) including a first type of image (e.g., CC image) and a second type of image (e.g., MLO image) and applies one or more of the ML models 116, 118, 120 to the CC-MLO image pairs to determine whether a first lesion in a CC image of the image pair correlates to a second lesion in the MLO image of the image pair. The correlation evaluator 122 determines that the first and second lesions are correlated when the first lesion identified in the CC image is determined to be the same object as the second lesion identified in the MLO image based on the ML models 116, 118, 120. The correlation evaluator 122 determines and outputs a confidence level indicator indicating a level of confidence in the correlation. In an embodiment, the correlation evaluator 122 is absent and the confidence level indicator is determined by the matching engine 106 through application of one or more of the ML models 116, 118, 120 without the use of the correlation evaluator 122.

[0042] In one example, the confidence level indicator is a numeric value. In another example, the confidence level indicator may indicate a category of confidence, such as "high", "medium", or "low". In alternative examples, the confidence level indicator is provided as a percentage, such as "99%", "75%", or "44%". In an example, the confidence level indicator indicates the likelihood that the correlation correctly determines that the first and second lesions are the same object.

[0043] In an embodiment, the confidence level indicator includes two parts: a first confidence level indicator indicating a level of confidence in the correlation, and a second confidence level indicator indicating a level of confidence in the calculation of the first confidence level indicator. For example, a system embodying the present disclosure determines that a first lesion in an image of a first image type is correlated with a second lesion in an image of a second image type. The system calculates a first confidence level indicator that is a probability that the correlation between the first lesion and the second lesion is accurate. The system also determines a second confidence level indicator that indicates a level confidence that the first confidence level indicator was properly calculated. In this example, the first confidence level indicator is an assessment of the correlation between the first lesion and the second lesion, while the second confidence level indicator is an assessment of the calculation that produces the first confidence level indicator.

[0044] A graphical user interface (GUI), presented on the display of the computing system 102, operates to present information to a radiologist or other clinician. In some examples, the GUI displays a confidence level indicator that is overlaid on one or more image pairs being reviewed by the radiologist. For example, the GUI presents CC and MLO images displayed side-by-side, with a confidence level indicator overlaying the image pair to indicate a level of confidence that a first lesion identified in the CC image is the same object as a second lesion identified in the MLO image. In another example, the radiologist indicates a region of interest in the CC image, and in response, a corresponding MLO image is brought on the display having the region of interest with the highest confidence level of correlation with the region of interest in the CC image. In an embodiment, the corresponding MLO image is expanded to the region of interest that has the indicator or is correlated.

[0045] Additionally or alternatively, the displayed information may include data such as a rationale or reason for the value of the confidence level indicator or logic indicating why the matching engine 106 determined a particular value for the confidence level indicator. For example, coordinate location data from the geometric matching model 118 or shared shape property data from the similarity matching model 116 may be displayed. In an embodiment, the confidence level indicator may be displayed on the composite or Ms image. In addition, the GUI may display a letter or other symbol indicia to indicate that a first lesion in the CC image correlates to a second lesion in the MLO image. For example, a first lesion in the CC image and a correlated second lesion in the MLO image are labeled as "A". If multiple regions of interest appear in an image pair, multiple letters or symbols may be used. In an embodiment, a letter or other symbol indicia is displayed in addition to the confidence level indicator.

[0046] The data storage device 108 operates to store information received from the X-ray imaging system 104 and the matching engine 106. In some embodiments, the data storage device 108 is actually two or more separate data storage devices. For example, one data storage device may be a remote data storage device that stores images from one or more X-ray imaging systems, such as the X-ray imaging system 104. Another data storage device may be stored locally within the computing system 102. In some embodiments, the data storage device 108 may be part of an electronic medical record (EMR) system.

[0047] FIG. 2 illustrates an example of the X-ray imaging system shown in FIG. 1. FIG. 3 illustrates a perspective view of the X-ray imaging system of FIG. 1. Referring to FIGS. 2 and 3 in parallel, the X-ray imaging system 104 immobilizes a patient's breast 202 for X-ray imaging via a breast compression immobilizer unit 204, which includes a static breast support platform 206 and a moveable compression paddle 208. The breast support platform 206 and the compression paddle 208 each have compression surfaces 210 and 212, respectively, that move toward each other to compress and immobilize the breast 202. In some systems, the compression surfaces 210, 212 are exposed to direct contact with the breast 202. The breast support platform 206 also houses an image receiver 216, and optionally a tilt mechanism 218, and optionally an anti-scatter grid (not shown). The breast compression immobilizer unit 204 is in the path of an imaging beam 220 , emanating from an x-ray source 222 , such that the imaging beam 220 impinges on the image receiver 216 .

[0048] The breast compression immobilizer unit 204 is supported on a first support arm 224 and the x-ray source 222 is supported on a second support arm 226. For mammography, the first and second support arms 224, 226 can rotate as a unit about an axis 228 between different imaging orientations, such as CC and MLO, so that the x-ray imaging system 104 can take mammogram projection images (x-ray images) in each orientation. In operation, the image receiver 216 remains in a fixed position relative to the breast support platform 206 while images are taken. The breast compression immobilizer unit 204 releases the breast 202 for movement of the first and second support arms 224, 226 to the different imaging orientations. For tomosynthesis, the first support arm 224 remains in a fixed position with the breast 202 immobilized and remaining in a fixed position while at least a second support arm 226 rotates the x-ray source 222 about an axis 228 relative to the breast compression immobilizer unit 204 and the compressed breast 202. The x-ray imaging system 104 takes multiple tomosynthesis projection images of the breast 202 at distinct angles of the imaging beam 220 relative to the breast 202.

[0049] In parallel, optionally, the receiver 216 may be tilted in synchronization with the rotation of the second support arm 226 relative to the breast support platform 206. The tilt may be through the same angle as the rotation of the x-ray source 222, but may also be through a different angle selected such that the imaging beam 220 remains in substantially the same position on the receiver 216 for each of the multiple images. The tilt may be about an axis 230, which may be, but need not be, in the image plane of the receiver 216. A tilt mechanism 218, coupled to the receiver 216, may drive the receiver 216 in a tilting motion.

[0050] When the x-ray imaging system 104 is operated, the image receiver 216 generates imaging information in response to illumination by the imaging beam 220 and provides it to the image processor 232 for processing and generation of a mammogram. A system control and workstation unit 238, including software, controls the operation of the system and interacts with the operator, receiving commands and delivering information, including processed light images.

[0051] 4 illustrates the X-ray imaging system shown in FIG 1 in a breast positioning state for a left MLO (LMLO) imaging orientation. The tube head 258 of the X-ray imaging system 104 is set at an orientation such that it is generally parallel to the gantry 256 of the X-ray imaging system 104 or is otherwise not normal to the flat portion of the support arm 260 against which the breast is placed. In this position, a technician can more easily position the breast without having to bend or squat under the tube head 258.

[0052] The X-ray imaging system 104 includes a floor mount or base 254 for supporting the X-ray imaging system 104 on a floor. A gantry 256 extends upwardly from the floor mount 252 and rotatably supports both a tube head 258 and a support arm 260. The tube head 258 and the support arm 260 are configured to rotate discretely relative to one another and may also be raised and lowered along the face 262 of the gantry to accommodate patients of different heights. An X-ray source, described elsewhere herein and not shown here, is disposed within the tube head 258. The support arm 260 includes a support platform 264 that includes an X-ray receiver therein and other components (not shown). A compression arm 266 extends from the support arm 260 and is configured to linearly raise and lower a compression paddle 268 (relative to the support arm 260) for compression of the patient's breast during an imaging procedure. Together, the tube head 258 and the support arm 260 may be referred to as a C-arm.

[0053] Several interfaces and display screens are disposed on the X-ray imaging system 104. These include a foot display screen 270, a gantry interface 272, a support arm interface 274, and a compression arm interface 276. In general, the various interfaces 272, 274, and 276 may include one or more display screens, including one or more tactile buttons, knobs, switches, and capacitive touch screens with graphic user interfaces (GUIs) to allow a user to interact with and control the X-ray imaging system 104. In an embodiment, the interfaces 272, 274, 276 may include control functionality that may also be available on system control and workstations, such as the X-ray computing device 112 of FIG. 1. Any individual interface 272, 274, 276 may include functionality available on the other interfaces 272, 274, 276, either continuously or selectively, based at least in part on predetermined settings, user preferences, or operational requirements. Generally, as described below, the foot display screen 270 is primarily a display screen, although a capacitive touch screen may be utilized if required or desired.

[0054] In an embodiment, gantry interface 272 may enable functionality such as selection of imaging orientation, display of patient information, adjustment of support arm elevation or support arm angle (tilt or rotation), safety features, etc. In an embodiment, support arm interface 274 may enable functionality such as adjustment of support arm elevation or support arm angle (tilt or rotation), adjustment of compression arm elevation, safety features, etc. In an embodiment, compression arm interface 276 may enable functionality such as adjustment of compression arm elevation, safety features, etc. Additionally, one or more displays associated with compression arm interface 276 may display more detailed information such as the compression arm force applied, selected imaging orientation, patient information, support arm elevation or angle setting, etc. Foot display screen 270 may also display information as displayed by the display of compression arm interface 276 or additional or different information as required or desired for a particular application.

[0055] As described above, for example, a CC-MLO image pair taken by X-ray imaging system 104 is analyzed to determine a correlation between a first lesion in the CC image and a second lesion in the MLO image. The CC-MLO image pair is analyzed by a matching engine using one or more ML models. The machine learning models include one or more of a similarity matching ML model, a geometric matching ML model, and an ensemble ML matching model (e.g., ML models 116, 118, 120 in FIG. 1).

[0056] When applying the similarity matching ML model, the matching engine compares the first and second lesions and determines a confidence level indicator representing the degree to which the first and second lesions are similar to each other in appearance or location. The similarity matching ML model provides a criterion for determining the similarity of two or more lesions based on factors or features such as shape, perimeter, proximity, or relationship to anatomical landmarks or other prominent features in the image. The similarity matching ML model may be a neural network, and in an embodiment is a feature-based network. The neural network may be a fully connected network. In an example embodiment, the similarity matching ML model is a deep learning neural network, such as a deep convolutional network.

[0057] When applying the geometric matching ML model, the matching engine maps the first and second lesions in each of the first image (e.g., CC image) and the second image (e.g., MLO image) based on one or more criteria to determine whether the first lesion is the same object as the second lesion. In one embodiment, the criteria include a distance between the lesion and one or more identifiable physical or anatomical landmarks. Anatomical landmarks include, but are not limited to, the nipple, the chest wall, and the pectoral muscle. In an embodiment, the criteria include a probability that the lesion is located within a particular quadrant in the image or volume (e.g., breast) being imaged. The geometric matching ML model may be a neural network or other ML model. In an embodiment, the geometric matching model is a rule-based AI.

[0058] When applying the ensemble matching model, the matching engine compares correlation data received from other models in the system, such as the similarity matching model 116 and the geometric matching model 118, and determines and outputs a joint probability of correlation. In an embodiment, the ensemble matching model may be a feature-based neural network. The ensemble matching model may generate features related to the data received from other matching models in the system as well as features related to lesions in different image types. For example, data or confidence level indicators generated by other ML models may be features in the ensemble matching model.

[0059] FIG. 5 illustrates a technique for geometric matching using the distance between the nipple and the first or second lesion in an image pair. In one embodiment, a first lesion 500 is identified in a right breast CC image (RCC). A geometric matching ML model is applied to find the location of a second lesion 502 in a right breast MLO image (RMLO). The geometric matching ML model is used to calculate the distance (d) from the first lesion 500 to the nipple 504. An arc is superimposed on the RMLO image with a radius equal to the distance (d). Once the second lesion 502 is identified in the RMLO image, the geometric matching ML model is applied to determine a confidence level indicator (e.g., probability) that the second lesion 502 in the RMLO image is the same object as the first lesion 500 in the RCC image. In other embodiments, different or additional anatomical landmarks may be used in geometrically matching the target lesion.

[0060] FIG. 6 illustrates a quadrant technique that may be used when applying the geometric matching ML model. The quadrant technique is used for quadrant-based matching of lesions. A breast 600 is viewed from the front of the breast. A horizontal line 602 and a vertical line 604 are superimposed on the breast 600 and intersect with the nipple 606. The intersecting horizontal and vertical lines 602, 604 logically divide the breast 600 into four quadrants (quadrants 0, 1, 2, and 3). In the RCC image, the vertical line 604 corresponds to the posterior nipple line (PNL). The PNL line is drawn posteriorly and vertically from the nipple in the CC image to the pectoral muscle (or posterior image edge in the CC). In the RCC image, quadrants 0 and 3 are combined into one half of the breast 612, and quadrants 1 and 2 are combined into the other half of the breast 614.

[0061] As shown in the graphic representation 608 of the breast, the target lesion 610 extends into quadrants 0 and 3. If the vertical line 604 represents the x-axis and the horizontal line 602 represents the y-axis of a Cartesian coordinate system, the lesion 610 has a z-maximum 616 in quadrant 0 and a z-minimum 618 in quadrant 3. A geometric matching ML model is applied to estimate the quadrant in which a first lesion in the CC image is located, estimate the quadrant in which a second lesion in the MLO image is located, compare the two estimated quadrants, and determine whether the estimated quadrants match. The geometric matching module can be used to determine a confidence level indicator (e.g., probability) that the first and second lesions in the CC and MLO images are the same lesion. In some embodiments, a probability map for quadrants 0, 1, 2, 3 is determined and used in conjunction with the geometric matching ML model to correlate the first lesion in the CC image and the second lesion in the MLO image.

[0062] In some cases, due at least in part to factors such as density of breast tissue, positioning of the breast, compression of the breast, or any movement of breast tissue during compression, similarity matching techniques or geometric matching techniques may be more suitable for correlating the first and second lesions in the first and second image types.

[0063] 7A-7C illustrate CC-MLO image pairs showing soft tissue lesions. In the CC-MLO image pairs 700, 702, 704, the CC image is the top image and the MLO image is the bottom image. In the illustrated image pairs, one or both of the geometric matching and similarity matching techniques may be effective in correlating a first soft tissue lesion in the CC image with a second soft tissue lesion in the MLO image in each CC-MLO image pair 700, 702, 704. However, in certain examples, due to factors such as those listed above (e.g., breast tissue motion during compression), the location of some soft tissue lesions may shift between the first and second image types, which may make it more difficult for the geometric matching techniques to correlate the first and second soft tissue lesions across the first and second image types.

[0064] For example, due to breast tissue motion, a first soft tissue lesion in the CC image may be estimated to be in one quadrant while a second soft tissue lesion in the MLO image may be estimated to be in a different quadrant, even though the first and second lesions actually represent the same object within the breast. Because the soft tissue lesions in individual CC-MLO image pairs remain similar in one or more of the shape, circumference, orientation, density, size, or depth of the soft tissue lesions within the breast, despite factors such as movement during compression, similarity matching techniques may be more effective than geometric matching techniques in correlating the first and second soft tissue lesions in the CC-MLO image pair in this exemplary scenario.

[0065] In contrast, geometric matching techniques may be more suitable for correlating first and second lesions, such as those seen in the exemplary CC-MLO image pair 800, 802, 804 shown in FIGS. 8A-8C, where each CC-MLO image pair 800, 802, 804 depicts a cluster of calcifications. Each cluster of calcifications includes multiple calcifications that may vary in shape, circumference, orientation, density, or size within the breast, and across different types of images. Because the CC and MLO images are captured at different angles, it may be difficult to correlate a cluster of calcifications in the CC image with a cluster of calcifications in the MLO image using similarity matching techniques. As shown in FIGS. 8A-8C, the clusters may look very different when projected into different planes due to the motion between the calcium elements that form the clusters. It may be difficult for a similarity matching model to identify that the cluster in the MLO image is the same object as the cluster in the CC image. Thus, geometric matching techniques may be more effective in cross-correlating the first and second lesions in a CC-MLO image pair in this exemplary scenario, where each lesion is a cluster of calcifications.

[0066] Based at least on the different strengths and effectiveness between similarity matching techniques and geometric matching techniques, embodiments disclosed herein use an ensemble matching technique (e.g., ensemble matching ML model 120 in FIG. 1) to correlate first and second lesions in CC and MLO image pairs. The ensemble matching technique operates on all types of lesions and improves the effectiveness of the correlation process. In some cases, embodiments employing the ensemble matching technique can improve the correlation process by more than 90 percent.

[0067] FIG. 9 illustrates a process flow 906 for processing CC-MLO image pairs. In one embodiment, the CC-MLO image pairs 900, 902 are received by the matching engine 106 and the geometric matching ML model 118 is applied to the CC-MLO image pairs 900, 902. The geometric matching ML model 118 has previously received geometric matching model training 908. The geometric matching model training 908 comprises training data for training the geometric matching ML model 118 and may be stored in a training data store, such as the training data store 114 of FIG. 1. In an embodiment, the geometric matching model training 908 is a set of CC-MLO image pairs with correlated first and second lesions. The geometric matching model training 908 may include both true matched lesions that demonstrate accurate correlation and false matched lesions that demonstrate inaccurate correlation. In an embodiment, such as an embodiment where the geometric matching ML model 118 is instead a rule-based AI, the geometric matching model training 908 is a set of geometric matching rules.

[0068] Using the geometric matching ML model 118, the matching engine 106 determines whether one or more lesions are present in the CC-MLO image pair 900, 902 and outputs result data 912 including a confidence level indicator that a first lesion present in the CC image 900 is correlated to a second lesion present in the MLO image 902. The geometric matching ML model 118 is used to determine the result data 912 based on features or rules learned from the geometric matching model training 908.

[0069] The similarity matching ML model 116 is applied to the CC-MLO image pairs 900, 902 by the matching engine 106. The similarity matching ML model 116 previously received similarity matching model training 910. The similarity matching model training 910 comprises training data for training the similarity matching ML model 116 and may be stored in a training data store, such as the training data store 114 of FIG. 1. In an embodiment, the similarity matching model training 910 is a set of CC-MLO image pairs with correlated lesions. The similarity matching model training 910 may include both true matched lesions that demonstrate accurate correlation and false matched lesions that demonstrate inaccurate correlation. In an embodiment, the similarity matching model training 910 is the same data set as the geometric matching model training 908. In an embodiment, the similarity matching model training 910 is a different data set than the geometric matching model training 908.

[0070] The similarity matching ML model 116 is used to determine whether one or more lesions are present in the CC image 900 and the MLO image 902, and output result data 914 including a confidence level indicator that a first lesion in the CC image 900 is correlated to a second lesion in the MLO image 902. The similarity matching ML model 116 is used to determine the result data 914 based on the similarity matching model training 910. In an embodiment, the matching engine 106 runs the similarity matching ML model 116 on the similarity matching model training 910, e.g., by generating a feature set and applying the feature set to the CC-MLO image pair 900, 902, to generate the result data 914.

[0071] The matching engine 106 applies the ensemble ML model 102 to the result data 912 (e.g., geometric matching confidence level indicators) from the geometric matching ML model 118 and the result data 914 (e.g., similarity confidence level indicators) from the similarity matching ML model 116. In an embodiment, the ensemble matching ML model 120 is also employed to analyze the CC-MLO image pairs 900, 902 directly.

[0072] The ensemble matching ML model 120 may be applied to the CC-MLO image pairs 900, 902 as output data from one or both of the geometric matching ML model 118 and the similarity matching ML model 116. For example, the result data 912 from the geometric matching ML model 118 may include the CC-MLO image pairs annotated to indicate quadrant or other location data used to calculate the geometric matching confidence level indicator. The result data 914 from the similarity matching ML model 116 may include the CC-MLO image pairs annotated to indicate shape or perimeter data for the first and second lesions. The ensemble matching ML model 120 may be applied to the CC-MLO image pairs 900, 902 independently of either the geometric matching ML model 118 or the similarity matching ML model 116.

[0073] The result data 912, 914 generally includes a geometric matching confidence level from the geometric matching ML model 118 and a similarity confidence level from the similarity matching ML model 116. In an embodiment, the result data 912, 914 comprises additional data from one or both of the geometric matching ML model 118 and the similarity matching ML model 116. For example, the geometric matching result data 912 from the geometric matching ML model 118 may also include a distance between a lesion and an anatomical landmark for each of the CC image 900 and the MLO image 902, or may include a quadrant estimate for each of a first lesion in the CC image 900 and a second lesion in the MLO image 902. The similarity result data 914 from the similarity matching ML model 116 may include similarity data such as shape, texture, orientation, or perimeter.

[0074] The matching engine 106 uses the ensemble ML model 120 to evaluate the CC and MLO images 900, 902 with the result data 912, 914 from the geometric matching and similarity matching ML models 118, 116 to generate a joint probability of correlation 904, which represents the probability that the first and second lesions in the CC and MLO image pair are the same object in the breast. The joint probability of correlation 904 may be a single value, which represents the probability that the correlation of the first and second lesions between the CC image 900 and the MLO image 902 is a reliable correlation. In an embodiment, the joint probability of correlation 904 includes at least two components: a correlation probability 918, which represents the likelihood that the correlation is reliable, and a correlation confidence level 920, which represents the likelihood that the probability calculation itself is reliable.

[0075] The ensemble ML model 120 is trained using ensemble model training 916 prior to receiving the result data 912, 914. The ensemble model training 916 comprises training data for training the ensemble matching ML model 120 and may be stored in a training data store, such as the training data store 114 of FIG. 1. In an embodiment, the ensemble model training 916 is a set of CC-MLO image pairs with correlated first and second lesions. The ensemble model training 916 may include both true matched lesions and false matched lesions. In an embodiment, the ensemble model training 916 is the same data set as the geometric matching model training 908 and the similarity matching model training 910. In an embodiment, the ensemble model training 916 is a different data set than one or both of the geometric matching model training 908 and the similarity matching model training 910.

[0076] Whether the ensemble model training 916 is on the same or a different data set compared to training data received by other models in the system, the ensemble ML model 120 generates a unique feature set. The ensemble ML model 120 develops a feature set that includes features related to images, lesions, and other ROIs, as well as features related to result data 912, 914, such as scoring, confidence levels, location data, shape data, perimeter data, etc.

[0077] The ensemble ML model 116 is used by the matching engine 106 to determine the degree to which correlations made using the geometric matching ML model 118 and the similarity matching ML model 116 are reliable and present this reliability determination as a joint probability of correlation 904. The joint probability of correlation 904 may be a single output, such as a numerical probability or a color-coded confidence level indicator, or may have two or more parts. The exemplary process flow 906 presents the joint probability of correlation as a correlation probability 918 and an associated correlation confidence level 920.

[0078] To consider an exemplary case, the particular first and second lesions may be of a type for which geometric matching is more effective, e.g., a cluster of calcifications. The ensemble matching ML model may be trained to identify when the target lesion is a calcification and, in such cases, place a larger weight on the data received from the geometric matching ML model. However, in this exemplary scenario, the cluster of calcifications may be located on the boundary between two or more quadrants. The ensemble matching ML model may also be trained to reduce the confidence in the geometric matching ML model in this situation where the lesion crosses across a quadrant boundary, since this prevents the lesion from being accurately estimated to be in one quadrant or the other. In this scenario, both the geometric matching ML model and the similarity matching ML model may determine a high correlation likelihood, which would result in a high correlation probability. However, the confidence in the similarity matching ML model is reduced due to the type of lesion, and the confidence in the geometric matching ML model is reduced due to the location of the lesion on the quadrant boundary, so the high correlation probability is accompanied by a low correlation confidence level.

[0079] The ensemble ML model 120 is trained to determine the joint probability of correlation 904 based on the ensemble model training 916. In an example, the ensemble ML model 120 runs an ML algorithm on the ensemble model training 916, e.g., to generate a feature set, and the joint probability of correlation 904 is determined when the matching engine 106 applies the feature set to the CC-MLO image pairs 900, 902 and the result data 912, 914 using the ensemble ML model. In an embodiment, the process flow 906 may further include a correlation evaluator, such as the correlation evaluator 122 of FIG. 1. The correlation evaluator may receive data from one or more of the geometric matching ML model 118, the similarity matching ML model 116, and the ensemble ML model 120 as components of the matching engine 106, and the correlation evaluator may generate and output the joint probability of correlation 904.

[0080] FIG. 10 illustrates a flow chart of a method 1012 for correlating a first lesion in a CC image to a second lesion in an MLO image. The method is performed by a matching engine, such as matching engine 106 in FIGS. 1 and 9, using a geometric matching ML model, such as geometric matching ML model 118 in FIGS. 1 and 9. Initially, an image pair is received, consisting of a first image of a first image type and a second image of a second image type, as shown in block 1000. In one embodiment, the first and second image types are CC and MLO images. The geometric matching model is used to estimate a first distance between at least one anatomical landmark and the first lesion in the CC image and a second distance between at least one anatomical landmark and the second lesion in the MLO image (block 1002). The geometric matching ML model is also used to estimate a quadrant of the breast in which the first and second lesions in the CC and MLO images are located (block 1004).

[0081] Data from the distance determination and quadrant determination are combined. In embodiments, the geometric matching ML model is used to perform both distance estimation to anatomical landmarks and quadrant assignment, while in other embodiments the geometric matching ML model may be used to perform only one or the other method of evaluation, or alternative or geometry-based evaluation of lesion location within the breast or other imaged volume.

[0082] In an embodiment, each image of an image pair is individually evaluated as a whole using a geometric matching model. For example, each CC image and MLO image in an image pair may be fully evaluated using a geometric matching ML model before the geometric matching ML model is used to determine whether a correlation exists between a first lesion identified in the CC image and a second lesion identified in the MLO image to identify the lesion and estimate the location of the lesion. One or more first lesions may be identified in the CC image and one or more second lesions may be identified in the MLO image. One or more correlations may be determined to exist between the first lesion and the second lesion in the CC image and the MLO image. For example, two first lesions, i.e., a first lesion (a) and a first lesion (b), may be identified in the CC image and two second lesions, i.e., a second lesion (a) and a second lesion (b), may be identified in the MLO image. A first correlation may be determined to exist between the first lesion (a) and the second lesion (a), and a second correlation may be determined to exist between the first lesion (b) and the second lesion (b).

[0083] The geometric matching ML model is used to calculate a confidence level indicator in block 1006. As explained above, the confidence level indicator represents the probability that a first lesion in an image of a first image type (e.g., a CC image) is the same object as a second lesion in an image of a second image type (e.g., an MLO image). In some cases, a radiologist is interested in understanding the confidence level determination based only on the geometric matching model. The confidence level indicator may then be presented to the radiologist in block 1008. In one embodiment, the confidence level indicator is displayed to the radiologist on a display device. In an example, a letter or other symbol indicia may be displayed to mark each of the first lesion in the CC image and the second lesion in the MLO image. If multiple regions of interest appear in the image pair, multiple letters or marks may be used. In an embodiment, the letter or other symbol indicia is displayed without the confidence level indicator.

[0084] Next, as shown in optional block 1010, additional information is presented to the radiologist. The additional information may include, but is not limited to, a commentary as to why the matching engine derived the presented confidence level indicator or why the present CC-MLO pair has a higher (or lower) confidence level indicator. For example, the measurement or other distance data and the identification of the anatomical landmarks used are displayed. An anatomical quadrant map may be displayed with the orientation of the first or second lesion in a particular quadrant of each image type overlaid on the anatomical quadrant map. Sources of uncertainty may be identified on the display, such as indicating that the lesion is on a quadrant boundary or indicating an inconsistency between the relationship of the first lesion to a particular anatomical landmark and the relationship of the second lesion to a particular anatomical landmark.

[0085] 11 illustrates a flow chart of a method 1112 for correlating a first lesion in a CC image to a second lesion in an MLO image. The method is performed by a matching engine, such as matching engine 106 in FIGs. 1 and 9, using a similarity matching ML model, such as similarity matching ML model 116 in FIGs. 1 and 9. The illustrated method 1112 shares several similar steps with the method 1012 shown in FIG.

[0086] The CC and MLO images are received by a matching engine at block 1000. One or more characteristics of a first lesion in the CC image and a second lesion in the MLO image are determined using a similarity matching ML model at block 1100. Characteristics assigned to the lesions include, but are not limited to, size, shape, circumference, location, density, color, orientation, texture, pattern, or depth within the breast.

[0087] In an embodiment, each image of an image pair is evaluated individually and as a whole. For example, each CC image and MLO image in the pair is fully evaluated using a similarity matching ML model before the similarity matching ML model is used to determine whether a correlation exists between a first lesion in the CC image and a second lesion in the MLO image, to identify a lesion or potential lesion, and to fully characterize the identified lesion. Similar to the geometric matching ML model, the similarity matching ML model may be used to identify and correlate one or more lesions in an image pair.

[0088] The similarity matching ML model is used to calculate a confidence level indicator in block 1006. As explained above, the confidence level indicator represents the probability that a first lesion in an image of a first image type (e.g., a CC image) is the same as a second lesion in an image of a second image type (e.g., an MLO image). In some cases, a radiologist is interested in understanding the confidence level determination based solely on the similarity matching ML model. The confidence level indicator may then be presented to the radiologist in block 1008. In one embodiment, the confidence level indicator is displayed to the radiologist on a display device. In an example, a letter or other symbolic indicia pair may be displayed to mark each of the first lesion in the CC image and the second lesion in the MLO image. If multiple regions of interest appear in the image pair, multiple letters or marks may be used. In an embodiment, the letter or other symbolic indicia is displayed without the confidence level indicator.

[0089] Next, as shown in optional block 1010, additional information is presented to the radiologist. The additional information may include, without limitation, commentary as to why the matching engine derived the presented confidence level indicator or why a particular CC-MLO pair has a higher (or lower) confidence level indicator. For example, shape, texture, orientation, or other characteristic data is displayed. One or more overlays may be presented over each of the first and second lesions, for example, showing the shape or peripheral boundary between the first and second lesions and indicating similarities or differences in their characteristics.

[0090] 12 illustrates a flowchart of a method 1212 for correlating a first lesion in a CC image to a second lesion in an MLO image. The method 1212 is performed by a matching engine, such as matching engine 106 in FIGs. 1 and 9, using an ensemble matching ML model, such as ensemble matching ML model 120 in FIGs. 1 and 9. In an embodiment, some or all of the method 1212 may be performed by a correlation evaluator, such as correlation evaluator 122 in FIG. 1. The illustrated method 1212 shares some steps with the methods 1012, 1112 shown in FIGs. 10 and 11.

[0091] CC and MLO images are received at block 1000. At block 1200, a confidence level indicator is received, generated using a geometric matching ML model. In some cases, for each of the CC and MLO images, a distance between each of the first and second lesions and one or more anatomical landmarks, or an estimated quadrant for each of the first and second lesions in the CC and MLO images, or other additional data, is also received. At block 1202, a confidence level indicator is received, generated using a similarity matching ML model. In an embodiment, one or more characteristics of the first and second lesions in the CC and MLO images, or other additional data, is also received.

[0092] The ensemble matching ML model is used to analyze the CC and MLO images, the confidence level indicators provided from each of the geometric matching ML model and the similarity matching ML model, and in some embodiments, any additional data received from one or both of the geometric matching and similarity matching ML models. Based on the analysis using the ensemble matching ML model, the matching engine generates 1204 a joint probability of correlation.

[0093] Part of the analysis may include emphasizing data received from one type of model over another, or associating greater or lesser weighting with data received from a particular model. For example, based on the type of lesion (e.g., soft tissue lesion or cluster of calcifications) from which the first and second lesions are identified, the ensemble matching ML model may determine, based on its training or previous analysis, that data from either the similarity matching ML model and the geometric matching ML model are more accurate in predicting the probability that target lesions are correlated with each other (and vice versa).

[0094] Additionally or alternatively, if the confidence level indicator generated by one model is relatively high while the confidence level indicator received from the other model is relatively low, the ensemble matching ML model may rely on the data received from the model that is provided with the relatively higher confidence level indicator (or vice versa). For example, for a dense breast, the similarity matching ML model may result in a relatively low confidence level indicator due to obscuration of the lesion by the density of the surrounding tissue, while the geometric matching location ML model can compensate for the low confidence level indicator from the similarity matching ML model by providing a relatively high confidence level indicator indicating that the CC and MLO images match. Such adjustments may be indicated to the radiologist along with the joint confidence level of the correlation, which is associated with the joint probability of the correlation. In an embodiment, such adjustments may be opaque to the radiologist, but may be reflected in the relative weights assigned to one or both of the contributions of the geometric matching ML and similarity matching ML models to the joint probability.

[0095] The joint probability of correlation may then be presented to the radiologist in block 1206. In one embodiment, the joint probability of correlation is displayed to the radiologist on a display device. The joint probability of correlation may also be used to determine other output to the display device. For example, a threshold may be associated with the joint probability of correlation such that if a user selects a lesion or ROI in a CC image and a joint probability of correlation above a predetermined threshold is associated with that lesion or ROI, the system may automatically display the MLO image with the correlated lesion or ROI.

[0096] In an example, a letter or other symbol indicia may be displayed to mark each of a first lesion in the CC image and a second lesion in the MLO image. If multiple regions of interest appear in an image pair, multiple letters or labels may be used. In an embodiment, the letter or other symbol indicia is displayed without an accompanying confidence level indicator.

[0097] Next, as shown in optional block 1010, the additional information is presented to the radiologist. The additional information may include, but is not limited to, any of the additional information discussed above in connection with methods 1012, 1112.

[0098] 13 illustrates a flowchart of a method 1312 for training an ensemble matching ML model, such as the ensemble matching ML model 120 of FIGS. 1 and 9. In one embodiment, the ensemble matching ML model is trained using a supervised training process. First, a radiologist or matching engine reviews multiple CC-MLO image pairs and labels or otherwise identifies first and second lesions in the CC and MLO images that are correlated with each other (block 1300). The labels can identify information, such as the type of lesion or whether the lesion is cancerous, for each of the first and second lesion pairs.

[0099] The labeled CC-MLO image pairs are input to an ensemble matching ML model at block 1302. Training data for the similarity matching ML model and for the geometric matching ML model are input to the ensemble matching ML model at block 1304. The ensemble matching ML model is trained at block 1306 using the human labeled CC-MLO image pairs and the training data for the similarity and geometric matching ML models. In an embodiment, the ensemble matching ML model may generate a feature set based on the human labeled CC-MLO image pairs and the training data for the similarity and geometric matching ML models. The trained ensemble matching ML model is then used in a lesion identification and correlation system, such as the system 100 shown in FIG. 1, at block 1308.

[0100] In some embodiments, the similarity matching and geometric matching ML models are trained using known training techniques. In addition, human-labeled CC-MLO image pairs are also used to train the similarity matching ML model and the geometric matching ML model. Each model can be trained independently of the other models. Alternatively, the geometric matching, similarity matching, and ensemble matching ML models may be trained in one step or training process.

[0101] Other embodiments may arrange the blocks within the flowcharts shown in FIGS. 10-13 in a different order than that shown.

[0102] 14 is a block diagram illustrating example physical components of a computing device. The computing device 1400 can be any computing device used in conjunction with the identification system 100, such as the computing system 102, the X-ray computing device 112, and the computing system 110.

[0103] In the example shown in FIG. 14, computing device 1400 includes at least one processing device (collectively, processing device 1402), at least one memory (collectively, memory 1408), and at least one bus (collectively, bus 1422) that couples memory 1408 to processing device 1402. Exemplary processing devices 1402 include, but are not limited to, central processing units, microprocessors, application specific integrated circuits, digital signal processors, and graphic processors. Exemplary memory 408 includes random access memory ("RAM") 1410 and read only memory ("ROM") 1412. A basic input / output system containing basic routines that help to transfer information between elements within computing device 1400, such as during start-up, is stored in ROM 1412.

[0104] The computing device 1400 further includes one or more storage devices (collectively, storage devices 1414). The storage devices 1414 may store software instructions and data. For example, the storage devices 1414 may store the GUI shown in FIG.

[0105] The storage device 1414 is connected to the processing device 1402 through a storage controller (not shown), which is connected to the bus 1422. The storage device 1414 and its associated computer-readable storage media provide non-volatile, non-transitory data storage for the computing device 1400. Although the description of computer-readable storage media contained herein refers to storage devices such as hard disks or solid-state disks, it will be understood by those skilled in the art that a computer-readable data storage medium can include any available tangible physical device or article of manufacture from which the processing device 1402 can read data and / or instructions. In some embodiments, the computer-readable storage medium includes an entirely non-transitory medium.

[0106] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented in any manner or technology for storage of information, such as computer-readable software instructions, data structures, program modules, or other data. Exemplary types of computer-readable data storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, digital versatile disks ("DVDs"), other optical storage media, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 1400.

[0107] According to some embodiments, computing device 1400 may operate in a networked environment using logical connections to remote network devices through a network 1452, such as a wireless network, the Internet, or another type of network. Computing device 1400 may connect to network 1452 through a network interface unit 1404, which is connected to bus 1422. It should be appreciated that network interface unit 1404 may also be utilized to connect to other types of networks and remote computing systems.

[0108] The computing device 1400 also includes an input / output controller 1406 for receiving and processing input from a number of other devices, including a touch-sensitive user interface display screen or another type of input device. Similarly, the input / output controller 1406 may provide output to the touch-sensitive user interface display screen or other type of output device.

[0109] As briefly mentioned above, the storage device 1414 and RAM 1410 of the computing device 1400 can store software instructions and data. The software instructions include an operating system 1418 suitable for controlling the operation of the computing device 1400. The storage device 1414 and / or RAM 1410 also store software instructions that, when executed by the processing device 1402, cause the computing device 1400 to provide the functionality discussed herein.

[0110] Working Example:

[0111] Illustrative examples of the systems and methods described herein are provided below. An embodiment of the system or method described herein may include any one or more of the following notes and any combination thereof:

[0112] Appendix 1. A method for correlating regions of interest (ROIs) in an image pair comprising a cranio-caudal (CC) image and a medio-lateral oblique (MLO) image, the method comprising: receiving, by an ensemble matching machine learning (ML) model, data from a similarity matching ML model, the data from the similarity matching ML model including at least the ROIs of the matched pair and a first confidence level indicator associated with the ROIs of the matched pair; receiving, by the ensemble matching ML model, data from a geometric matching (GM) model, the data from the GM model including at least the ROIs of the matched pair and a second confidence level indicator; determining, by the ensemble matching ML model, a joint probability of correlation based on an evaluation of each of the first and second confidence levels by the ensemble matching ML model, the joint probability of correlation providing a probability that an ROI in the CC image is correlated to a corresponding ROI in the MLO image, and vice versa; and providing the joint probability of correlation to an output device.

[0113] Appendix 2. The method of Appendix 1, further comprising receiving data associated with the training CC-MLO image pairs, and training an ensemble matching ML model using the data associated with the training CC-MLO image pairs.

[0114] Appendix 3. The method of any one of Appendixes 1 to 2, further comprising determining, by the ensemble matching ML model, a third confidence level indicator based on the assessment of each of the first and second confidence levels and the joint probability of the correlation, the third confidence level indicator being a likelihood of confidence associated with the joint probability of the correlation.

[0115] Addendum 4. The method of any of Addendums 1-3, wherein the joint probability is the probability that the similarity correlation and the GM correlation adequately correlate the CC-ROI and the MLO-ROI.

[0116] Addendum 5. The method of any of Addendums 1-4, wherein providing the joint probability of correlation to the output device includes a numerical value associated with the joint probability of correlation.

[0117] Addendum 6. The method of any of Addendums 1-5, wherein each image of the matched lesion pair is a whole breast image.

[0118] Addendum 7. The method of any of Addendums 1-6, wherein each image of the matched lesion pair contains only an ROI for each of the CC image and the MLO image.

[0119] Addendum 8. The method of any of Addendums 1-7, wherein providing the joint probability of correlation to an output device includes a numerical display.

[0120] Addendum 9. The method of any of Addendums 1-8, wherein providing the joint probability of correlation to an output device includes receiving a series of CC-ROIs and presenting an MLO-ROI in response to receiving a selection of the CC-ROI.

[0121] Appendix 10. The method of Appendix 9, further comprising: in response to receiving a selection of a CC-ROI, determining that a joint probability of correlation exceeds a predetermined threshold; and in response to determining that the joint probability of correlation exceeds a predetermined threshold, presenting an MLO-ROI.

[0122] Addendum 11. The method of any of Addendums 1-10, wherein the matched pair of ROIs includes a similarity correlation between a CC-ROI in the CC-image and an MLO-ROI in the MLO image, and the first confidence level indicator is a probability associated with the correlation between the CC-ROI and the MLO-ROI.

[0123] Addendum 12. The method of any of Addendums 1-11, wherein the second confidence level indicator is a probability associated with the GM correlation between the CC-ROI and the MLO-ROI.

[0124] Addendum 13. The method of any of Addendums 1-12, further comprising displaying a pair of symbols on the output display, each symbol of the pair of symbols marking a ROI of a matched pair of ROIs.

[0125] Addendum 14. The method of any of Addendums 1-13, wherein the data from the GM model comprises location data for each ROI of the matched pair of ROIs.

[0126] Addendum 15. The method of Addendum 14, wherein the second confidence level indicator indicates a probability that a first location of a first ROI of the matched pair of ROIs and a second location of a second ROI of the matched pair of ROIs are the same location.

[0127] Addendum 16. The method of Addendum 14 or 15, wherein the CC image and the MLO image each depict the breast and the GM model logically divides the breast into quadrants.

[0128] Addendum 17. The method of any of Addendums 1-16, wherein the data from the similarity model comprises characteristic data for each ROI of the matched pair of ROIs.

[0129] Addendum 18. The method of Addendum 17, wherein the first confidence level indicator indicates a degree of similarity between a first set of characteristics associated with a first ROI of the matched pair of ROIs and a second set of characteristics associated with a second ROI of the matched pair of ROIs.

[0130] Clause 19. The method of clause 17 or 18, wherein the characteristic data includes one or more of size, shape, one or more perimeter, location, density, one or more colors, orientation, texture, pattern, and depth.

[0131] Appendix 20. A system for ensemble matching of cranio-caudal (CC) and medio-lateral oblique (MLO) images, comprising: at least one processor in communication with at least one memory; and an ensemble matching module executing on the at least one processor and configured during operation to receive from a similarity matching model a matched CC-MLO image pair and a similarity confidence level indicator associated with the matched CC-MLO image pair; receive from a geometric matching (GM) model a matched CC-MLO image pair and a GM confidence level indicator associated with the matched CC-MLO image pair; apply the ensemble matching model; determine an ensemble confidence level based on an ensemble machine learning (ML) algorithm trained on a plurality of matched CC-MLO pairs; associate the ensemble confidence level with the matched CC-MLO image pair; and output the ensemble confidence level with the matched CC-MLO image pair.

[0132] Clause 21. The system of clause 20, further comprising an image acquisition module.

[0133] Addendum 22. The system of Addendum 20 or 21, wherein the matched CC-MLO image pair comprises a first region of interest (ROI) identified in the CC image and a second ROI identified in the MLO image, and either the similarity ML model or the GM model assigns correlation to the first ROI and the second ROI.

[0134] Addendum 23. The system of Addendum 22, wherein the ensemble matching module further receives location data from the GM model, the location data being associated with the correlation between the first ROI and the second ROI.

[0135] Addendum 24. The system of Addendum 22 or 23, wherein the ensemble matching module further receives shape data from the similarity ML model, the shape data being associated with the correlation between the first ROI and the second ROI.

[0136] Addendum 25. The system of any of Addendums 22-24, wherein the ensemble matching module further receives marginal data from the similarity ML model, the marginal data being associated with the correlation between the first ROI and the second ROI.

[0137] Addendum 26. A system as described in any of Addendums 22-25, wherein the similarity matching model determines a matched CC-MLO image pair by identifying a CC region of interest (ROI) in a CC-image received from the image acquisition module, searching the MLO image received from the image acquisition module for an MLO-ROI, the MLO image including a plurality of regions, the similarity matching model examining each of the plurality of regions, determining at least one similarity characteristic of each of the CC-ROI and the MLO-ROI, and correlating the CC-ROI and the MLO-ROI based on the at least one similarity characteristic.

[0138] Appendix 27. A method for ensemble matching craniocaudal (CC) images and mediolateral oblique (MLO) images, comprising: training an ensemble matching model using training CC-MLO image pairs, the training CC-MLO image pairs including paired CC-MLO images with regions of interest (ROIs) that are correlated with high confidence and paired CC-MLO images with ROIs that are miscorrelated; receiving matched CC-MLO pairs from a similarity matching model, the matched CC-MLO pairs including a similarity confidence score; and A method comprising: receiving CC-MLO pairs from a geometric matching model, the matched CC-MLO pairs including location confidence scores; determining an ensemble confidence level for correlation between ROIs in one or more paired CC-MLO images by an ensemble matching model; analyzing the matched CC-MLO pairs along with similarity confidence scores and location confidence scores based on training CC-MLO image pairs; calculating the ensemble confidence level; and generating an output representation of the ensemble confidence level.

[0139] This disclosure has described several embodiments of the technology with reference to the accompanying drawings, in which only some of the possible embodiments are shown. However, other aspects may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the possible embodiments to those skilled in the art.

[0140] Although various embodiments and examples are described herein, those skilled in the art will understand that many modifications may be made thereto within the scope of the present disclosure. Thus, specific structures, acts, or media are disclosed only as illustrative examples. Examples according to the present technology may also combine elements or components that are generally disclosed but not explicitly illustrated in combination, unless otherwise stated herein. Thus, the scope of the present disclosure is not intended to be limited in any way by the examples provided.

Claims

1. 1. A method for correlating a region of interest (ROI) in an image pair comprising a cranio-caudal (CC) image and a medio-lateral oblique (MLO) image, the method comprising: receiving, by an ensemble matching machine learning (ML) model, data from a similarity matching ML model, the data from the similarity matching ML model including at least matched pair ROIs and a first confidence level indicator associated with the matched pair ROIs; receiving, by the ensemble matching ML model, data from a geometric matching (GM) model, the data from the GM model including at least the matched pair of ROIs and a second confidence level indicator; determining, by the ensemble matching ML model, a joint probability of correlation based on an assessment of each of the first and second confidence levels by the ensemble matching ML model, the joint probability of correlation providing a probability that the ROI in the CC image is correlated to a corresponding ROI in the MLO image, and vice versa; providing the joint probability of the correlations to an output device; and A method comprising:

2. Receiving data associated with training CC-MLO image pairs; training the ensemble matching ML model using data associated with the training CC-MLO image pairs; The method of claim 1 further comprising:

3. 3. The method of claim 1, further comprising determining, by the ensemble matching ML model, a third confidence level indicator based on the assessment of each of the first and second confidence levels and the joint probability of the correlation, the third confidence level indicator being a likelihood of confidence associated with the joint probability of the correlation.

4. The method of any of claims 1-3, wherein the joint probability is the probability that the similarity correlation and the GM correlation adequately correlate the CC-ROI and the MLO-ROI.

5. The method of any of claims 1-4, wherein providing the joint probability of correlation to an output device comprises a numerical value associated with the joint probability of correlation.

6. The method of any of claims 1-5, wherein each image of the matched lesion pair is a whole breast image.

7. The method of any of claims 1-6, wherein each image of the matched lesion pair contains only the ROI for each of the CC image and the MLO image.

8. The method of any of claims 1-7, wherein providing the joint probability of correlation to an output device comprises a numerical display.

9. providing the joint probability of correlation to an output device, receiving a series of said CC-ROIs; presenting the MLO-ROI in response to receiving a selection of the CC-ROI; The method of any one of claims 1 to 8, comprising:

10. determining, in response to receiving a selection of the CC-ROI, that a joint probability of the correlation exceeds a predetermined threshold; presenting the MLO-ROI in response to determining that the joint probability of the correlation exceeds a predetermined threshold; The method of claim 9 further comprising:

11. A method according to any of claims 1-10, wherein the matched pair of ROIs comprises a similarity correlation between a CC-ROI in a CC-image and an MLO-ROI in an MLO image, and the first confidence level indicator is a probability associated with the correlation between the CC-ROI and the MLO-ROI.

12. The method of any of claims 1-11, wherein the second confidence level indicator is a probability associated with a GM correlation between the CC-ROI and the MLO-ROI.

13. displaying a pair of symbols on the output display, each symbol of the pair of symbols marking an ROI of the matched pair of ROIs; The method of any of claims 1-12, further comprising:

14. A method according to any preceding claim, wherein the data from the GM model comprises location data for each ROI in the matched pair of ROIs.

15. 15. The method of claim 14, wherein the second confidence level indicator indicates a probability that a first location of a first ROI of the matched pair of ROIs and a second location of a second ROI of the matched pair of ROIs are the same location.

16. The method of claim 14 or 15, wherein the CC images and the MLO images each depict a breast, and the GM model logically divides the breast into quadrants.

17. A method according to any preceding claim, wherein the data from the similarity model comprises characteristic data for each ROI in the matched pairs of ROIs.

18. 18. The method of claim 17, wherein the first confidence level indicator indicates a degree of similarity between a first set of characteristics associated with a first ROI of the matched pair of ROIs and a second set of characteristics associated with a second ROI of the matched pair of ROIs.

19. 19. The method of claim 17 or 18, wherein the characteristic data includes one or more of size, shape, one or more perimeter, location, density, one or more colors, orientation, texture, pattern, and depth.

20. 1. A system for ensemble matching of craniocaudal (CC) and mediolateral oblique (MLO) images, comprising: at least one processor in communication with at least one memory; an ensemble matching module, the ensemble matching module executing on the at least one processor, during operation: receiving a matched CC-MLO image pair and a similarity confidence level indicator associated with the matched CC-MLO image pair from a similarity matching model; receiving the matched CC-MLO image pair and a GM confidence level indicator associated with the matched CC-MLO image pair from a geometric matching (GM) model; applying an ensemble matching model to determine an ensemble confidence level based on an ensemble machine learning (ML) algorithm trained on the plurality of matched CC-MLO pairs; Associating the ensemble confidence level with the matched CC-MLO image pair; outputting the ensemble confidence level together with the matched CC-MLO image pair; an ensemble matching module configured to: A system comprising:

21. The system of claim 20 further comprising an image acquisition module.

22. 22. The system of claim 20 or 21, wherein the matched CC-MLO image pair comprises a first region of interest (ROI) identified in the CC image and a second ROI identified in the MLO image, and wherein either the similarity ML model or the GM model assigns correlation to the first ROI and the second ROI.

23. 23. The system of claim 22, wherein the ensemble matching module further receives location data from the GM model associated with a correlation between the first ROI and the second ROI.

24. 24. The system of claim 22 or 23, wherein the ensemble matching module further receives shape data associated with a correlation between the first ROI and the second ROI from the similarity ML model.

25. The system of any of claims 22-24, wherein the ensemble matching module further receives marginal data associated with the correlation between the first ROI and the second ROI from the similarity ML model.

26. The similarity matching model is Identifying a CC region of interest (ROI) in a CC-image received from an image acquisition module; searching an MLO image received from the image acquisition module for an MLO-ROI, the MLO image including a plurality of regions, the similarity matching model probing each region of the plurality of regions; determining at least one similarity characteristic of each of the CC-ROI and the MLO-ROI; correlating the CC-ROI and the MLO-ROI based on the at least one similarity characteristic; The system of any of claims 22-25, further comprising: determining the matched CC-MLO image pair by:

Citation Information

Patent Citations

  • Focus matching method and device

    CN110674885A

  • Mammary gland image focus matching method and device and storage medium

    CN112561908A

  • 3D reconstruction of the breast from two x-ray mammograms

    JP2003527700A

  • Medical image processing method and apparatus, electronic medical device, and storage medium

    US20210343016A1

  • Systems and methods for correlating regions of interest in multiple imaging modalities

    WO2021195370A1