System and method for correlating region of interest in multiple imaging modalities

Abstract

To solve the problem that it is difficult to correlate a lesion location from an X-ray image to an ultrasound image.SOLUTION: A method and a system for identifying a region of interest in breast tissue utilize artificial intelligence to confirm that a target lesion identified during imaging the breast tissue by using a first imaging modality (e.g. x-ray imaging) has been identified by using a second imaging modality (e.g. ultrasound imaging). A lesion matching engine 110 utilizes a machine learning classifier algorithm trained on cases of x-ray images and corresponding ultrasound images in which lesions were identified. The lesion matching engine analyzes a target lesion identified with x-ray imaging and a potential lesion identified with ultrasound imaging to determine a likelihood that the target lesion is the same as the potential lesion. An indicator of a confidence level for lesion match is presented on a display of a computing device to aid a healthcare provider in locating a lesion in breast tissue.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This application was filed as a PCT International Patent Application on March 25, 2021, and claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 000,702, filed on March 27, 2020, the disclosure of which is incorporated by reference in its entirety. [Background technology]

[0002] background Medical imaging provides a non-invasive method for visualizing a patient's internal structures. Visualization methods can be used to screen and diagnose cancer in patients. For example, early screening can detect lesions within the breast that may be cancerous, allowing treatment at an early stage of the disease.

[0003] Mammography and tomosynthesis utilize x-ray radiation to visualize breast tissue. These techniques are often used to screen patients for potentially cancerous lesions. A traditional mammogram involves taking two-dimensional images of the breast from various angles. Tomosynthesis creates multiple x-ray images of each individual layer or slice of the breast, throughout its thickness. Tomosynthesis provides a three-dimensional visualization of the breast. Mammography and tomosynthesis are typically performed with the patient standing and with compression of the patient's breast tissue.

[0004] If a lesion is found, a diagnostic ultrasound may be performed as the next step to determine if a tumor is present. Ultrasound uses sound waves, typically generated by a piezoelectric transducer, to image the patient's tissue. Ultrasound imaging allows the tissue to be viewed from different angles, making solid masses easier to identify. An ultrasound probe generates and focuses arc-shaped sound waves that travel through the body and are partially reflected from different tissue layers of the patient. The reflected sound waves are detected by a transducer, converted into electrical signals, and processed by an ultrasound scanner to form an ultrasound image of the tissue. Ultrasound examinations are typically performed with the patient in a supine position and with the patient's breast tissue uncompressed. Summary of the Invention [Problem to be solved by the invention]

[0005] During diagnostic ultrasound imaging procedures, technicians and radiologists often have difficulty navigating and locating lesions previously identified during x-ray imaging. Because the former is performed with the patient upright and breast tissue compressed, and the latter is performed with the patient lying down and breast tissue uncompressed, correlating the location of a lesion from an x-ray image to an ultrasound image is difficult. Furthermore, ultrasound images have a different level of contrast and a different appearance than x-ray images. Lesions detected by x-ray imaging are becoming increasingly smaller due to improved technology, making it more difficult to locate small lesions in ultrasound images.

[0006] It is against this background that the present disclosure has been made. Techniques and improvements are provided herein. [Means for solving the problem]

[0007] overview An embodiment of the present disclosure relates to a method for locating a region of interest within a breast. An indication of the location of a target lesion within the breast is received by a computing device. The target lesion is identified during imaging of the breast using a first imaging modality. Images of the breast are acquired by a second imaging modality, and a potential lesion is identified within the images. The first image containing the target lesion is analyzed by a lesion matching engine operating on a computing system and compared to a second image containing the potential lesion using artificial intelligence. A probability that the potential lesion corresponds to the target lesion is determined, and an indicator of the level of confidence is output for display on a graphical user interface.

[0008] In another aspect, a lesion identification system includes a processing device and a memory storing instructions that, when executed by the processing device, facilitate performance of operations including accessing an x-ray image of a breast, the x-ray image including an identified lesion indicated with a visual marker, receiving an ultrasound image of the breast, the ultrasound image including indications of a potential lesion, analyzing the potential lesion and the identified lesion with an artificial intelligence lesion classifier, generating a confidence score indicating a likelihood that the potential lesion in the ultrasound image matches the identified lesion in the x-ray image, and displaying an output associated with the confidence score on a graphical user interface.

[0009] In yet another aspect, a non-transitory machine-readable storage medium stores executable instructions that, when executed by a processor, facilitate performance of operations including: acquiring target lesion data from a data store, the data being acquired by x-ray imaging and including at least an image of the target lesion and coordinates of a location of the target lesion within the breast; recording an image of the breast acquired by ultrasound imaging; identifying a general region of interest in the recorded image of the breast acquired by ultrasound based on the coordinates of the target lesion; identifying a potential lesion in the general region of interest; analyzing the potential lesion using an artificial intelligence lesion classifier and comparing the potential lesion to the target lesion to determine a confidence level that the potential lesion corresponds to the target lesion; and outputting an indicator of the confidence level on a graphical user interface.

[0010] In another aspect, a lesion identification system includes at least one optical camera, a projector, a processing device, and a memory storing instructions that, when executed by the processing device, facilitate performance of operations comprising: capturing at least one optical image of a patient's breast using the at least one optical camera, accessing at least one tomosynthesis image of the breast, receiving indications of a target lesion on the at least one tomosynthesis image, co-registering the at least one optical image and the at least one tomosynthesis image of the breast by analyzing with artificial intelligence algorithms for region matching and non-rigid deformation models, creating a probability map based on the co-registration and the indications of the target lesion, the map indicating a likelihood that the target lesion is located at each of a plurality of points on the breast, and projecting the probability map onto the breast with a projector.

[0011] The details of one or more techniques are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of these techniques will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an exemplary system for identifying a region of interest within a breast. [Figure 2] FIG. 2 is a schematic diagram of an exemplary system for managing healthcare data, including imaging data. [Figure 3] FIG. 3 is a schematic diagram of an exemplary X-ray imaging system. [Figure 4] FIG. 4 is a perspective view of the X-ray imaging system of FIG. [Figure 5] FIG. 5 illustrates the x-ray imaging system in breast positioning for the left longitudinal section (LMLO) imaging direction. [Figure 6] FIG. 6 illustrates an exemplary ultrasound imaging system. [Figure 7] FIG. 7 illustrates an example of the ultrasound imaging system of FIG. 6 being used on a patient's breast. [Figure 8] FIG. 8 is a schematic diagram of an exemplary computing system that can be used to implement one or more aspects of the present disclosure. [Figure 9] FIG. 9 is a flow diagram illustrating an exemplary method for identifying a region of interest within a breast. [Figure 10] FIG. 10 is a diagram showing a display example of the graphical user interface of FIG. [Figure 11] FIG. 11 is a schematic diagram of an exemplary system for correlating lesions in images obtained from different modalities. [Figure 12] FIG. 12 is a flow diagram illustrating an exemplary method for correlating lesions. DETAILED DESCRIPTION OF THE INVENTION

[0013] Detailed Description The present disclosure is directed to a system and method for locating lesions within breast tissue using an imaging device. In particular, a computing system employs machine learning to navigate to potential lesions and provide a confidence level indicator for the correlation between lesions identified in the breast by ultrasound imaging and lesions identified by x-ray imaging.

[0014] A key step in assessing breast health is a screening x-ray imaging procedure (e.g., mammography or tomosynthesis). In approximately 10-15% of cases, x-ray images identify a lesion that causes the patient to be referred back for additional imaging to determine whether the lesion is potentially cancerous. At that time, imaging is performed, usually ultrasound. Ultrasound images can more accurately distinguish between cysts and solid masses, and if a biopsy is required, ultrasound is the preferred imaging modality.

[0015] Despite the advantages of ultrasound, clinicians sometimes find it difficult to locate the same lesions identified in x-ray imaging while performing ultrasound imaging. This is due to three main factors. First, the position of the breast is different in ultrasound procedures compared to x-ray imaging procedures (e.g., mammography or tomosynthesis). For x-ray imaging, the patient is typically in an upright position with the breast compressed, whereas for ultrasound imaging, the patient is typically in a supine position with the breast uncompressed. This misalignment can make it difficult to correlate lesions found in x-ray images with images generated by ultrasound.

[0016] The second reason is that ultrasound imaging modality appears different from x-ray imaging, with different contrast, making it difficult to be confident that a lesion identified on ultrasound is the same one previously identified on x-ray imaging.

[0017] Third, as technology continues to improve, the lesions that can be detected by x-ray imaging are becoming smaller and smaller, making it more difficult for medical professionals to spot lesions in ultrasound images.

[0018] The computing system described herein operates to provide a confidence level indicator of the correlation between lesions identified on ultrasound and lesions identified on radiographic imaging. The computing system uses an artificial intelligence (AI) model trained on a library of digital breast tomosynthesis (DBT) cases and corresponding radiologist-correlated diagnostic ultrasound cases. The AI ​​model analyzes new DBT and ultrasound images to determine whether one lesion correlates with another. Furthermore, the AI ​​model can provide a confidence level indicator to the user to assist them in determining whether they have found the same lesion. In some examples, the AI ​​model can be employed in combination with electromagnetic or optical tracking input to speed navigation to target areas during ultrasound examinations and reduce the image set to be analyzed. In some examples, the AI ​​model utilizes morphological factors to calculate the confidence level. The morphological factors include morphological features of breast tissue that can be used to identify specific areas of the breast. In some examples, the morphological factors are visual features on the surface of the breast, such as moles, freckles, and tattoos.

[0019] 1 illustrates an exemplary lesion identification system 100 for locating a region of interest within a breast. System 100 includes a computing system 102, an X-ray imaging system 104, and an ultrasound imaging system 106. In some examples, lesion identification system 100 operates to guide a medical professional to a region of interest within the breast during ultrasound imaging based on data collected during the X-ray imaging procedure in which the region of interest was initially identified. In some examples, lesion identification system 100 provides a confidence level indicator to a medical professional on a display to assist the medical professional in confirming that a lesion visible on the ultrasound image is the same lesion previously identified in the X-ray image.

[0020] Computing system 102 operates to process and store information received from x-ray imaging system 104 and ultrasound imaging system 106. In the example of FIG. 1 , computing system 102 includes a lesion matching engine 110 and a data store 112. In some examples, lesion matching engine 110 and data store 112 are contained within the memory of computing system 102. In some examples, computing system 102 accesses lesion matching engine 110 and data store 112 from a remote server, such as a cloud computing environment. While FIG. 1 depicts computing system 102 as separate from other components of system 100, it may also be incorporated into x-ray computing device 116, ultrasound computing device 118, or other computing devices utilized in patient care. In some examples, computing system 102 includes more than one computing device.

[0021] The lesion matching engine 110 operates to analyze X-ray images of target lesions and ultrasound images of potential lesions to determine whether the potential lesions are the same as the target lesions. In the example of FIG. 1, a DBT training data store 122 is used to train the artificial intelligence unit 124. The DBT training data store 122 stores multiple instances of identified lesions, each associated with an ultrasound image and an X-ray image. Examples include matches confirmed by medical personnel. The artificial intelligence unit 124 analyzes these instances using machine learning algorithms to identify features that can be used to match ultrasound images with X-ray images. These features are used to generate an image classifier.

[0022] Various machine learning techniques can be utilized to generate the lesion classifier. In some examples, the machine learning algorithm is a supervised machine learning algorithm. In other examples, the machine learning algorithm is an unsupervised machine learning algorithm. In some examples, the machine learning algorithm is based on an artificial neural network. In some examples, the neural network is a deep neural network (DNN). In some examples, the machine learning algorithm is a convolutional deep neural network (CNN). In some examples, a combination of two or more networks is utilized to generate the classifier. In some examples, two or more algorithms are utilized to generate features from example case data.

[0023] The resulting trained machine learning classifier is utilized by the image analyzer 126 to compare sets of X-ray and ultrasound images. Various metrics are utilized to compare lesions, including shape, color, margins, orientation, texture, pattern, density, consistency, size, and depth within the breast. In some examples, the metrics are numerical. The confidence evaluator 128 operates in conjunction with the image analyzer 126 to determine a level of confidence that a potential lesion identified in the ultrasound imaging is the same as a lesion identified in the X-ray imaging. In some examples, a confidence score is generated by the confidence evaluator 128. In some examples, the confidence level may indicate a category of confidence, such as “high,” “medium,” or “low.” In alternative examples, the confidence level is provided as a percentage, such as “99%, “75%,” or “44%.” Finally, the graphical user interface (GUI) 130 operates to present information on a display of a computing device. In some examples, the GUI 130 displays a confidence level indicator on one or more images of the tissue being analyzed.

[0024] In some examples, the lesion matching engine 110 operates to perform region matching on two different types of images. In some examples, region matching is performed using artificial intelligence algorithms to match breast regions between two different imaging modalities using morphological factors and other features of breast tissue. In some examples, the analysis uses co-registration techniques to provide probabilistic values ​​for where a lesion is expected to be located on the breast. In some examples, the artificial intelligence model operates in conjunction with a non-rigid deformable model to determine the likelihood that a target lesion and a potential lesion are the same.

[0025] Data store 112 operates to store information received from X-ray imaging system 104, ultrasound imaging system 106, and lesion matching engine 110. In some examples, data store 112 is actually two or more separate data stores. For example, one data store may be a remote data store that stores images from an X-ray imaging system. Another data store may be housed locally within computing system 102. In some examples, data store 112 may be part of an electronic medical record (EMR) system.

[0026] The X-ray imaging system 104 operates to capture images of breast tissue using X-ray radiation. The X-ray imaging system 104 includes an X-ray imaging device 114 and an X-ray computing device 116 in communication with the X-ray imaging device 114. In some examples, the X-ray imaging system 104 performs digital breast tomosynthesis (DBT). The X-ray imaging device 114 is described in further detail in connection with FIGS. 3-5 . The X-ray computing device 116 operates to receive input from a medical professional H to operate the X-ray imaging device 114 and to display images received from the X-ray imaging device 114.

[0027] The ultrasound imaging system 106 operates to capture images of breast tissue using ultrasound. The ultrasound imaging system 106 is described in further detail in connection with Figures 6-7. The ultrasound imaging system 106 includes an ultrasound computing device 118 and an ultrasound imaging device 120. The ultrasound computing device 118 operates to receive input from a medical professional H to operate the ultrasound imaging device 120 and to display images received from the ultrasound imaging device 120.

[0028] 1 illustrates how information obtained from an X-ray imaging system 104 may be utilized by an ultrasound imaging system 106. A medical professional H operates an X-ray computing device 116 to take X-ray images of a patient P's breast using an X-ray imaging device 114. The X-ray images may be taken as part of a routine medical examination. During screening, the medical professional H identifies one or more regions of interest in the patient P's breast that require further analysis to determine whether lesions within those regions of interest are potentially cancerous and require a biopsy.

[0029] In some examples, coordinates of the region of interest may be recorded by the X-ray computing device 116 and transmitted to the computing system 102. The coordinates recorded by the X-ray computing device 116 are analyzed using a tissue deformation model, as described in co-pending U.S. patent application numbered Attorney Docket 04576.0112USP1, which is incorporated by reference in its entirety.

[0030] In some examples, a first set of coordinates identifies a location of a lesion identified while the breast is compressed. The first set of coordinates is transformed into a second set of coordinates that identifies a predicted location of a lesion identified while the breast is not compressed. A region of interest in the ultrasound image corresponding to the second coordinates is identified. This allows a medical professional to identify potential lesions in the ultrasound image.

[0031] The output of the analysis is a set of predicted coordinates that can be communicated to ultrasound computing device 118 for use in a subsequent imaging examination, which may be at a different location than where the imaging procedure took place. Medical personnel H operating ultrasound computing device 118 uses the predicted coordinates to navigate to a region of interest on patient P's breast using ultrasound imaging device 120.

[0032] In some embodiments, the x-ray image is displayed on a user interface of the ultrasound computing device 118 along with the ultrasound image received from the ultrasound imaging device 120. Additional information, such as predicted coordinates of the region of interest and the appearance of biomarkers on the image of the patient's breast, can be displayed on the ultrasound computing device 118. In some embodiments, a visual marker is displayed on the image indicating the location of the target lesion. In some instances, a probability map may be displayed on the image indicating where the target lesion is most likely to be located.

[0033] A medical professional H operating an ultrasound computing device 118 locates potential lesions in the ultrasound images that potentially match lesions previously identified in X-ray images for the same patient P. The ultrasound images and indications of potential lesions are transmitted to the computing system 102 for analysis. In some examples, the mammography image, the target region of interest, and the B-mode image are displayed on the same GUI. The GUI 130 helps to visually guide the ultrasound system operator to the region of interest while also automating documentation of the ultrasound probe position, orientation, and annotations.

[0034] In some examples, the x-ray images containing the identified lesions and the ultrasound images containing the potential lesions are analyzed by the lesion matching engine 110 of the computing system 102. The lesion matching engine 110 outputs a confidence level indicator for the potential lesions and communicates the confidence level indicator to the ultrasound computing device 118. The confidence level indicator can be a number, color, or category that is displayed in a GUI on the ultrasound computing device 118. An exemplary GUI is illustrated in FIG.

[0035] In some embodiments, the lesion matching engine 110 operates to generate a probability mapping, as described in FIGS. 11-12 . In some embodiments, an optical camera captures images of the breast being examined by ultrasound. Previously obtained x-ray images of the breast are accessed and analyzed using co-registration techniques and artificial intelligence region matching. The probability mapping generated from the analysis is visually projected onto the breast to assist medical professional H in locating the target lesion. An exemplary GUI 130 including the probability mapping is shown in FIG. 11 .

[0036] 2 is a schematic diagram of an example system 150 for managing healthcare data, including imaging data. The system 150 includes multiple computing components in communication with each other via a communications network 152. The computing components may include a tracking system 154, a navigation system 156, an electronic medical record (EMR) system 158, and a display system 160, in addition to the computing system 102, the X-ray imaging system 104, and the ultrasound imaging system 106 described in FIG.

[0037] 1 as functional blocks, it should be noted that different systems may be integrated into a common device and communication links may be coupled between less than all of the systems. For example, tracking system 154, navigation system 156, and display system 160 may be included in an acquisition workstation or technician workstation that may control the acquisition of images in a radiology suite. Alternatively, navigation system 156 and tracking system 154 may be integrated into ultrasound imaging system 106 or may be provided as stand-alone modules with separate communication links to display 160, x-ray imaging system 104, and ultrasound imaging system 106. Similarly, one skilled in the art will additionally appreciate that communication network 152 may be a local area network, a wide area network, a wireless network, the Internet, an intranet, or other similar communication network.

[0038] In one embodiment, the X-ray imaging system 104 is a tomosynthesis acquisition system that captures a set of projection images of a patient's breast as an X-ray tube scans across a path on the breast. The set of projection images is then reconstructed into a three-dimensional volume that can be viewed as slices or slabs along any plane. The three-dimensional volume may be stored locally on the X-ray imaging system 104 (either on the X-ray imaging device 114 or on the X-ray computing device 116) or in a data store, such as data store 112, in communication with the X-ray imaging system 104 via a communications network 152. In some examples, the three-dimensional volume may be stored in the patient's file within an electronic medical record (EMR) system 158. Additional details regarding exemplary X-ray imaging systems are described in connection with FIGS. 3-5 .

[0039] The X-ray imaging system 104 transmits the three-dimensional X-ray image volume via the communications network 152 to the navigation system 156, where such X-ray images can be stored and viewed. The navigation system 156 displays the X-ray images obtained by the X-ray imaging system. Once reconstructed for display on the navigation system 156, the X-ray images can be reformatted and repositioned to view the image in any plane and at any slice position or orientation. In some embodiments, the navigation system 156 displays multiple frames or windows on the same screen showing alternative positions or orientations of the X-ray image slices.

[0040] A skilled person will appreciate that the x-ray image volume acquired by the x-ray imaging system 104 can be transmitted to the navigation system 156 at any time, and does not necessarily have to be transmitted immediately after the x-ray image volume is acquired, but instead can be transmitted at the request of the navigation system 156. In an alternative example, the x-ray image volume is transmitted to the navigation system 156 by a flash drive, CD-ROM, diskette, or other such transportable media device.

[0041] The ultrasound imaging system 106 typically uses an ultrasound probe to obtain ultrasound images of the patient's tissue to image portions of the patient's tissue within the field of view of the ultrasound probe. For example, the ultrasound imaging system 106 may be used to image a breast. The ultrasound imaging system 106 acquires and displays ultrasound images of the patient's anatomical structures within the field of view of the ultrasound probe, typically displaying the images in real time as the patient is being imaged. In some examples, the ultrasound images may also be stored on a storage medium such as a hard drive, CD-ROM, flash drive, or diskette for reconstruction or playback at a later time. Additional details regarding ultrasound imaging systems are described with reference to Figures 6-7.

[0042] In some embodiments, the navigation system 156 can access the ultrasound images, and in such embodiments, the ultrasound imaging system 106 can be further connected to a communications network 152, and copies of the ultrasound images acquired by the ultrasound imaging system 106 can be transmitted to the navigation system 156 via the communications network 152. In other embodiments, the navigation system 156 can remotely access and copy the ultrasound images via the communications network 152. In alternative examples, copies of the ultrasound images can be stored on the data store 112 or the EMR system 158, which are in communication with the navigation system 156 via the communications network 152, and can be remotely accessed by the navigation system 156.

[0043] The tracking system 154 is in communication with the navigation system 156 via the communication network 152 and can track the physical location where the ultrasound imaging system 106 is imaging the patient's tissue. In some examples, the tracking system 154 can be directly connected to the navigation system 156 via a direct or wireless communication link. The tracking system 154 tracks the locations of transmitters connected to the ultrasound imaging system 106 and provides the navigation system 156 with data representing their coordinates in the tracker coordinate space. In some examples, the tracking system 154 can be an optical tracking system consisting of an optical camera and an optical transmitter, although those skilled in the art will understand that any device or system capable of tracking the location of an object in space can be used. For example, those skilled in the art will understand that in some examples, a radio frequency (RF) tracking system consisting of an RF receiver and an RF transmitter can be used.

[0044] The ultrasound imaging system 106 can be configured for use with the navigation system 156 through a calibration process using the tracking system 154. A transmitter connected to the ultrasound probe of the ultrasound imaging system 106 may transmit its position to the tracking system 154 in the tracker coordinate space, and the tracking system 154 provides this information to the navigation system 156. For example, a transmitter may be located on the probe of the ultrasound imaging system 106 such that the tracking system 154 can monitor the position and orientation of the ultrasound probe and provide this information to the navigation system 156 in the tracker coordinate space. The navigation system 156 can use this tracked position to determine the position and orientation of the ultrasound probe relative to the tracked position of the transmitter. In some examples, the navigation system 156 and the tracking system 154 operate to provide real-time guidance to a medical professional H performing ultrasound imaging of a patient P.

[0045] In some embodiments, configuration is performed using a configuration tool. In such examples, the position and orientation of the configuration tool may be additionally tracked by tracking system 154. During configuration, the configuration tool contacts the transducer face of the ultrasound probe of ultrasound imaging system 106, and tracking system 154 transmits information representing the position and orientation of the configuration tool in tracker coordinate space to navigation system 156. Based on the tracked position of a transmitter connected to the ultrasound probe, navigation system 156 may determine a configuration matrix that can be used to determine the position and orientation of the ultrasound probe's field of view in tracker coordinate space. In an alternative example, a database having configuration data for multiple brands or models of various ultrasound probes may be used to preload field of view configurations into navigation system 156 during configuration.

[0046] When the ultrasound imaging system 106 is configured with the navigation system 156, the patient's tissue can be imaged with the ultrasound imaging system 106. During ultrasound imaging, the tracking system 154 monitors the position and orientation of the ultrasound probe of the ultrasound imaging system 106 and provides this information in tracker coordinate space to the navigation system 156. Because the ultrasound imaging system 106 is configured for use with the navigation system 156, the navigation system 156 is able to determine the position and orientation of the field of view of the ultrasound probe of the ultrasound imaging system 106.

[0047] The navigation system 156 can be configured to co-register the ultrasound image with the X-ray image. In some examples, the navigation system 156 can be configured to transform the position and orientation of the ultrasound probe's field of view from the tracker coordinate space to a position and orientation in the X-ray image, e.g., X-ray system coordinates. This can be achieved by tracking the position and orientation of the ultrasound probe, transmitting this position information in the tracker coordinate space to the navigation system 156, and relating this position information to the X-ray coordinate system. In some examples, the co-registered image is displayed on the GUI 130.

[0048] For example, a user can select an anatomical plane within an x-ray image, and then the user can manipulate the position and orientation of a tracked ultrasound probe to align the ultrasound probe's field of view with the selected anatomical plane. Once alignment is achieved, the associated tracking space coordinates of the ultrasound image can be captured. Registration of the anatomical axes (superior-inferior (SI), left-right (LR), and anterior-posterior (AP)) between the x-ray image and the tracker coordinate space can be determined from the relative rotational difference between the tracked ultrasound field of view direction and the selected anatomical plane, using techniques well known to those skilled in the art.

[0049] This configuration can further include selecting a landmark within the X-ray image, for example, using an interface that allows the user to select an anatomical target. In some examples, the landmark can be an internal tissue landmark such as a vein or artery, while in other examples, the landmark can be a fiducial skin marker or an external landmark such as a nipple. The same landmark selected in the X-ray image can be positioned with the ultrasound probe, and a mechanism can be provided for capturing the coordinates of the target's representation in the tracker coordinate space upon positioning. The relative difference between the target's coordinates in the X-ray image and the target's coordinates in the tracker coordinate space is used to determine the translation parameters required to align the two coordinate spaces. Previously acquired plane orientation information can be combined with the translation parameters to provide a complete 4×4 transformation matrix capable of co-aligning the two coordinate spaces.

[0050] The navigation system 156 can then reformat the displayed x-ray image using a transformation matrix so that the displayed slice of tissue is in the same plane and orientation as the field of view of the ultrasound probe of the ultrasound imaging system 106. The matched ultrasound and x-ray images may then be displayed side-by-side in a single image display frame, or directly overlaid. In some examples, the navigation system 156 can display an additional x-ray image in a separate frame or location on the display screen. For example, the x-ray image can be displayed along with a graphical representation of the field of view of the ultrasound imaging system 106, with the graphical representation of the field of view being sliced ​​through the 3D representation of the x-ray image. In other examples, annotations can additionally be displayed, representing the location of an instrument imaged by the ultrasound imaging system 106, such as a biopsy needle, guidewire, imaging probe, or other similar device.

[0051] In other examples, the ultrasound image displayed by the ultrasound imaging system 106 may be superimposed on a slice of the x-ray image displayed by the navigation system 156 so that the user can simultaneously view both the x-ray image and the ultrasound image superimposed on the same display. In some examples, the navigation system 156 may enhance certain aspects of the superimposed ultrasound or x-ray image to improve the quality of the resulting combined image.

[0052] 1, the computing system 102 operating the lesion matching engine 110 analyzes a set of X-ray and ultrasound images to determine a confidence level that a lesion identified in the ultrasound image is the same lesion identified in the X-ray image. An indicator of the confidence level can be displayed on the computing device to assist a user operating the ultrasound imaging system 106 in determining whether a lesion previously identified in an X-ray image has been found in the corresponding ultrasound image.

[0053] The electronic medical record system 158 stores multiple electronic medical records (EMRs). Each EMR contains a patient's medical and treatment history. Examples of electronic medical record systems 158 include those developed and maintained by Epic Systems Corporation, Cerner Corporation, Allscripts, and Medical Information Technology, Inc. (Meditech).

[0054] FIG. 3 is a schematic diagram of an exemplary X-ray imaging system 104. FIG. 4 is a perspective view of the X-ray imaging system 104. Referring simultaneously to FIGS. 3 and 4, the X-ray imaging system 104 immobilizes a patient's breast 202 for X-ray imaging (either or both mammography and tomosynthesis) via a breast compression immobilizer unit 204 that includes a static breast support platform 206 and a movable compression paddle 208. The breast support platform 206 and the compression paddle 208 each have compression surfaces 210 and 212 that move toward each other to compress and immobilize the breast 202. In known systems, the compression surfaces 210, 212 are exposed for direct contact with the breast 202. The platform 206 also houses an image receptor 216, an optional tilt mechanism 218, and an optional anti-scatter grid. The immobilizer unit 204 is in the path of an imaging beam 220 emanating from an x-ray source 222 such that the beam 220 impinges on the image receptor 216 .

[0055] The immobilizer unit 204 is supported by a first support arm 224, and the x-ray source 222 is supported by a second support arm 226. For mammography, the support arms 224 and 226 can rotate as a unit about an axis 228 between different imaging orientations, such as CC and MLO, so that the system 104 can capture mammogram projection images in each orientation. In operation, the image receptor 216 remains in a predetermined position relative to the platform 206 while images are captured. The immobilizer unit 204 releases the breast 202 for movement of the arms 224, 226 to different imaging orientations. For tomosynthesis, the support arm 224 immobilizes the breast 202 and remains in a fixed position, while at least the second support arm 226 rotates the x-ray source 222 about the axis 228 relative to the immobilizer unit 204 and compressed breast 202. The system 104 captures multiple tomosynthesis projection images of the breast 202 at each angle of the beam 220 relative to the breast 202 .

[0056] Simultaneously and optionally, the image receptor 216 can be tilted relative to the breast support platform 206 in synchronization with the rotation of the second support arm 226. The tilt can be through the same angle as the rotation of the x-ray source 222, or through a different angle selected so that the beam 220 remains at substantially the same position on the image receptor 216 for each of the multiple images. The tilt can be about an axis 230, which can be, but need not be, in the image plane of the image receptor 216. A tilt mechanism 218 coupled to the image receptor 216 can drive the image receptor 216 in a tilting motion.

[0057] For tomosynthesis and / or CT imaging, the breast support platform 206 can be horizontal or oblique to the horizontal, e.g., in a similar orientation to conventional MLO imaging in mammography. The x-ray imaging system 104 can be a mammography system alone, a CT system alone, a tomosynthesis system alone, or a "combo" system capable of performing multiple forms of imaging. One example of such a combo system has been offered by the assignee of the present invention under the trade name Selenia Dimensions. In some instances, initial imaging is performed with magnetic resonance imaging (MRI).

[0058] When the system is in operation, the image receptor 216 generates image information in response to illumination by the imaging beam 220 and supplies it to the image processor 232 for processing and generation of mammograms. A system control and workstation unit 238, including software, controls the operation of the system and interacts with the operator to receive commands and provide information, including processed x-ray images.

[0059] 5 is a diagram illustrating an exemplary x-ray imaging system 104 in breast positioning for a left longitudinal oblique (MLO) imaging orientation. The tube head 258 of the system 104 is oriented so that it is generally parallel to the gantry 256 of the system 104, or otherwise not normal to the flat portion of the support arm 260 on which the breast rests. In this position, the technician can more easily position the breast without having to crawl or crouch under the tube head 258.

[0060] The X-ray imaging system 104 includes a floor mount or base 254 for supporting the X-ray imaging system 104 on a floor. A gantry 256 extends upward from the floor mount 252 and rotatably supports both a tube head 258 and a support arm 260. The tube head 258 and support arm 260 are configured to rotate discretely relative to one another and to be raised and lowered along a gantry face 262 to accommodate patients of different heights. An X-ray source, described elsewhere herein and not shown here, is located within the tube head 258. The support arm 260 includes a support platform 264 that includes an X-ray receptor and other components (not shown) therein. A compression arm 266 extends from the support arm 260 and is configured to linearly raise and lower (relative to the support arm 260) a compression paddle 268 for compressing the patient's breast during the imaging procedure. The tube head 258 and support arm 260 are sometimes referred to together as a C-arm.

[0061] The X-ray imaging system 104 is equipped with numerous interfaces and display screens. These include a foot display screen 270, a gantry interface 272, a support arm interface 274, and a compression arm interface 276. Generally, the various interfaces 272, 274, and 276 may include one or more display screens, including one or more tactile buttons, knobs, switches, and a capacitive touchscreen having a graphic user interface (GUI), to enable user interaction and control with the X-ray imaging system 104. In examples, the interfaces 272, 274, and 276 may include control functionality also available in system controls and workstations, such as the X-ray computing device 116 of FIG. 1. Any individual interface 272, 274, and 276 may include functionality available in the other interfaces 272, 274, and 276, either continuously or selectively, based at least in part on predetermined settings, user preferences, or operational requirements. Generally, and as described below, the foot display screen 270 is primarily a display screen, although a capacitive touch screen may be utilized if necessary or desired.

[0062] In an embodiment, gantry interface 272 may enable functionality such as selection of imaging direction, display of patient information, adjustment of support arm elevation or support arm angle (tilt or rotation), safety features, etc. In an example, support arm interface 274 may enable functionality such as adjustment of support arm elevation or support arm angle (tilt or rotation), adjustment of compression arm elevation, safety features, etc. In an example, compression arm interface 276 may enable functionality such as adjustment of compression arm elevation, safety features, etc. Additionally, one or more displays associated with compression arm interface 276 may display more detailed information such as applied compression arm force, selected imaging direction, patient information, support arm elevation or angle setting, etc. Foot display screen 270 may also display information as displayed by the display of compression arm interface 276, or additional or different information, as needed or desired for a particular application.

[0063] 6 is a diagram illustrating an example of an ultrasound imaging system 106. The ultrasound imaging system 106 includes an ultrasound probe 302 that includes an ultrasound transducer 304. The ultrasound transducer 304 is configured to emit an array of ultrasound waves 306. The ultrasound transducer 304 converts electrical signals into ultrasound waves 306. The ultrasound transducer 304 may also be configured to detect ultrasound waves, such as ultrasound waves reflected from an internal portion of a patient, such as a lesion within a breast. In some examples, the ultrasound transducer 304 may incorporate capacitive and / or piezoelectric transducers, as well as other suitable transduction technologies.

[0064] The ultrasound transducer 304 is also operatively connected (e.g., wired or wirelessly) to a display 310. The display 310 may be part of a computing system, such as the ultrasound computing device 118 of Figure 2, which includes a processor and memory configured to generate and analyze ultrasound images. The display 310 is configured to display ultrasound images based on ultrasound imaging of the patient.

[0065] The ultrasound imaging performed by the ultrasound imaging system 106 is primarily B-mode imaging, which provides a two-dimensional ultrasound image of a cross-section of a portion of the patient's interior. The brightness of the pixels in the resulting image generally corresponds to the amplitude or intensity of the reflected ultrasound waves.

[0066] Other ultrasound imaging modes may also be utilized, for example, the ultrasound probe may operate in a 3D ultrasound mode in which ultrasound image data is acquired from multiple angles relative to the breast to construct a 3D model of the breast.

[0067] In some instances, the ultrasound image may not be displayed during the acquisition process. Rather, the ultrasound data is acquired and a 3D model of the breast is generated without the B-mode image being displayed.

[0068] The ultrasonic probe 302 may also include a probe localization transceiver 308. The probe localization transceiver 308 is a transceiver that emits signals that provide localization information for the ultrasonic probe 302. The probe localization transceiver 308 may include a radio frequency identification (RFID) chip or device for transmitting and receiving information, as well as an accelerometer, a gyroscope device, or other sensors that can provide directional information. For example, the signals emitted by the probe localization transceiver 308 may be processed to determine the orientation or position of the ultrasonic probe 302. The orientation and position of the ultrasonic probe 302 may be determined or provided in three-dimensional components, such as Cartesian or spherical coordinates. The orientation and position of the ultrasonic probe 302 may also be determined or provided relative to other items, such as a cutting instrument, a marker, a magnetic direction, or a normal to gravity. The orientation and position of the ultrasonic probe 302 may generate and provide additional information to a surgeon to assist the surgeon in guiding the surgeon to a lesion within a patient, as described further below. Although the term transceiver is used herein, this term is intended to cover both transmitters, receivers, and transceivers, along with any combination thereof.

[0069] FIG. 7 illustrates an example of an ultrasound imaging system 106 in use with a patient's breast 312. An ultrasound probe 302 is in contact with a portion of the breast 312. In the position depicted in FIG. 7, the ultrasound probe 302 is being used to image a lesion 314 in the breast 312. To image the lesion 314, an ultrasound transducer 304 emits an array of ultrasound waves 306 into the interior of the breast 312. Some of the ultrasound waves 306 are reflected from internal components of the breast, such as the lesion 314, if the lesion is within the field of view, and return to the ultrasound probe 302 as reflected ultrasound waves 316. The reflected ultrasound waves 316 may be detected by the ultrasound transducer 304. For example, the ultrasound transducer 304 may receive the reflected ultrasound waves 316 and convert the reflected ultrasound waves 316 into electrical signals that can be processed and analyzed to generate ultrasound image data on a display 310.

[0070] The depth of a lesion 314 or the like in the imaging plane can be determined from the time between the emission of a pulse of ultrasound 306 from the ultrasound probe 302 and the detection of the reflected ultrasound 316 by the ultrasound probe 302. For example, the speed of sound is known, and the effect of the speed of sound due to soft tissue can also be determined. Therefore, the depth of an object within an ultrasound image can be determined based on the time of flight of the ultrasound 306 (more specifically, half the time of flight). Other corrections or methods for determining object depth, such as compensating for refraction and shifts in speed of waves through tissue, may also be implemented. Those skilled in the art will appreciate further details of depth measurement in medical ultrasound imaging techniques. Such depth measurement and determination may be used to construct a 3D model of the breast 312.

[0071] Additionally, multiple frequencies or modes of ultrasound technology may be utilized. For example, real-time and simultaneous transmit / receive multiplexing of localization frequencies as well as imaging and capture frequencies may be implemented. Utilizing these capabilities provides information for co-registration or fusion of multiple data sets from ultrasound technology to enable visualization of lesions and other medical images on the display 310. Imaging frequencies and capture sequences may include B-mode imaging (with or without compounding), Doppler modes (e.g., color, duplex), harmonic modes, shear wave and other elastography modes, and contrast-enhanced ultrasound, among other imaging modes and technologies.

[0072] 8 is a block diagram illustrating an example of the physical components of computing device 400. Computing device 400 may be any computing device utilized in combination with lesion identification system 100 or system for managing image data 150, such as computing system 102, x-ray computing device 116, and ultrasound computing device 118.

[0073] 8, computing device 400 includes at least one central processing unit ("CPU") 402, a system memory 408, and a system bus 422 that couples system memory 408 to CPU 402. System memory 408 includes random access memory ("RAM") 410 and read-only memory ("ROM") 412. A basic input / output system, containing the basic routines that help to transfer information between elements within computing device 400, such as during start-up, is stored in ROM 412. Computing system 400 also includes mass storage device 414. Mass storage device 414 may store software instructions and data.

[0074] The mass storage device 414 is connected to the CPU 402 via a mass storage controller (not shown) connected to the system bus 422. The mass storage device 414 and its associated computer-readable storage media provide non-volatile, non-transitory data storage for the computing device 400. While the descriptions of computer-readable storage media contained herein refer to mass storage devices such as hard disks or solid-state disks, those skilled in the art will understand that a computer-readable data storage medium can include any available tangible, physical device or article of manufacture from which the CPU 402 can read data and / or instructions. In certain examples, the computer-readable storage medium includes entirely non-transitory media.

[0075] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable software instructions, data structures, program modules, or other data. Exemplary types of computer-readable data storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, digital versatile disks (“DVDs”), other optical storage media, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computing device 400.

[0076] According to some examples, computing device 400 may operate in a networked environment using logical connections to remote network devices via network 152, such as a wireless network, the Internet, or other types of networks. Computing device 400 may connect to network 152 via a network interface unit 404 connected to system bus 422. It should be understood that network interface unit 404 may also be utilized to connect to other types of networks and remote computing systems. Computing device 400 also includes an input / output controller 406 for receiving and processing input from a number of other devices, including a touch-sensitive user interface display screen or another type of input device. Similarly, input / output controller 406 may provide output to a touch-sensitive user interface display screen or other type of output device.

[0077] As briefly mentioned above, the mass storage device 414 and RAM 410 of the computing device 400 may store software instructions and data. The software instructions include an operating system 418 suitable for controlling the operation of the computing device 400. The mass storage device 414 and / or RAM 410 also store software instructions that, when executed by the CPU 402, cause the computing device 400 to provide the functionality discussed herein.

[0078] 9, an exemplary method 500 for locating a region of interest within a breast is described. In some examples, the systems and devices described in FIGS. 1-8 can be used to perform the method 500. In particular, the computing system 102 of FIGS. 1-2 operates to perform the steps of the method 500 to assist a medical professional in locating a region of interest within a breast during an imaging procedure.

[0079] At operation 502, a first image obtained by an X-ray imaging modality is received. In some examples, the X-ray imaging device 114 of the X-ray imaging system 104 of FIGS. 1-2 operates to record the X-ray image as a result of input provided by medical personnel H at the X-ray computing device 116. In some embodiments, the X-ray image is acquired using digital breast tomosynthesis. In some examples, the X-ray image may be acquired from a remote data store. In such examples, the X-ray image may be recorded at a different time and location and then stored in an EMR or other data store. In some examples, the first image is received at the computing system 102.

[0080] In operation 504, an indication of a target lesion on the X-ray image is received. In some examples, the indication is received from medical personnel H at the X-ray computing device 116. The computing device 116 may be operative to display a user interface that allows the medical personnel H to easily interact with the X-ray image to highlight the target lesion via input provided at an input device in communication with the X-ray computing device 116, such as a mouse, touch screen, or stylus. In some examples, the target lesion may be indicated with a visual marker. The target lesion may be identified by the medical personnel H as requiring additional analysis. In some examples, the target lesion may be identified later by a clinician after the X-ray image is taken. In some examples, the target lesion may be identified in real time as the X-ray image is being recorded using an artificial intelligence system.

[0081] In operation 506, the location coordinates of the target lesion are recorded. The coordinates of the target lesion are recorded during the X-ray imaging process using the X-ray imaging system 104. In some embodiments, the coordinates may be Cartesian or polar. In some instances, the region of interest is identified within a particular slice (z coordinate) within the tomosynthesis image stack, and its location may be further identified by x and y coordinates within that image slice.

[0082] At operation 508, a second image of the breast tissue is obtained by ultrasound imaging. The ultrasound image includes an area of ​​breast tissue corresponding to the location coordinates of the target lesion. In some embodiments, the ultrasound imaging device 120 of the ultrasound imaging system 106 of FIGS. 1-2 operates to record the ultrasound image as a result of input provided by medical personnel H at the ultrasound computing device 118. In some examples, the ultrasound image may be obtained from a remote data store. In such examples, the ultrasound image may be recorded at a different time and location and then stored in an EMR or other data store. In some examples, the second image is received at the computing system 102 for processing along with the first image.

[0083] In operation 510, a potential lesion is identified within a region of breast tissue corresponding to the location coordinates of the target lesion. In some examples, the region is identified based on coordinates converted for ultrasound from coordinates stored for the target lesion during x-ray imaging. In some examples, the location coordinates include at least two of a clock position relative to the nipple, a depth from the surface of the breast, and a distance from the nipple. In some examples, the potential lesion may be highlighted by medical practitioner H on the ultrasound computing device 118. The computing device 118 may be operative to display a user interface that allows medical practitioner H to easily interact with the ultrasound image to highlight the potential lesion via input provided by an input device in communication with the ultrasound computing device 118, such as a mouse, touchscreen, or stylus. In some embodiments, the GUI 130 displays the DBT image and the ultrasound image side-by-side. An example of this GUI 130 is shown in FIG. 10 . Medical practitioner H identifies the potential lesion as possibly being the same as the target lesion identified in the x-ray image. In some examples, a real-time artificial intelligence system may analyze the recorded DBT images to identify the potential lesion. An example of such a system is described in a co-pending application entitled "Real-time AI for Physical Biopsy Marker Detection" (Insert Information for Docket No. 04576.0110USP1), which is incorporated herein by reference in its entirety.

[0084] In operation 512, the potential lesion is analyzed using artificial intelligence to determine a level of confidence that the potential lesion in the second image corresponds to the target lesion in the first image. In some examples, the lesion matching engine 110 operates to analyze the potential lesion and the target lesion to determine whether the two lesions match. As described above, the machine learning lesion classifier analyzes various aspects of the lesion, such as size, shape, and texture, to match the ultrasound and x-ray images of the lesion. In some examples, consistency and density may also be compared to determine a match.

[0085] At operation 514, an indicator of the level of confidence is output. In some examples, the indicator of the level of confidence is generated on the GUI 130. In some embodiments, the GUI 130 includes the ultrasound image and the X-ray image along with an indicator of the level of confidence. In some examples, the indicator may be displayed as text, graphics, color, or a symbol. More detailed information regarding an example GUI 130 is provided in FIG. 10.

[0086] Figure 10 shows one implementation of the GUI 130 of Figure 1. In some examples, the GUI 130 is displayed on a computing device, such as the ultrasound computing device 118 of Figure 1. In the example of Figure 10, the GUI 130 displays a side-by-side x-ray image 602 and ultrasound image 604 of the breast 202. A target lesion 606, previously identified during x-ray imaging, is indicated by a visual marker in the x-ray image 602. The corresponding ultrasound image 604 of the breast 202 shows signs of a potential lesion 608. A confidence level indicator 610 is displayed, providing a percentage of the likelihood that the target lesion 606 and the potential lesion 608 match. In this example, there is a 99.9% match.

[0087] In some examples, other indicators of confidence level may be provided, such as a colored circle around the potential lesion 608. Different colors may represent different levels of confidence. For example, a high level of confidence may be indicated by a green circle. A medium level of confidence may be indicated by a yellow circle. A low level of confidence may be indicated by a red circle. In some examples, both a visual indicator on the ultrasound image 604 and a textual confidence level indicator 610 may be used.

[0088] The GUI 130 also includes a diagram 612 showing the location on the breast 202 where the ultrasound image 604 was taken, and an arrow indicating the location of the target lesion 606. Additionally, coordinates 614 are displayed. In this example, coordinates 614 indicate the location of the potential lesion at the 11 o'clock position on the right breast, 2 cm from the nipple. Diagram 612 shows a corresponding visual representation of the potential lesion 608.

[0089] Figure 11 shows another embodiment of a lesion identification system 700. A lesion correlator 701 operates on the computing device 102, with functionality similar to the lesion matching engine 110 of Figure 1. However, in this example, lesions are correlated using probability mapping. Real-time ultrasound image guidance is provided through the use of at least two optical cameras 702 and a projector 402. The optical cameras 702 operate to capture multiple images of the patient's torso. The multiple stereotactic optical images are analyzed in combination with previously acquired x-ray images on the computing device 102.

[0090] In this example, the AI ​​image analyzer 724 is configured to match regions of the breast between two different imaging modalities. In some examples, a deep learning model is utilized to generate a probability that a target lesion identified in one type of image is located at any given location on another type of image. For example, a target lesion 712 shown in the tomosynthesis view 710 is analyzed to determine the probability of its location on the ultrasound image 708.

[0091] In some examples, the probability mapper 728 generates a probability map of the breast that indicates where the potential lesion 714 is most likely to be located in different types of images. In some examples, this may be an optical image obtained by the optical camera 702. The potential lesion 714 is indicated on the ultrasound image with a color gradient, with the center representing where the target lesion is most likely to be located. In some embodiments, the probability map is a visual map that is overlaid on the ultrasound image or tomosynthesis image, as shown in GUI 730 of FIG. 11 . In other embodiments, the probability map is a visual map that is projected onto the patient's actual breast during an ultrasound examination using the projector 704. The potential lesion 714 is indicated by a colored region in the probability map. This visual probability map is used to guide the medical professional H in obtaining ultrasound images using the ultrasound probe 302.

[0092] In some embodiments, the tracking system 154 and navigation system 156 work in conjunction with the lesion correlator 701 to guide the medical personnel H during the ultrasound imaging session. The current position of the ultrasound probe 302 is communicated to the computing device 102, and the current position of the probe is visually indicated on the image presented on the GUI 730 in real time.

[0093] 12, an exemplary method 800 for locating a region of interest within a breast is described. In some examples, the system of FIG.

[0094] A series of stereo optical images of at least one breast is captured in operation 802. This is typically performed while the patient is lying on an imaging table or other support. The images are captured by two or more optical cameras 702 positioned above the patient.

[0095] At least one tomosynthesis image of the breast is accessed in operation 804. In some embodiments, the tomosynthesis image is accessed at computing device 102 in response to receiving input from a user. In some examples, the tomosynthesis image is accessed from an electronic medical record associated with the patient being imaged. In some embodiments, the tomosynthesis image is then presented on a display of computing device 102.

[0096] In operation 806, an indication of a target lesion on the tomosynthesis image is received. In some examples, the indication is received from medical personnel H at the X-ray computing device of Figure 1. An example of an indication 712 of a target lesion is shown in GUI 130 of Figure 11. As discussed above with respect to Figure 9, there are other ways in which the target lesion may be indicated.

[0097] In operation 808, a co-registration image analysis of the optical and tomosynthesis images is performed. In some embodiments, an artificial intelligence algorithm for region matching is used to generate a virtual deformable of the breast to which both the optical and tomosynthesis images can be co-registered. In some examples, the artificial intelligence algorithm is a deep learning-based region matching method.

[0098] A probability mapping is created based on the image analysis in operation 810. The probability mapping indicates the likelihood that an indicated lesion is located at a particular point on the breast.

[0099] In the example shown in FIG. 11, the visual probability map uses color indicators to indicate higher or lower probabilities at various locations on the breast. For example, red can indicate the highest probability and blue the lowest. In other examples, a grayscale is used, with black indicating the highest probability and white indicating the lowest probability. As seen in FIG. 11, the resulting visual of the probability map likely includes a region of highest probability, indicating where the lesion is most likely to be. This region is surrounded by an area of ​​decreasing probability of spreading outward. For example, the region may be red, and its surrounding colors may extend from orange to yellow to green to blue. In other examples, the visual probability map is displayed as different types of hashing or shading. In some examples, the probability map provides different numerical values ​​for various probabilities. In some examples, a single target is projected onto the point of highest probability.

[0100] In operation 812, the probability map is projected onto the patient P. In some instances, the map is projected onto only one breast. In some embodiments, the map is projected onto both breasts of the patient, thereby providing the medical professional H performing the ultrasound imaging with a visual guide to the most likely location of the target lesion.

[0101] In some embodiments, additional feedback can be provided to the medical practitioner H to indicate that the ultrasound probe 302 is approaching the location of the target lesion. In some embodiments, the ultrasound probe 302 blocks the path of the projection of the probability map onto the patient, casting a shadow. To compensate for this interference with visual guidance, feedback such as tactile or audio feedback can be used to help the medical practitioner H determine when the ultrasound probe 302 is aligned with the target lesion.

[0102] In some examples, additional guidance is provided to the medical practitioner in the form of real-time navigation aids. The real-time position of the ultrasound probe is tracked during imaging, and position information is provided on a display for the medical practitioner. In some examples, the display shows an indication of the current position and orientation of the ultrasound probe relative to the image of the patient's breast.

[0103] The methods and systems described herein provide navigation and lesion-matching technology to help medical professionals quickly and accurately locate mammographic lesions under ultrasound. The system allows medical professionals to identify a region of interest during a mammogram. During a subsequent ultrasound examination, the mammogram, region of interest, and B-mode image are simultaneously displayed. This guides the professional to the region of interest while automating the documentation of probe position, orientation, and annotation. Once the professional navigates to the region of interest, the system automatically analyzes the image, matches the lesion, and provides a visual confidence indicator.

[0104] The systems and methods provided herein enable medical professionals to navigate to within 1 cm of a target lesion using ultrasound. The artificial intelligence-based system is built on thousands of confirmed cases. Lesions can be matched with greater accuracy than medical professionals could achieve on their own. Furthermore, lesions that would otherwise be identified using x-ray images can be located faster and more easily using ultrasound.

[0105] While various embodiments and examples have been described herein, those skilled in the art will recognize that many variations thereon may be made within the scope of the present disclosure, and therefore, the examples provided are not intended to limit the scope of the present disclosure in any way.

Claims

1. 1. A method for locating a lesion within a breast, comprising: receiving, at a computing system, an indication of a location of a target lesion within the breast on a first image of the breast obtained by a first imaging modality; receiving, at the computing system, a second image of the breast obtained by a second imaging modality; analyzing a first image including the target lesion and a second image including the potential lesion using a lesion matching engine operating on a computing system and correlating the first image with the second image using artificial intelligence; determining the probability that the potential lesion corresponds to the target lesion; outputting an indicator of the probability for display on a graphical user interface; A method comprising:

2. Furthermore, recording the location coordinates of a target lesion shown in the first image; recording the location coordinates of the potential lesion; Including, analyzing the first image and the second image includes comparing location coordinates of the target lesion and location coordinates of the potential lesion; The method of claim 1.

3. the location coordinates of the potential lesion include a clock position relative to the nipple of the breast, a depth from the surface of the breast, and a distance from the nipple; The method of claim 2.

4. the target lesion and the potential lesion are indicated by receiving a selection on an image of the breast presented on a display of the computing system. The method of claim 1.

5. the first imaging modality is digital breast tomosynthesis; The method of claim 1.

6. the first imaging modality is magnetic resonance imaging (MRI); The method of claim 1.

7. the second imaging modality is ultrasound; The method of claim 1.

8. analyzing the first and second images using an artificial intelligence system trained with a library of digital breast tomosynthesis cases and corresponding diagnostic ultrasound cases; 2. The method of claim 1.

9. analyzing the first image and the second image is performed using an artificial intelligence system configured to co-register the images using region matching; The method of claim 1.

10. analyzing the potential lesion includes comparing morphological parameters of breast tissue surrounding the target lesion and the potential lesion; The method of claim 1.

11. the indicator includes displaying at least one of a shape, a color, a number, and a reference on the received image; The method of claim 1.

12. the indicator comprises a probability map consisting of a visual indication of the probability that each location on the breast corresponds to the target lesion; The method of claim 1.

13. 1. A lesion identification system, comprising: a processing device; a memory storing instructions that, when executed by the processing device, facilitate the performance of operations; Including, The operation is accessing an x-ray image of the breast, the x-ray image including an identified lesion indicated with a visual marker; receiving an ultrasound image of the breast, the ultrasound image including an indication of a potential lesion; analyzing the potential lesions and the identified lesions using an artificial intelligence lesion classifier; generating a confidence score indicating the likelihood that a potential lesion in the ultrasound image matches an identified lesion in the x-ray image; Displaying the output related to the confidence score in a graphical user interface; Including, the system.

14. Furthermore, The operation is accessing a first set of coordinates identifying the location of the identified lesion while the breast is compressed; Transforming the first set of coordinates into a second set of coordinates that identify a predicted location of the identified lesion while the breast is uncompressed; identifying a region of interest in the ultrasound image corresponding to a second set of coordinates; Identifying potential lesions in ultrasound images The system of claim 13, comprising:

15. Furthermore, The operation is and navigating to said second set of coordinates by receiving a transmission from the ultrasound probe indicative of said location. The system of claim 14 .

16. Furthermore, The operation is Analyzing morphological factors of breast tissue surrounding identified lesions and potential lesions to generate a confidence score. The system of claim 14 .

17. the potential lesions are analyzed using an artificial intelligence lesion classifier; The artificial intelligence lesion classifier is trained on digital breast tomosynthesis images and corresponding ultrasound images. The system of claim 13.

18. The artificial intelligence lesion classifier analyzes lesions for correlation with one or more of density, consistency, shape, margin, orientation, texture, pattern, size, and depth within the breast; 17. The system of claim 16.

19. The ultrasound image is received from the ultrasound system, the confidence score is communicated to a display in communication with the ultrasound system, and a visual indicia of the confidence score is displayed on the display in communication with the ultrasound system as a visual indicator on the ultrasound image. The system of claim 13.

20. A non-transitory machine-readable storage medium containing executable instructions that, when executed by a processor, facilitate the performance of operations, The operation is acquiring target lesion data from a data store, the data being acquired by x-ray imaging and including at least an image of the target lesion and coordinates of the location of the target lesion within the breast; recording an image of the breast obtained by ultrasound imaging; identifying a general region of interest in the recorded image of the breast obtained by ultrasound imaging based on the coordinates of the target lesion; identifying potential lesions in the outlined region of interest; analyzing the potential lesions using an artificial intelligence lesion classifier and comparing the potential lesions to the target lesions to determine a level of confidence that the potential lesions correspond to the target lesions; Outputting an indicator of the level of confidence in a graphical user interface; 1. A non-transitory machine-readable storage medium, comprising:

21. Furthermore, The operation is recording an image of breast tissue using digital breast tomosynthesis; receiving imaging indications of a target lesion within the breast; determining coordinates of the location of the target lesion; Save the target lesion signs and coordinates in a data store.

21. The non-transitory machine-readable storage medium of claim 20, comprising:

22. 1. A lesion identification system, comprising: at least one optical camera; A projector and a processing device; a memory storing instructions that, when executed by the processing device, facilitate the performance of operations; Including, The operation is capturing at least one optical image of the patient's breast with at least one of the optical cameras; accessing at least one tomosynthesis image of the breast; receiving an indication of a target lesion in at least one of the tomosynthesis images; co-registering at least one of the optical images with at least one of the tomosynthesis images by analyzing with artificial intelligence algorithms for region matching and non-rigid deformation models; generating a probability map based on the co-registration and the indication of the target lesion, the probability map indicating the likelihood that the target lesion is located at each of a plurality of points on the breast; projecting the probability map onto the breast with a projector; Including, the system.

Images