Methods and systems for early detection and localization of lesions

Abstract

The present disclosure provides various methods and systems for identifying future occurring lesions in a patient. In one example, a method includes training a model to predict a location of a future lesion based on images collected prior to any lesion occurring (305, 307, 705, 707) and images collected during a detected confirmed lesion (321, 717). The predicted location can be displayed in a probability map (335, 729) to enable early detection of a lesion.

Description

TECHNICAL FIELD

[0001] Embodiments of the subject matter disclosed herein relate to radiological imaging, and more specifically to detecting and localizing lesions in images prior to visual recognition by a radiologist. BACKGROUND

[0002] In the field of oncology, radiological imaging (e.g., X-ray, ultrasound, MRI, etc.) is used to identify regions of lesions or abnormal tissue, which can include tumors. Lesions can be benign (e.g., non-cancerous) or malignant (e.g., cancerous). Typically, lesion discovery on radiological images becomes visible after a lesion has been growing in a patient for years. In the case of radiological imaging and cancer screening, this can result in an interval lesion or cancer, in which case the cancer is detected / presented within a period of time (such as twelve months) after imaging, in which case the discovery is considered normal. In some examples, more frequent patient examinations and / or monitoring of the patient can enable earlier detection of a lesion. Typically, a late-stage lesion is discovered after it has begun growing in the patient, and can be treated using more intense treatments, which can be harmful to the patient. Additionally, the probability of cancer cell metastasis increases with the size of the lesion, and a lesion with cancer cell metastasis can be treated using more intense treatments.

[0003] Accordingly, there is a need for a method to aid in the detection of a lesion or possible lesion prior to visual detection by a physician or radiologist in a radiological image, which is the conventional practice. Analysis of patient images and data to determine suspicious regions in the images that exhibit signatures of clinical findings that have been identified through a learning process as precursors to a lesion, can enable the identification of a possible lesion in a patient prior to the lesion being visible on a radiological image. Additionally, the lesion can be localized prior to its appearance, which can allow a clinician to develop personalized monitoring and care for the patient. SUMMARY

[0004] In one embodiment, a method includes detecting and localizing a possible lesion in a radiological image prior to visual recognition by a radiologist. For example, a machine learning classifier or deep neural network can be trained to build a model based on inputs such as image and data sets taken when a lesion is not identified in a patient, and image and data sets taken at a later time with a clinical finding corresponding to a proven lesion in the same patient detected by a radiologist. The model can be applied to new patient image and data sets to predict and localize a lesion, outputting a map indicating the probability of a lesion appearing in the image in the future that is not visible on the current image. In this way, a lesion can be detected and localized earlier than the conventional method of visual recognition by a radiologist, which can allow the patient to be monitored or treated in the early stages of the lesion growth.

[0005] It is to be understood that the above brief description is provided to introduce in simplified form a selection of concepts that are further described in the detailed description below. This brief description does not identify key or essential features of the claimed subject matter, the scope of which is defined exclusively by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in the above background or any part of this disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0006] The present disclosure will be better understood when reading the following description of non-limiting embodiments, in reference to the appended drawings in which:

[0007] Figure 1 is a block diagram of an X-ray imaging system according to embodiments of the present disclosure.

[0008] Figure 2 is shown a flowchart of an exemplary method for detecting and localizing lesions.

[0009] Figures 3A-3B is shown an exemplary flowchart for training and implementing a machine learning classifier in conjunction with the method of Figure 2

[0010] is shown an exemplary comparison of registered and unregistered images in conjunction with the method of Figure 4 Figures 3A-3B is shown an exemplary image stack of registered images in conjunction with the method of

[0011] Figure 5 Figures 3A-3B is shown an exemplary grid deformation of registered images in conjunction with the method of

[0012] Figure 6 is shown an exemplary grid deformation of registered images in conjunction with the method of Figures 3A-3B

[0013] is shown an exemplary flowchart for training and implementing a deep neural network in conjunction with the method of Figures 7A-7B DETAILED DESCRIPTION Figure 2 The following description relates to various embodiments of systems and methods for detecting and localizing possible lesions in radiological images by training a machine learning classifier or deep neural network to build a model based on images and data collected when a lesion is not identified on a patient and when a lesion is identified in the same patient. The model is then implemented in images and data of new patients to predict and localize lesions and output a map indicating the probability of the presence of a lesion not visible on the current image.

[0014] The following description relates to various embodiments of systems and methods for detecting and localizing possible lesions in radiological images by training a machine learning classifier or deep neural network to build a model based on images and data collected when a lesion is not identified on a patient and when a lesion is identified in the same patient. The model is then implemented in images and data of new patients to predict and localize lesions and output a map indicating the probability of the presence of a lesion not visible on the current image.

[0015] ​​​Radiology images are generated by single or multiple imaging modalities including X-ray, ultrasound, MRI, etc. The example embodiments described herein describe systems and methods in the context of X-ray mammography for breast imaging. Figure 1 A block diagram of an example X-ray imaging system is shown. The methods as described in Figure 1 may be implemented at an X-ray imaging system such as the system of Figure 2 to detect and localize lesions in patient images. The method 200 can include training a prediction and localization model (referred to herein as "the model") using a first set of patient data and images input into the system such as the system of Figure 1 to detect and localize lesions during an inference phase of the method. The second set of patient data and images can belong to the same patient or a different patient relative to the patient from which the first set of patient data and images were collected. The first set of data is used to train the model and can statistically represent a particular population, including the patient providing the second set of patient data and images for the inference phase.

[0016] The method 200 can include outputting a map indicating the probability of a lesion occurring in a patient image input during the inference phase. The model of the method 200 can be a machine learning classifier as shown in Figures 3A-3B or a deep learning neural network as shown in Figures 7A-7B When using a machine learning classifier, as described in Figures 3A-3B the images input into the system during the learning phase can be registered as shown in Figures 4-6 to match image characteristics compared to one or more reference images, where the reference images can be one or more of the patient images used to train the model during the learning phase or analyzed by the model during the inference phase. As an example, four images can be acquired during a breast X-ray screening exam: two images of the left (L) breast (e.g., LCC, LMO) and two images of the right (R) breast (RCC, RMLO), where CC is a cranio-caudal view and MLO is a mediolateral oblique view. If it is desired to identify asymmetries between the L and R breasts, the one or more reference images can be one of the R or L images. As another example, the one or more reference images can be an RMLO view acquired at an initial time (to) for determining any changes in the R breast from a series of RMLO images acquired at different times relative to to.

[0017] Turning now to Figure 1This illustration shows a block diagram of an embodiment of an imaging system 10 according to an exemplary embodiment, configured to acquire raw image data and process the image data for display and / or analysis. It will be understood that various embodiments are applicable to many medical imaging systems that implement X-ray tubes, such as X-ray or mammography systems. Other imaging systems, such as computed tomography (CT) systems and digital radiography (RAD) systems that acquire three-dimensional image data of volume, also benefit from this disclosure. The following discussion of the imaging system 10 is merely an example of such an implementation and is not intended to limit it in terms of modality.

[0018] like Figure 1 As shown, the imaging system 10 includes an X-ray tube or source 12 configured to project a beam of X-rays 14 through an object 16. The object 16 may include a human subject. The X-ray source 12 may be a conventional X-ray tube that generates X-rays 14 having an energy spectrum typically ranging from thirty (30) keV to two hundred (200) keV. The X-rays 14 pass through the object 16 and, after being attenuated, strike a detector assembly 18. Each detector module in the detector assembly 18 generates an analog or digital electrical signal that represents the intensity or number of photons striking the X-ray beam as the X-ray beam passes through the object 16, and thus represents the attenuated beam. In one embodiment, the detector assembly 18 is a scintillator-based detector assembly; however, direct conversion or photon counting detectors (e.g., CZT detectors, etc.) are also conceivable and implementable.

[0019] Processor 20 receives signals from detector assembly 18 and generates an image corresponding to the scanned object 16. Computer 22 communicates with processor 20 to enable the operator to use operator console 24 to control scanning parameters and view the generated images. That is, operator console 24 includes some form of operator interface, such as a keyboard, mouse, voice-activated controller, or any other suitable input device that allows the operator to control imaging system 10 and view reconstructed images or other data from computer 22 on display unit 26. Additionally, console 24 allows the operator to store the generated images in storage device 28, which may include hard disk drive, floppy disk, optical disk, etc. The operator can also use console 24 to provide commands and instructions to computer 22 to control source controller 30, which provides power and timing signals to X-ray source 12.

[0020] Figure 2 A flowchart of an exemplary method 200 for predicting the occurrence of clinical findings such as lesions is shown. Method 200 can be implemented in imaging systems such as... Figure 1implemented at an X-ray system and executed by a processor of the imaging system. The method 200 includes inputting patient images and data to generate and / or train a model during a learning phase. The learning phase includes analyzing images and data of a group of patients that were acquired when no lesion was identified in the images by a radiologist and using the analysis output in conjunction with images and data acquired at a later time when a lesion was identified on the patient. The group of patients can represent a target population that can use the model to apply an inference phase. Increasing the number of patients included in the group of patients can improve the accuracy of the model to predict the occurrence of a lesion. The model can be constructed by a machine learning classifier as described in Figures 3A-3B or via a deep neural network generation as described in Figures 7A-7B Both types of processes can be used to predict and locate a possible lesion. The model can be applied to images and data of a patient (e.g., any other patient) during an inference phase of the method. The inference phase includes using the trained model to predict and locate a lesion in images of a patient, which can be newly acquired images with or without previously acquired prior images. In some examples, predicting the occurrence of a lesion in prior images can provide the model more opportunities to establish a robust prediction regardless of whether a lesion is actually present. The method 200 can include generating a map indicating the likelihood of a lesion occurrence and outputting the map at a display device.

[0021] At 202, the method 200 includes collecting input patient images and data. The input patient images and data can include images and data from one or more patients. As described above, the one or more patients can be a group of patients representing a particular population. The input images of the patients can include prior images and data acquired when no lesion was identified on the patient, as well as subsequent images of the same patient from which a radiologist detected a clinical finding, hereinafter referred to as ground truth images. The ground truth images and images acquired when no lesion was identified can be used to train the model. The patient images include radiological images collected from an imaging system, such as the system 10 of Figure 1 The input patient data can include, but is not limited to, patient history covariates (e.g., gender, age, age at menarche, age at first full-term pregnancy, age at menopause, parity, family history, alcohol use, body mass index, hormone use, tamoxifen use, etc.), breast composition covariates (e.g., percent glandular tissue, absolute dense tissue volume, etc.), and genomics and proteomics covariates (e.g., expression of estrogen receptor, expression of progesterone receptor, BRCA1 and BRCA2 mutations, expression of human epidermal growth factor receptor 2 (HER2 / neu), expression of Ki-67 protein, etc.). For digital processing, the data can be quantified if the data is not available in a digital format. Additionally, the patient data can include measurements taken on biological samples of the patient’s body and additional information related to relatives of the patient.

[0022] At 204, the inputted patient images and data are processed during a learning phase of the process to build a model to predict and localize where a lesion can occur in the imaged region of the patient's anatomy. In one example, the model is built using a machine learning classifier further described in Figure 3A or a deep neural network further described in Figure 7A . The model can be generated and / or trained based on prior images, analysis outputs based on prior images, and comparisons of the analysis outputs to subsequent images of the patient, e.g., where one or more clinical findings are identified. Thus, when a new clinical finding is detected, a training data set is generated. The training data set can be used to build or update the machine learning classifier or deep neural network to generate the model, and the model can be applied to patient images and data to predict and localize where a lesion can occur.

[0023] At 206, radiological images of a new patient can be obtained by an imaging system such as the system 10 of Figure 1 and inputted to the model. The new patient can be a different patient than the one or more patients whose images and data were inputted to the learning phase, and the images of the new patient are inputted for inference in parallel with the images and data of 202 that were inputted for learning. Additionally, new data of the patient can be inputted and can include, but is not limited to, patient medical history covariates, breast composition covariates, and genomics and proteomics covariates, as described above at 202.

[0024] At 208, the model is applied to the inputted images during an inference phase of the model to predict and localize where a lesion can occur on the images. Depending on the process by which the model was generated, e.g., by a machine learning classifier Figure 3A or a deep neural network Figure 7A , the lesion is predicted and localized as further described with respect to the machine learning classifier and deep neural network, respectively, with respect to Figure 3B and Figure 7B . For both processes, a map is generated that indicates a probability of a lesion occurring in the image, which can not yet be visible on the image, but can become visible in the future on the patient's image.

[0025] At 210, the method 200 displays the output results, e.g., the generated map showing the probability of a lesion occurring, on a display device such as the display unit 26 of Figure 1 . The method 200 ends.

[0026] By learning the characteristics of the image and patient data that can be precursors to identifying a lesion, the model can utilize one of a machine learning algorithm or a deep neural network to enable prediction and localization of a lesion on a patient image prior to the visual appearance of the lesion on the image. In this way, when the probability of a lesion forming is estimated to be high, a clinician can choose to begin treatment more frequently, or earlier in the growth of a lesion, than would be the case if the lesion were identified through conventional methods (e.g., through visual recognition by a radiologist or other clinician), at which point the lesion can have grown further than if identified through method 200.

[0027] Figures 3A-3B An exemplary flowchart of a first process for training and implementing a machine learning classifier as described in method 200 is shown. The first process includes a learning phase 300, as shown in Figure 3A , in which input images of one or more representative patients are registered, and local features of the input images are extracted, and classified with a machine learning classifier along with quantified input patient data. Further details regarding image registration are described in Figures 4-6 . In addition, images of representative patients with detected lesion localization can be used to train the machine learning classifier, as further described below. The first process also includes an inference phase 302, as shown in Figure 3B , during which a trained machine learning classifier is used to analyze images of a new patient, the trained machine learning classifier inferring a location where a lesion can appear based on extracted local features, quantified data, and measured vectors. The analysis by the machine learning classifier generates a map of possible lesions on the patient image, the map indicating a probability of a lesion appearing in the image, which can not yet be visible on the current image, but can become visible on a later image of the patient.

[0028] Turning first to Figure 3A , the figure shows a learning phase 300 of a machine learning classifier, which can be implemented at 204 of method 200. At 303, input patient images 305 and 307 from previous exams of a representative patient for which no lesion was detected. Image 305 shows multiple views of at least one breast from previous exams at times t-1, t-2, and t-3, where time t represents an exam in which a lesion was detected. Image 307 shows images acquired with different imaging modalities. For example, images can be obtained through ultrasound, MRI, etc. Thus, images input to the model can be multi-modal, and subsequent processing of the images can be modified depending on the modality used. Images 305 and 307 can be registered at 309, which will be further described in Figures 4-6 .

[0029] At 311, feature extraction is performed on the local regions of interest of the patient images 305, 307 based on the image registration. More specifically, the image registration enables the identification of differences between the images, for example, at least one difference image can be computed after the registration to generate a vector of features computed from different regions of interest (ROIs) in the at least one difference image. The features can be extracted from a set of images including the images input at 303 and the at least one difference image. The features can be measurements taken on the regions of interest (ROIs) of the set of images. Additionally, in parallel with the image registration and feature extraction, patient data is input at 313 and quantified at 315. For example, the patient data can include, but is not limited to, patient history covariates, breast composition covariates, and genomics and proteomics covariates, as discussed with respect to Figure 2 the patient data can also include measurements taken on biological samples of the patient’s body and additional information related to the patient’s relatives. For digital processing by the machine learning classifier, the patient data is digitally quantified at 315 if it is not already available in digital format.

[0030] At 300, a vector of features is generated using the quantified patient data at 315 and the features extracted from the set of images at 311. Each feature vector can correspond to features computed from the set of images of a given ROI and the quantified patient data. Each vector can be input to a machine learning classifier. For example, if the number of features is N, the machine learning classifier operates in an N-dimensional space. As an example, if the mean value of the pixel gray scale and the variance of the pixels are measured at each ROI, a two-dimensional vector (e.g., N=2) can be obtained. The ROIs can then be assigned to one of K classes (e.g., K=3), the K classes being defined as, for example, normal, benign, and malignant. The machine learning classifier can be configured to classify the ROIs based on the coordinates of each feature vector. But to do so, the machine learning classifier is first trained with examples. The examples can be cases where the ground truth information is known, for example, images and patient data, where one or more lesions in the images are identified at time t, and the spatial coordinates of the lesions. For each ROI of the examples, a vector of features (e.g., a feature vector) can be computed by comparing the spatial coordinates of the ROI to the ground truth information and assigned to one of the K classes.

[0031] An example of ground truth information presented to the machine learning classifier at 317 is described at 319 of the learning process 300. In this example, the ground truth corresponds to a set of labeled images 321 of a patient taken at time t, where a lesion (as indicated by dots) is detected by a radiologist. Each pixel of the image 321 has a label corresponding to one of K classes according to the nature of the tissue detected by the radiologist (e.g., normal, benign, and malignant), where the lesion is indicated by dots. The labeled images 321 are fed to the machine learning classifier at 317 for comparison with the feature vectors 312 extracted from the ROIs of the first set of images 303 and 307 after registration 309 and the patient data 313 after quantification 315. The paired information between the labels of the first set of images 321 and the feature vectors from the second set of images 303 and 307 and the patient data 313 are used to modify the parameters of the machine learning classifier to improve its accuracy in predicting where a lesion is likely to occur.

[0032] The predictions of the machine learning classifier are illustrated in a feature space map 323. The map 323 depicts an N-dimensional feature space (N=2 in this illustration) divided by a dashed line between K classes (e.g., K=3, where the classes correspond to benign, normal, and malignant tissue). The dashed line is plotted based on the comparison of the ground truth information (or labels) with the feature vectors 331. Each vector can be plotted according to the vector coordinates, and as more and more images and patient data are input to the machine learning classifier, the dashed line is modified to improve the accuracy of the classifier 317, thereby reducing the classification error. As an example, the feature vector 318 measured from the ROIs on the image 303 and the image 307 is shown in the feature space 323 and corresponds to malignant tissue indicated by the labels of the ground truth image 319 at the ROI locations. Machine learning algorithms that can be used to train the classifier as shown in the map 323 include multilayer perceptron (MLP), support vector machine (SVM), random forest, etc.

[0033] Thus, the first process includes learning the classifier parameters from the feature vectors (quantified data and image features) by comparing the images at time t (where the radiologist can detect a lesion on the image) and the images with known ground truth (e.g., one class per image pixel) to the images at time t-1, t-2, etc. (where the radiologist does not detect a lesion on the image). In some examples, after the learning process 300 is complete, the model can be updated via continuous training, which can update the parameters of the machine learning classifier based on subsequent cases where a lesion is detected. These cases can be labeled / reported and sent to the computer and algorithms configured to update the parameters.

[0034] As Figure 3BAs shown, during the inference phase 302, the trained machine learning classifier is applied to different patient images at 323. The patient images can be generated by an imaging system at time t, such as Figure 1 System 10, and can be done with any previously obtained images, such as at t-1, t-2, etc.

[0035] The new patient’s images can be registered at 324, as described in reference to Figures 4-6 in further detail, and feature extraction at the ROIs can be performed at 325, as described above in reference to Figure 3A of learning process 300 at 311. Additionally, patient data is input at 327 and quantified at 329, performed in parallel with the feature extraction, as described at 315 in reference to Figure 3A of learning process 300 at 311. Additionally, patient data is input at 327 and quantified at 329, performed in parallel with the feature extraction, as described at 315 in reference to

[0036] The trained machine learning classifier is applied to the vector of image features and the quantified patient data 331 at 333. For example, the vector can be shown in feature space map 323, where the partitioning between classes is defined by solid lines (e.g., based on the final boundaries of learning process 300). The feature vector extracted for the ROIs of patient image 323 and patient data 327 can be mapped to map 323 to determine to which class the ROIs can be assigned. The result of the classification can be displayed at 335 as a map of possible lesions. The map can include highlighted regions of interest on the image that have been determined by the trained machine learning classifier to have an increased probability of lesion appearance. In other words, the lesions can not yet be visible on the current and previous images of the patient under inference, but can become visible on future images of the patient based on the trained classifier. In one example, regions with a higher probability of lesion appearance can have brighter or stronger highlighted regions compared to regions with a lower probability of lesion appearance. For example, the brightness of the ROIs can be modified, or a color assigned for the ROI, where each of the classes of the machine learning classifier can be represented with a different color. As another example, a marker with a shape representing the assigned class can be drawn at the location of the ROI. As yet another example, annotation text can be added near the bounding box corresponding to the ROI, where the annotation text can indicate the class assigned to the ROI by the machine learning classifier.

[0037] As described above in reference to Figures 2-3BThe model used to predict lesion occurrence can be generated by a machine learning classifier through a learning phase. Images without lesions, images showing one or more detected lesions, and patient information can be used to train the machine learning classifier to accurately predict the location of potential lesions on the patient. The predicted lesion locations can be presented to the operator as a probability map, thus displaying the results in an efficient manner.

[0038] It can be used for feature extraction and Figures 4-6 The image undergoes optional preprocessing as shown. For example, as mentioned above, training and inference may include image registration. Now turn to Figure 4 The diagram illustrates the combination Figure 3A and Figure 3B An exemplary display of registered and unregistered images obtained from the flowchart. Unregistered images can be collected, for example, on different dates (e.g., t vs. t-1) and at different events (e.g., x-ray paths, such as CC views vs. MLO views) and under different imaging modalities (or apparatuses). The use of multimodal images can enhance the differentiation of breast content based on image processing. As described above, images can be collected via different imaging modalities, and therefore image registration can be modified according to the modality. For example, x-ray images can be registered together, ultrasound images can be registered together and separate from x-ray images, x-ray images can be registered with ultrasound images, and so on. For example, a contralateral image can be registered using the registration operator Γ to match a reference image 406 (e.g., image f). t The properties of f (e.g., f) t and f t-1 Images labeled with ) can be images of the right breast, and are labeled with g (e.g., g t and g t-1 The image marked on the opposite side can be an image of the left breast.

[0039] The first set of images 402 can be acquired at time t. In one example, time t is the examination time based on when the lesion is detected by imaging. The second set of images 404 can be acquired at time t-1, where time t-1 can be a time prior to time t. At time t, the reference image 406 and the first contralateral image 408 can be obtained by a radiographic imaging system, such as... Figure 1 The system 10 captures the images. The second set of images 404 includes the first image 410 and the second contralateral image 412. The reference image 406 can be positioned in the desired orientation / view to which the other images will be adjusted by the registration operator Γ. The registration operator Γ can register images to match features, including but not limited to skin lines, shape, intensity, etc.

[0040] For example, in Figure 3A At position 309 of the learning phase 300, the image is modified by the registration operator Γ for registration. Figure 4The top row of images shows unregistered images from each breast at times t and t-1. The bottom row of images shows reference image 406 and modified images at times t and t-1, including a modified first contralateral image 418, a modified first image 420, and a modified second contralateral image 422. Each of the modified images has the same orientation as reference image 406. For example, the first contralateral image 408 can be flipped across a vertical plane to generate the modified first contralateral image 418. Similarly, the second contralateral image 412 can be flipped across a vertical plane to generate the second modified contralateral image 422. (See below for reference...) Figure 6 More details are described in the process of generating a registered image from an unregistered image.

[0041] like Figure 5 As shown, modifying the image to be registered, and capturing images with different orientations and views to produce images with the same orientation and view, allows for direct comparison between images. This direct comparison can help identify features that may be precursors to clinical findings associated with lesions. Models, such as those derived from... Figure 3A The model generated by the machine learning classifier can be trained with features to detect and locate potential lesions in patient images before they become visible to radiologists and may require stringent treatment protocols.

[0042] Figure 5 An exemplary set of registered images 500 is shown. Figure 5 The image shown can be Figure 4 The images are registered to each other and therefore have the same number. The set of registered images 500 includes reference image 406, Figure 4 Modified images, and combinations of images, such as compiled images. For example, the first edited image 504 is a combination of images 406 and 420, the second edited image 506 is a combination of images 406 and 422, and the third edited image 508 is a combination of images 406 and 418. Different combinations of images can reveal features of interest. For example, differences between parent images of the same view of the same breast but taken on different dates can capture temporal evolution; differences between parent images showing the same view of two breasts taken on the same date can capture asymmetry; or differences between parent images showing the same view of two breasts taken on different dates can capture both temporal evolution and asymmetry. Thus, the first edited image 504 shows temporal evolution, the second edited image 506 shows temporal evolution and asymmetry, and the third edited image 508 shows asymmetry. For example, differences specific to the right breast can be depicted with a different color than those specific to the left breast. The set of registered images 500 can be generated as the output of image registration, for example in... Figure 3A At position 309, and is input into feature extraction, for example, at...Figure 3A 311 locations and in Figure 3B 325 locations.

[0043] Figure 6 An example of a warping process performed by the registration operator Γ using mesh deformation is shown. Figure 4 Unregistered images, such as reference image 406, first contralateral image 408, first image 410 and second contralateral image 412, are displayed in the set of unregistered images 602. Figure 6 The warping process can be applied to a set of unregistered images 602 to generate Figure 4 and Figure 5 The registered images are shown in Figure 604. To the right of the set of unregistered images 602, a set of processed images is shown in Figure 604.

[0044] Skin lines 614 are generated based on reference image 406 and applied to each of the first image 410, the first contralateral image 408, and the second contralateral image 412. Skin lines 614 can be used to constrain the deformation of mesh 615, as shown in the set of process diagrams 604, applied to the first image 410, the first contralateral image 408, and the second contralateral image 412. Mesh 615 may include nodes 617 and edges 619 and may cover regions of the imaged breast to enable warping of the image that satisfies constraints of different definitions, such as spatial and intensity constraints. For example, the modified image can be warped such that the skin lines 614 of the modified image can be superimposed on the skin lines of reference image 406.

[0045] As described below, the deformation along the x-axis, the deformation along the y-axis, and the intensity deformation can be estimated by the difference between the grid 615 applied to the image and the reference image 406 at node 617 of the grid 615. For example, a node in node 617 of the grid 615 applied to the first image 410, for example, as shown in FIG. 620, can be compared with the corresponding node at the reference image 406, and the first image 410 can be deformed accordingly by shifting the node at the first image 410 to match the position of the corresponding node at the reference image. During measurement, the deformation of each node in node 617, and the horizontal (e.g., x-axis) and vertical (e.g., y-axis) displacement of each node can be interpolated to each pixel of the image (e.g., 408, 410, 412). For example, bilinear or bicubic spline interpolation strategies can be used. The deformation applied to each pixel of the image can make it possible to combine paired images according to, for example, skin line and intensity matching images, such as Figure 5 As shown.

[0046] The chart 604 of this set of processes shows the measured kinematics of deformation along the x-axis (e.g., x-deformation) at the first column 616 of the chart 604, the deformation along the y-axis (e.g., y-deformation) at the second column 618, and the intensity deformation based on the set of unregistered images 602 at the third column 619. For example, the x-deformation of the first image 410 matching the mesh 615 of the reference image 406 under the constraints of similar skin line shape and location is depicted at the graph 620, the x-deformation of the first contralateral image 408 is depicted at the graph 624, and the x-deformation of the second contralateral image 412 is depicted at the graph 628. Similarly, the y-deformation of the first image 410 is shown at the graph 622, the y-deformation of the first contralateral image 408 is shown at the graph 626, and the y-deformation of the second contralateral image 412 is shown at the graph 630.

[0047] The correction based on intensity deformation can also be applied to the set of unregistered images 602. The changes in intensity can be used to deform the images with the mesh 615 to match the intensity features common to the warped images and the reference image 406. Thereby, abnormal intensity features can be identified. For example, the intensity deformation of the first image 410 is shown at the chart 632, the intensity deformation of the first contralateral image 408 is shown at the chart 634, and the intensity deformation of the second contralateral image 412 is shown at the chart 636. The images after being deformed can be used to generate Figure 5 the compiled images shown.

[0048] As Figure 2 shown, instead of the machine learning classifier, the learning phase at 204 and the inference phase at 208 can include in a second process that relies on training and implementing a deep neural network to generate a model for predicting the occurrence of a lesion. The second process is described in Figures 7A-7B by an exemplary flowchart for generating a model by training a deep neural network. During a learning phase 700 of the second process, as Figure 7A shown, a deep neural network is trained with input images of patients and quantified patient data. The input images include images without a lesion paired with images of the same patient with a lesion detected by a radiologist and are used as ground truth information in the learning phase 700. During an inference phase 702 of the second process, as Figure 7B shown, images of a new patient are analyzed using the trained deep neural network that infers, based on the input images and quantified patient data, a location where a lesion is likely to appear in a future image of the new patient. A map of possible lesions can be output by the trained deep neural network model that indicates a probability of a lesion appearing in the image, which can not yet be visible on the current image but can become visible on a later image of the patient.

[0049] Turning first to Figure 7A, which illustrates a learning phase 700 of the second process, which can be implemented at 204 of the method 200. At 703, patient images 705 and 707 are input from previous exams of patients who did not have a lesion observed. The images 705 show multiple views of at least one breast from previous exams at times t-1, t-2, and t-3, where time t is the exam corresponding to the detection of at least one lesion. The images 707 show images obtained via different modalities, such as ultrasound, MRI, etc. In addition, patient data is input at 709 and quantified at 711. For example, the patient data can include, but is not limited to, patient history covariates, breast composition covariates, and genomics and proteomics covariates, as described in Figure 2 . The patient data can also include measurements taken on biological samples of the patient’s body and additional information related to the patient’s relatives. To be digitally processed by the deep neural network, if the patient data is not already available in a digital format, the patient data can be digitally quantified at 711, as described above with reference to Figure 3A .

[0050] The quantified patient data and the input images 705, 707 are fed to a deep neural network at 713, which can be configured as a detector and classifier. In addition, at 715, a set of images 717 and 718, e.g., ground truth images taken at time t when a lesion is detected (as indicated by the dots), can be paired with the input images 705 and 707. The ground truth images 717 and 718 provide ground truth information to compare against the predictions generated by the deep neural network based on the input images and quantified data. The deep neural network, similar to a machine learning classifier, can partition the data into K classes (e.g., normal, benign, and malignant). The deep neural network can be configured to determine an error (e.g., a loss function) between the predicted partition of classes and the ground truth information, and update its internal parameters based on the error. For example, the weights and biases of each artificial neuron can be modified in response to the computation of the loss function. The training of the deep neural network can continue, e.g., using other examples of images paired with ground truth images, until the loss function is small enough to be acceptable. Thus, the training converges to a solution and can be considered complete, enabling the trained model to be used for inference of new patients.

[0051] As described above for the learning process of the machine learning classifier model, in some examples, the deep neural network model can continue to be trained based on new cases of lesions detected. The new cases can be labeled and / or automatically sent to a computer and algorithm to update the parameters of the deep neural network.

[0052] During the inference phase 702, as Figure 7BAs shown, the trained deep neural network can evaluate patient images at 719, corresponding to images of new patients, for example, patients different from one or more patients whose images and data inputs were collected at 703 and 709. New patient data can be collected at 721 and quantified at 723.

[0053] At position 727, a trained deep neural network is applied to new patient images and quantified data to generate and display a map of potential lesions at position 729. The map can resemble the output of a machine learning classifier, as described above. Figure 3B The description at point 335.

[0054] In some examples, such as Figure 7A As shown, the warping of the input image is as described in the reference above. Figures 4-6 The aforementioned process can be implicitly performed by a deep neural network. However, in other examples, the warping process can be performed before the input image is fed into the deep neural network. Furthermore, in other examples, the deep neural network (and...) Figures 3A-3B A machine learning classifier can be continuously updated and trained by feeding additional ground truth images into the deep neural network or machine learning classifier. The accuracy of the generated model can also improve as the number of ground truth images increases.

[0055] Thus, early identification of lesions is possible based on images acquired before the presence of visible lesions in a patient image dataset. Images can be used to build image-trained models to predict the future occurrence of lesions. These models can rely on machine learning or deep learning and can incorporate additional patient information beyond the images. By extracting features from the images and identifying differences between them, information about temporal evolution and / or asymmetry can be provided. This model can be applied to various imaging techniques and thus can reduce the intensity of treatment.

[0056] As used herein, elements or steps listed in the singular and beginning with the word "a" or "an" should be understood to not exclude a plurality of said elements or steps unless such exclusion is explicitly stated. Furthermore, references to "one embodiment" of the invention are not intended to be construed as excluding the existence of additional embodiments that also include the referenced features. Moreover, unless explicitly stated to the contrary, embodiments that "comprise," "include," or "have" elements or multiple elements having a particular characteristic may include additional such elements that do not have that characteristic. The terms "comprise" and "in..." are used as concise linguistic equivalents to the corresponding terms "comprising" and "wherein". Furthermore, the terms "first," "second," and "third," etc., are used merely as notations and are not intended to impose numerical requirements or a particular order of position on their objects.

[0057] This written description uses examples to disclose the application, including the best mode, and also to enable any person skilled in the art to practice the application, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the application is defined by the claims, and can include other examples that occur to those skilled in the art. Such other examples are intended to fall within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent

Claims

1. A method for identifying the occurrence of a lesion in a patient, comprising: Training a model to predict the location of future lesions, the training includes: Regions of interest (ROIs) in images collected from one or more patients before the presence of any lesions are compared with corresponding ROIs in ground truth images, wherein the ground truth images include images collected from the one or more patients showing the presence of one or more lesions; Feature vectors are calculated based on images collected from the one or more patients and digitally quantified patient data, and an N-dimensional space of features is partitioned according to N features in the feature vectors, wherein the N-dimensional space includes the estimated partitions; and The ROIs are classified according to the partitions; Use a trained model to infer the region of future lesions in images collected from new patients; and The inferred areas are displayed on the probability map to indicate areas where the likelihood of lesion formation is increased.

2. The method of claim 1, further comprising utilizing one of machine learning and deep learning.

3. The method of claim 1, wherein classifying the ROI according to partitions includes identifying the ROI based on features, and wherein the features are measurements of regions that depict differences between images collected before the presence of any lesions and images showing the presence of one or more lesions.

4. The method of claim 1, wherein training the model further comprises collecting patient data and quantifying the patient data digitally.

5. The method of claim 1, wherein calculating the feature vector comprises calculating a vector for each ROI in the ROI.

6. The method of claim 1, wherein training the model further comprises registering images collected from the one or more patients, and wherein registering the images comprises adjusting the orientation of the images to the orientation of a reference image.

7. The method of claim 6, wherein registering the image further comprises determining a skin line based on the reference image, and using the skin line to warp the image collected from the one or more patients.

8. The method of claim 7, wherein warping the image comprises applying a mesh to each image in the image, and shifting nodes of the mesh to deform the image such that the skin of the deformed image is superimposed on the skin line of the reference image.

Images