Salient visual explanations of feature evaluations via machine learning models

Abstract

Various embodiments of the present disclosure are directed to a salient medical imaging controller (80) employing an artificial intelligence engine (40) and a graphical user interface (70). In operation, the artificial intelligence engine (40) includes one or more machine learning models (42) trained to render feature evaluations of medical images. The graphical user interface (70) provides user interaction with the artificial intelligence engine (40) to manipulate salient visualization of the feature evaluations of the medical images by the machine learning model(s) (42).

Description

Technical Field

[0001] Various embodiments described in the present disclosure relate to systems, machines, controllers, and methods that include one or more machine learning models for feature evaluation of medical imaging of anatomical regions or anatomical organs. More specifically, but not exclusively, various embodiments relate to providing salient visual explanations of feature evaluations through (one or more) machine learning models of medical imaging of anatomical regions / organs. Background Art

[0002] Artificial intelligence (AI) technology based on deep learning has shown promising results in medical imaging-based diagnosis (e.g., deep learning diagnosis of ultrasound (US) imaging, X-ray imaging, computed tomography (CT) imaging, magnetic resonance imaging (MRI), positron emission tomography (PET) imaging, single photon emission computed tomography (SPECT) imaging, and diffuse optical tomography (DOT) imaging). Despite the promising results, most such algorithms only output the highest risk and will face challenges in regulation and clinical adoption. To address this problem, in the deep learning community, a significant amount of effort has been invested in the interpretability of deep neural networks for researchers. However, these solutions are optimized for deep learning practitioners rather than clinicians (e.g., clinicians and radiologists), whereby many solutions focus specifically on the visualization of the deep neural network itself, while some other solutions attempt to explore the relationship between parts of medical images and the probability of medical images. Therefore, components constructed for clinicians to interactively explore how deep neural networks work with medical images that exceed risk outputs do not currently exist. Furthermore, the image properties of medical images are different from natural images which have diffuse, indefinite structures and are often 3D+ dimensional.

[0003] More specifically, one problem with the adoption of AI technology based on machine learning models in the medical field is that machine learning models are often considered "black boxes" by clinicians. Although the performance of machine learning models in medical imaging-based diagnosis has been demonstrated, the opaque reasoning process of machine learning models makes it difficult for clinicians to understand how and why the models work. This is especially worrying for clinicians because, in part due to poor interactions between clinicians and AI, computer-assisted diagnosis has not yet produced the expected benefits. For example, because most cases in screening environments are negative, even good AI will typically produce many more false positives than true positives. Therefore, the positive predictive value of information is low. This reduces clinician trust and hinders the clinical adoption of AI that can save lives.

[0004] Although some work has been done to visualize predictions from two-dimensional (2D) convolutional neural networks (2D ConvNets) by presenting 2D localization maps (e.g., heat maps), methods for visualizing predictions / classifications from three-dimensional (3D) convolutional neural networks (3D ConvNets) have proven difficult to develop. This is especially the case when the 3rd dimension is hidden or the most relevant angles and viewpoints are not known in advance. More specifically, 3D ConvNets are often used for computer-aided diagnosis based on volumetric medical images (e.g., 3D US, 3D X-ray, CT, MRI, PET, SPECT, and DOT). Understanding the decision-making process of 3D ConvNets is crucial for well-trained clinicians to confirm and accept the proposed predictions. In addition, the challenges of visualizing 3D localization maps (heat maps) are mainly because such maps from the last or intermediate layers of a 3D ConvNet are: (1) multi-dimensional (4D tensors), (2) low resolution compared to the input volumetric medical image, and (3) difficult to interpret using simple rendering techniques. Summary of the invention

[0005] In light of the foregoing, the various embodiments described in this disclosure present saliency visualization techniques that will allow a feature assessment algorithm based on a machine learning model to 'explain' itself to a clinician.

[0006] One set of embodiments of the present disclosure relates to the "Explanation through salient visualization" aspect of the present disclosure. More specifically, this aspect of the present disclosure provides various embodiments of a graphical user interface that provides user interaction with one or more machine learning models that provide feature evaluation of medical imaging of the body, human, or animal.

[0007] A first embodiment of the group of embodiments is a significant medical imaging controller employing a memory storing an artificial intelligence engine and a graphical user interface, wherein the artificial intelligence engine includes one or more machine learning models. The significant medical imaging controller also employs one or more processors in communication with the memory, wherein the processor(s) are configured to: (1) apply the machine learning model(s) to medical image data representing one or more features of a medical image to plot feature evaluations of the medical image, and (2) control display of a graphical user interface providing user interaction with the artificial intelligence engine to manipulate significant visualization of the feature evaluations of the medical image by the machine learning model(s).

[0008] A second embodiment of the group of embodiments is a non-transitory machine-readable storage medium encoded with instructions for execution by one or more processors. The non-transitory machine-readable storage medium stores instructions for: (1) applying an artificial intelligence engine including one or more machine learning models to medical image data representing one or more features of a medical image to plot feature evaluations of the medical image, and (2) controlling display of a graphical user interface that provides user interaction with the artificial intelligence engine to manipulate a salient visualization of the feature evaluations of the medical image by the machine learning model(s).

[0009] A third embodiment of this group of embodiments is a remarkable medical imaging method comprising: (1) applying an artificial intelligence engine including one or more machine learning models to medical image data representing features of a medical image to draw feature evaluations of the medical image, and (2) controlling the display of a graphical user interface that provides user interaction with the artificial intelligence engine to manipulate remarkable visualization of the feature evaluations of the medical image through the (one or more) machine learning models.

[0010] For the purposes of describing and claiming the present disclosure:

[0011] (1) Terms in the field of the present disclosure, including but not limited to: "medical imaging", "artificial intelligence engine", "data preprocessor", "machine learning model", "neural network", "support vector machine", "salient image manager", "medical image display engine", "image viewer", "medical image", "medical image generator", "salient image generator", "image slice", "image segmentation" and "graphical user interface", are to be interpreted broadly as known and appreciated in the field of the present disclosure and exemplarily described in the present disclosure;

[0012] (2) the term “feature” broadly encompasses any type of object within a medical image that is identifiable and / or characterizable by one or more trained machine learning models, including but not limited to: anatomical objects (e.g., vessels, organs, etc.), foreign objects (e.g., procedural tools / instruments and implanted devices), and image artifacts (e.g., noise, grating lobes);

[0013] (3) The term "feature evaluation" broadly covers any type of prediction or classification of the identification and / or characterization of features;

[0014] (4) The term "salient image" broadly encompasses graphical representations of feature evaluations. Examples of salient images include, but are not limited to, heat maps, feature segmentation, and activation maps;

[0015] (5) The term "salient visualization" broadly covers the display of one or more salient images;

[0016] (6) The term "controller" broadly encompasses all structural configurations of a main circuit board or integrated circuit as understood in the art of the present disclosure and hereinafter contemplated for controlling the application of the various principles of the present disclosure as subsequently described in the present disclosure. The structural configuration of a controller may include, but is not limited to: processor(s), non-transitory machine-readable storage medium(s), operating system(s), application module(s), peripheral device controller(s), slot(s), and port(s); and

[0017] (7) "Data" may be implemented in the form of all forms of detectable physical quantities or pulses (e.g., voltage, current, magnetic field strength, impedance, color) as understood in the field of the present disclosure and as exemplarily described in the present disclosure for transmitting information and / or instructions supporting the application of various principles of the present disclosure as described later in the present disclosure. Data communications covered by the present disclosure may involve communication methods known in the field of the present disclosure, including but not limited to: data transmission / reception via any type of wired or wireless data link and reading of data uploaded to a computer-usable / computer-readable storage medium.

[0018] Hereinafter, in conjunction with the accompanying drawings, the aforementioned embodiments and other embodiments of the present disclosure and the various features and advantages of the present disclosure will become more apparent according to the detailed description of various embodiments of the present disclosure. These detailed descriptions and drawings are merely illustrative of the present disclosure and are not intended to be limiting. The scope of the present disclosure is defined by the accompanying claims and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] For a better understanding of various exemplary embodiments, reference is made to the accompanying drawings, in which:

[0020] Figure 1 An exemplary embodiment of a significant medical imaging method according to the principles of the present disclosure is illustrated;

[0021] Figure 2A illustrates an exemplary embodiment of an artificial intelligence engine and a medical image display engine according to the principles of the present disclosure;

[0022] Figure 2B A feature evaluation method according to the principles of the present disclosure is illustrated;

[0023] Figure 2C A saliency visualization method according to the principles of the present disclosure is illustrated;

[0024] Figure 3AAn exemplary embodiment of a two-dimensional (2D) convolutional neural network (CNN) according to the principles of the present disclosure is illustrated;

[0025] Figure 3B An exemplary embodiment of a two-dimensional (2D) convolutional neural network / support vector machine (CNN / SVM) according to the principles of the present disclosure is illustrated;

[0026] Figure 3C An exemplary embodiment of a 3D CNN according to the principles of the present disclosure is illustrated;

[0027] Figure 3D An exemplary embodiment of a 3D CNN / SVM according to the principles of the present disclosure is illustrated;

[0028] Figure 3E An exemplary embodiment of a series of networks of 2D CNNs according to the principles of the present disclosure is illustrated;

[0029] Figure 3F An exemplary embodiment of a series of networks of a 3D CNN according to the principles of the present disclosure is illustrated;

[0030] Figure 3G A first exemplary embodiment of a parallel network of 2D CNNs according to the principles of the present disclosure is illustrated;

[0031] Figure 3H A first exemplary embodiment of a parallel network of 3D CNNs according to the principles of the present disclosure is illustrated;

[0032] Fig. 3I A first exemplary embodiment of a parallel network of 2D CNNs according to the principles of the present disclosure is illustrated;

[0033] Figure 3J A first exemplary embodiment of a parallel network of 3D CNNs according to the principles of the present disclosure is illustrated;

[0034] Figure 4A illustrates an exemplary 2D planar CT image and an exemplary generation of a 2D planar heat map corresponding to the 2D planar CT image as known in the art of the present disclosure;

[0035] Figure 4B illustrates an exemplary rendering of a 3D volume CT image and an exemplary generation of a 3D volume heat map corresponding to the 3D volume CT image according to the principles of the present disclosure;

[0036] Figures 5A-5F illustrates an exemplary reslicing of a 3D volumetric saliency image at a first principal plane in accordance with the principles of the present disclosure;

[0037] Fig. 6Aillustrates an exemplary feature segmentation according to the principles of the present disclosure;

[0038] Figure 6B FIG. 1 illustrates a 3D volume CT image at a first principal plane and a Figure 4B Exemplary reslicing of a 3D volume heat map;

[0039] Figure 6C FIG. 3 illustrates a 3D volume CT image at a second principal plane and a Figure 4B Exemplary reslicing of a 3D volume heat map;

[0040] Fig.6D FIG. 1 illustrates a 3D volume CT image at a third principal plane and a Figure 4B Exemplary reslicing of a 3D volume heat map;

[0041] Figure 7 illustrates an exemplary embodiment of an activation map according to the principles of the present disclosure;

[0042] Figure 8 An exemplary embodiment of a clinical report according to the principles of the present disclosure is illustrated;

[0043] Fig. 9A illustrates an exemplary embodiment of a graphical user interface according to the principles of the present disclosure;

[0044] Fig. 9B A significant manipulation method according to the principles of the present disclosure is illustrated;

[0045] Fig. 10A illustrates an exemplary embodiment of an image mask graphical user interface (GUI) according to the principles of the present disclosure;

[0046] Fig. 10B illustrates an exemplary embodiment of a data modification graphical user interface (GUI) according to the principles of the present disclosure;

[0047] Fig. 10C illustrates an exemplary embodiment of a relevance threshold graphical user interface (GUI) in accordance with the principles of the present disclosure;

[0048] Fig. 10D illustrates an exemplary embodiment of an alternative evaluation graphical user interface (GUI) in accordance with the principles of the present disclosure;

[0049] Fig.11 An exemplary embodiment of a notable medical imaging controller in accordance with the principles of the present disclosure is illustrated;

[0050] Fig. 12A illustrates a first exemplary embodiment of a notable medical imaging system in accordance with the principles of the present disclosure; and

[0051] Fig. 12B A second exemplary embodiment of a salient medical imaging system according to the principles of the present disclosure is illustrated. DETAILED DESCRIPTION

[0052] As will be explained in further detail in this disclosure, the present disclosure is premised on significant visualization techniques that will allow a feature evaluation algorithm based on a machine learning model to 'explain' itself to a clinician. For the present disclosure, the machine learning model can be any type of predictive machine learning model or any type of classification machine learning model known in the art of this disclosure, including but not limited to: deep neural networks (e.g., convolutional neural networks, recurrent neural networks, etc.) and supervised learning machines (e.g., linear or nonlinear support vector machines, boosting classifiers, etc.).

[0053] To facilitate understanding of the present disclosure, Figure 1 The following description of teaches various embodiments of the saliency visualization method of the present disclosure. Figure 1 With the description of the disclosure, those skilled in the art will recognize how to apply the disclosure to many and various additional embodiments of making and using the saliency visualization methods of the disclosure.

[0054] refer to Figure 1 The salient medical imaging method 20 of the present disclosure implements a feature evaluation stage S22 , a salient visualization stage S24 , and a salient manipulation stage 26 .

[0055] The feature evaluation stage S22 includes applying an artificial intelligence engine including one or more machine learning models to the medical imaging 30 of the body, human or animal to draw a feature evaluation 31 of the medical imaging 30.

[0056] In practice, medical imaging 30 encompasses visualization of part or all of a body used in many and various procedures, including but not limited to clinical diagnostic procedures, interventional / surgical procedures, and patient monitoring. Examples of medical imaging 30 include but are not limited to two-dimensional (2D) medical imaging and three-dimensional (3D) medical imaging of anatomical regions / organs represented by raw imaging data generated by medical imaging machines (e.g., 2D / 3D US, 2D / 3D X-ray, CT, MRI, PET, SPECT, and DOT) or by medical images of anatomical regions / organs reconstructed from the raw imaging data.

[0057] In practice, feature evaluation 31 encompasses prediction or classification of one or more features illustrated within a medical imaging of a body based on identification and / or characterization of feature(s). More specifically, features encompass any type of object within a medical image that is identifiable and / or characterizable by one or more trained machine learning models, including, but not limited to: anatomical objects (e.g., vessels, organs, etc.), foreign bodies (e.g., procedure tools / instruments and implanted devices), and image artifacts (e.g., noise, grating lobes).

[0058] Examples of classification of features illustrated within medical imaging 30 include, but are not limited to, classification of: (1) anatomical structures (e.g., gray matter or white matter of the brain), (2) foreign bodies (e.g., guidewires, implanted stents, or valves), (3) malignancy of lesions (e.g., skin lesions are benign or malignant), (4) specific diseases (e.g., positive or negative for diabetic retinopathy), (5) multi-category lesion classification (e.g., normal, glioblastoma, sarcoma, and brain metastasis), and (6) changes within specific tissues (e.g., stenosis in vessels, blood pooling in tissues / organs, and changes in tissue stiffness).

[0059] Examples of predictions of features illustrated within medical imaging 30 include, but are not limited to: predicting (1) whether a mass is noncancerous (negative diagnosis) or cancerous (positive diagnosis), such as, for example, a risk score for a lung nodule; and (2) patient outcomes, such as, for example, a high likelihood and a low likelihood of, for example, five-year disease-specific survival for a patient diagnosed with colorectal cancer.

[0060] Still reference Figure 1 , after the rendering of the feature evaluation 31 during stage S22 , the saliency visualization stage 24 comprises the display of one or more medical images 32 and a saliency image 33 .

[0061] In practice, the medical image 32 encompasses any type of image acquired by a medical imaging machine (e.g., a 2D / 3D ultrasound machine, a 2D / 3D X-ray machine, a CT machine, an MRI machine, a PET machine, a SPECT machine, and a DOT machine). Examples of the medical image 32 include, but are not limited to: (1) an image frame or video as acquired by a medical imaging machine, (2) an image segmentation of (one or more) features illustrated within the medical image 30, and (3) an image overlay of an image frame / video acquired by one type of medical imaging machine onto an image frame / video acquired by a different type of medical imaging machine (e.g., an overlay of a 2D ultrasound image onto a 2D / 3D X-ray image). The medical image 32 may be displayed as a plan view or a volume view of the medical image 30.

[0062] In practice, the salient image 33 encompasses a graphical representation of the feature evaluations 31. Examples of salient images 33 include, but are not limited to, heat maps, feature segmentations, and activation maps.

[0063] As will be further described in this disclosure, a heat map encompasses a planar or volumetric graphical representation of feature evaluations 31 using a color scheme (eg, grayscale and visible spectrum) to highlight any correlation differences between features and feature evaluations 31 of medical imaging 30 .

[0064] As will be further described in the present disclosure, feature segmentation encompasses planar or volumetric segmentation of (one or more) features with specific minimal relevance to feature evaluation 31 of medical imaging 30, whereby (one or more) features can be displayed individually or highlighted within a displayed medical image 32.

[0065] As will be further described in this disclosure, an activation map encompasses a plan view of an n×m matrix of the outputs of (one or more) machine learning models (e.g., filtered outputs of a neural network) arranged in order of relevance (e.g., left to right or top to bottom), n≥1 and m≥1.

[0066] During the saliency visualization stage 24 , the present disclosure provides for both relational and user-specified reslicing of the volumetric saliency image 33 .

[0067] In practice, as will be further described in the present disclosure, correlation reslicing of a volumetric salient image involves reslicing a planar salient image from a volumetric salient image by (one or more) machine learning models based on the level of correlation of each feature with the feature evaluation 31 of the volumetric medical image. More specifically, the salient voxels of the volumetric salient image have a correlation level exceeding a correlation threshold, and a plurality of salient voxels will define a salient view of the evaluated feature of the medical image 30. For the purpose of reslicing, the center point and orientation of the resliced ​​planar salient image within the volumetric salient image are derived from the intersection of the resliced ​​salient plane with the volumetric salient image including the one or more salient voxels.

[0068] In a first correlation reslicing embodiment as will be further described in the present disclosure, the center point and orientation of the resliced ​​planar salient image within the volumetric salient image are derived from the intersection of the resliced ​​salient plane with the volumetric salient image comprising the highest spatial distribution of salient voxels, the highest sum of salient voxels, or the highest average of salient voxels relative to a coordinate plane of the volumetric salient image.

[0069] In a second associative reslicing embodiment as will be further described in the present disclosure, the position and orientation of the resliced ​​planar salient image within the volumetric salient image is derived from the intersection of the resliced ​​salient plane with the volumetric salient image based on a coordinate plane (i.e., XY plane, XZ plane, or YZ plane) of the volumetric salient image. More specifically, the position and orientation of the resliced ​​planar salient image within the volumetric salient image is derived from the intersection of the resliced ​​salient plane with the volumetric salient image including the highest spatially distributed salient voxel, the highest sum of salient voxels, or the highest average of salient voxels relative to the coordinate plane of the volumetric salient image.

[0070] In a third associative reslicing embodiment as will be further described in the present disclosure, the position and orientation of the resliced ​​planar salient image within the volumetric salient image is derived from the intersection of the resliced ​​salient plane with the volumetric salient image based on the centroid of the volumetric voxels. More specifically, the position and orientation of the resliced ​​planar salient image within the volumetric salient image is derived from the intersection of the resliced ​​salient plane with the volumetric salient image, relative to the centroid of the salient voxels, with the volumetric salient image comprising the highest spatially distributed salient voxels, the highest sum of salient voxels, or the highest average of salient voxels.

[0071] In practice, the user-specified reslicing of the volumetric salient image involves reslicing the planar salient image from the volumetric salient image based on a user-specified center point or orientation of the planar salient image including one or more salient voxels. More specifically, the position and orientation of the resliced ​​planar salient image within the volumetric salient image is derived from the intersection of the resliced ​​salient plane with the volumetric salient image including the highest spatially distributed salient voxel, the highest sum of salient voxels, or the highest average of salient voxels based on the user-specified center point or orientation of the planar salient image.

[0072] Still reference Figure 1 , prior to initial rendering of the feature evaluation 31 during stage S22 or after initial display of image 32 or image 33, the significant manipulation stage S26 includes user interaction with the artificial intelligence engine via a graphical user interface (GUI) to manipulate the significant visualization of the feature evaluation 31 of the medical imaging 30.

[0073] In practice, salient visualization of feature evaluations 31 of medical imaging 30 includes static or interactive display of (one or more) salient images 33 and optionally medical images 32 in a manner that supports underlying processes, and manipulation of salient visualization of medical imaging 30 includes medical imaging interactions 34 with an artificial intelligence engine for masking or modifying medical imaging 30, or feature evaluation interactions 35 for changing correlation thresholds for features of medical imaging 40 or assuming alternative feature evaluations 31 for medical imaging 30.

[0074] More specifically, in a data mask embodiment of a medical imaging interaction 34 as will be further described in the present disclosure, the displayed GUI enables the clinician to interact directly with a planar medical image or a volumetric medical image and see how the artificial intelligence engine responds. For example, the clinician can analyze (one or more) significant images 33 to confirm whether one or more specific regions of the planar medical image or the volumetric medical image are irrelevant to the current feature evaluation 31. If the clinician suspects that one or more specific regions of the planar medical image or the volumetric medical image are irrelevant to the current feature evaluation 31 of such an image, then the clinician can mask (one or more) irrelevant images and view (one or more) significant images to see the impact such masking has on the feature evaluation 31 of the medical imaging 30.

[0075] In a data modification embodiment of a medical imaging interaction 34 as will be further described in the present disclosure, the displayed GUI enables the clinician to test the effect of each input into the artificial intelligence engine on the feature evaluation 31 of the medical imaging 30 as visualized by the salient image(s) 30. Examples of data modification include, but are not limited to: (1) enabling / disabling a medical imaging input or combination of medical imaging inputs to the artificial intelligence engine, (2) increasing / decreasing pixel / voxel intensity values ​​of a medical imaging input or combination of medical imaging inputs to the artificial intelligence engine, (3) enabling / disabling an auxiliary information input or combination of auxiliary information inputs to the artificial intelligence engine, or (4) changing an auxiliary information input or combination of auxiliary information inputs to the artificial intelligence engine.

[0076] In a correlation threshold embodiment of a feature evaluation interaction 35 as will be further described in the present disclosure, the displayed GUI enables the clinician to set a correlation level of pixels of a planar medical image with a feature evaluation 31 of the planar medical image, or enables the clinician to set a correlation level of voxels of a volumetric medical image with a feature evaluation 31 of the volumetric medical image. The insight is that not all pixels of a planar medical image or all voxels of a volumetric planar image are equally correlated with the feature evaluation 31, and clinicians typically focus on the feature(s) of the image that they are most suspicious of. In this way, the correlation level of pixels of a planar medical image or voxels of a volumetric planar image can be set by the clinician to a value such that if a feature of the image reaches that correlation level, the feature will be more distinctively highlighted within the salient image(s) 33.

[0077] In an alternative feature evaluation embodiment of a feature evaluation interaction 35 as will be further described in the present disclosure, the clinician can make assumptions between different predictions or different classifications (e.g., prediction / classification x and prediction / classification y) of a feature evaluation 31 of the medical imaging 30, and if the artificial intelligence engine plots prediction / classification x of the medical imaging 30 as visualized by (one or more) salient images 31 illustrating prediction / classification x of the medical imaging 30, then the clinician can select prediction / classification y via the GUI to see the features most relevant to prediction / classification y of the medical imaging 30 as visualized by (one or more) salient images 31 illustrating prediction / classification y of the medical imaging 30.

[0078] In practice, the medical imaging interaction 34 and the feature evaluation interaction 35 may be used individually or in combination.

[0079] The flowchart 20 will continue to conditionally loop through the stages S22 - S26 until the associated process or the associated stages of the associated process are terminated.

[0080] In order to promote a further understanding of the present disclosure, Figure 2A-2C The following description teaches various embodiments of the artificial intelligence engine and medical image display engine of the present disclosure. Figure 2A-2C With the description of the present disclosure, a person of ordinary skill in the art will recognize how to apply the present disclosure to many and various additional embodiments of making and using the artificial intelligence engine and medical image display engine of the present disclosure.

[0081] refer to Figure 2A , the artificial intelligence engine 40 draws the feature evaluation 31 of the medical imaging 30, as previously described for Figure 1 To this end, the artificial intelligence engine 40 includes a method for implementing Figure 2B Flowchart 140 shows a feature evaluation method of a data preprocessor 41 , one or more machine learning models 42 , and a salient image manager 43 .

[0082] refer to Figure 2A and Figure 2BIn both cases, the data preprocessor 41 processes the planar medical imaging data 30a or the volumetric medical imaging data 30b during stage S142 of the flowchart 140 by inputting the data 30a / 30b into the (one or more) machine learning models 42 to thereby draw the feature evaluation 31a during stage S144 of the flowchart 140. In addition, the data preprocessor 41 may further process auxiliary imaging information 30c associated with the planar medical imaging data 30a or the volumetric medical imaging data 30b (e.g., the radius and / or sphericity of the anatomical organ segmented from the reconstructed medical image 132a) from a medical imaging machine (e.g., a 2D / 3D ultrasound machine, a 2D / 3D X-ray machine, a CT machine, an MRI machine, a PET machine, a SPECT machine, and a DOT machine) for inputting the information 30c into the (one or more) machine learning models 42 to thereby draw the feature evaluation 31a during stage S144. The auxiliary imaging information 30c may further include information related to the patient in terms of age, gender, and health history. Examples of processing the medical image data 30a / 30b and the auxiliary imaging information 30c by the data preprocessor 25 include, but are not limited to: encoding, embedding or time sampling of the medical image data 30a / 30b and the auxiliary imaging information 30c.

[0083] In practice, if such pre-processing is not necessary or has been previously performed before transmitting the medical image data 30a / 30b and optional auxiliary imaging information 30c to the AI ​​engine 40 , the data pre-processor 41 may be omitted from the AI ​​engine 40 .

[0084] Return to reference Figure 2A , the machine learning model 42 is a classification machine learning model or a prediction machine learning model, including but not limited to: (1) a deep neural network (e.g., a convolutional neural network, a recursive neural network, etc.) and (2) a supervised learning machine (e.g., a linear or nonlinear support vector machine, a boosting classifier, etc.). In practice, the machine learning model 43 can be trained to draw a feature evaluation 31a of medical imaging 30a / 30b for any type of medical procedure, any type of medical imaging machine, or any type of patient, or alternatively can be trained to draw a feature evaluation of medical imaging data 30a / 30b for one or more specific medical procedures or one or more specific types of medical imaging machines, whereby the machine learning model 43 can be further trained for one or more specific types of patients. Similarly, in practice, the AI ​​engine 40 may include a single machine learning model 42 or multiple similar and / or different machine learning models 42, including but not limited to: (1) a distributed arrangement of individual machine learning models 42, (2) a network of two or more similar and / or different deep neural networks, or (3) a network of two or more similar and / or different supervised learning machines.

[0085] refer to Figure 2A and Figure 2B In both cases, the machine learning model(s) 42 of the AI ​​engine 40 inputs (pre-processes) the medical imaging data 30a / 30b to draw feature evaluations 31a of the medical imaging data 30a / 30b during stage S144 of the flowchart 140, as previously described for Figure 1 After initial execution of stage S144, during stage S146 of flowchart 140, salient image manager 43 generates salient image data 36a representing the relevance of each feature of medical imaging data 30a / 30b to feature evaluation 31a.

[0086] In practice, the salient image data 36a represents the relevance of each feature of the masked / corrected medical imaging data 30a / 30b to the new feature evaluation 31a. In one embodiment for any type of machine learning model, the salient image data 36a provides a scaling of each pixel represented by the planar medical imaging data 30a and each voxel of the volumetric medical imaging data 40a from the least relevant pixel(s) / (s) voxel(s) (e.g., 0) to the most relevant pixel(s) / (s) voxel(s) (e.g., 169). The scaling enables the salient image generator 51 to generate a heat map or feature map. In a second embodiment specifically for a neural network, the scaled image data 36a provides filter outputs ranging from the least relevant filter output(s) to the most relevant filter output(s).

[0087] More specifically, a deep neural network encompasses multiple layers of activated neurons (i.e., feature elements) that are interconnected and trained for feature extraction and transformation to thereby compute a complex mapping between (preprocessed) medical imaging data 30a / 30b (e.g., 2D pixels or 3D voxels) / auxiliary information 30c (if applicable) and a prediction or classification via one or more output neurons (i.e., prediction elements). Each activated neuron applies a non-linear activation function to a weighted linear combination of the inputs (e.g., 2D pixels, 3D voxels, upstream activated neuron outputs, or downstream activated neuron outputs). Thus, the parameters that are important to a deep neural network are the structure of the connected neuronal network, the non-linear activation function, and the weights of the activated neurons. Likewise, the ability to perform any type of correlation mapping for confirming the interpretation of the prediction or classification is important, whereby such correlation mapping can be used for the salient image data 36a of the present disclosure (e.g., backpropagation, guided backpropagation, deconvolution, class activation mapping, gradient class activation mapping, etc.). For example, the detected class activations may be back-projected through the network into the input pixel / voxel space to detect which parts of the input medical image 30a / 30b are most relevant for prediction or classification.

[0088] For deep neural network embodiments of the machine learning model(s) 42, the salient image manager 43 performs the aforementioned correlation mapping based on a default feature correlation threshold for each activation to thereby generate salient image data 36a for the drawn feature evaluations 31a.

[0089] Furthermore, support vector machines as known in the art of the present disclosure encompass support vectors trained to delineate hyperplanes (i.e., prediction elements) for linearly or nonlinearly classifying feature vectors (i.e., feature elements) of medical imaging data 30a / 30b (e.g., 2D pixels or 3D voxels), auxiliary information 30c (if applicable) into one of two classes. A linear support vector machine specifically learns a linear discriminant function, whereby the weights of the linear discriminant function can be used to assign a score to each 2D pixel or 3D voxel to thereby support the drawing of feature evaluations 31a.

[0090] For support vector machine embodiments of the machine learning model(s) 42, the salient image manager 43 may utilize 2D pixel / 3D voxel scores in conjunction with a nearest neighbor algorithm relative to a default feature relevance threshold to thereby generate salient image data 36a for the drawn feature evaluations 31a.

[0091] Still reference Figure 2A and Figure 2BAfter generating the salient information data 36a during stage S146, the salient image manager 43 proceeds to stage S148 of flowchart 140, whereby the salient image manager 43 returns to stage S142 after receiving the data mask designation 34a or the data correction designation 34b, or returns to stage S146 after receiving the feature correlation threshold designation 35a or the alternative feature evaluation designation 35b.

[0092] More specifically, if the mask designation 34a or the data modification designation 34b is received by the significant image manager 43 via the GUI (as will be further described in the present disclosure), the significant image manager 43 directs the data preprocessor 41 to perform data masking of the planar medical image data 30a or the volumetric medical image data 30b (as directed by the data mask designation 34a, as previously described for Figure 1 or data modification of the planar medical image data 30a or the volumetric medical image data 30b (such as guided by the data mask specification 34a, as previously described for Figure 1 Subsequently, stage S144 is repeated to draw new feature estimates 31a based on the data mask or data correction, and stage S146 is repeated to generate new salient image data 36a representing the correlation of each feature of the masked / corrected medical imaging data 30a / 30b with the new feature estimates 31a.

[0093] Additionally, if a feature relevance threshold specification 35a or an alternative feature rating specification 35b is received by salient image manager 43 via the GUI (as will be further described in this disclosure), salient image manager 43 may then select the feature relevance threshold specified by feature relevance threshold specification 35a based on the feature relevance threshold specified by feature relevance threshold specification 35a (as previously described for salient image manager 43). Figure 1 or by the alternative feature evaluation designated by the alternative feature evaluation designation 35b (as previously described for Figure 1 The salient manipulation stage S26 described above generates salient image data 36a.

[0094] The flowchart 140 will continue to conditionally loop through stages S142 - S148 until the associated process or the associated stage of the associated process is terminated.

[0095] Figures 3A-3JIllustrative examples of trained convolutional neural networks (CNNs) that draw feature estimates for prediction or classification of planar medical image data 30a or volumetric medical image data 30b are illustrated. For each example, the convolutional neural network employs one or more convolutional / rectified linear unit (ReLU) / pooling layers and fully connected layers to output feature estimates therefrom (e.g., a convolutional neural network (CNN) with four (4) convolutional layers trained to provide a specific type of prediction (including but not limited to tumor classification, vessel segmentation, etc.)). Alternatively, the fully connected layers of the CNN are connected to or replaced by a support vector machine (CNN / SVM).

[0096] In practice, the CNN of the present disclosure is trained on 2D or 3D medical imaging of the body, which covers the entire range of prediction scores for the entire set of classifications. Additionally, the CNN of the present disclosure is constructed to output salient image data for the purpose of generating a planar salient image or a volumetric salient image, as will be further explained in the present disclosure. Specifically, the salient image data is the filter output of the last convolutional layer of the CNN.

[0097] Furthermore, in practice, the CNN of the present disclosure can process medical imaging interaction data indicating the following items: (1) a clinician masks a region of a medical planar medical image or a medical volumetric medical image to see the effect of the data mask on feature evaluation (as previously described for Figure 1 ), or (2) the clinician enables / disables and / or changes to the image and auxiliary information inputs to the CNN to see the impact of data modification on feature evaluation (as previously described for Figure 1 ) as described in stage S26.

[0098] Additionally, in practice, the CNN of the present disclosure may process feature rating interaction data indicating: (1) the level of correlation of pixels of planar medical imaging data or voxels of volumetric medical imaging set by the clinician for generating (one or more) saliency images (as previously described for Figure 1 ) or indicates (2) a clinician's choice between different hypotheses (e.g., x and y) (as previously described for stage S26 of Figure 1 ) as described in stage S26.

[0099] In such as Figure 3AIn one embodiment of the AI ​​engine 40 shown in , the 2D CNN 242a is trained on a variety of interpreted medical images of previous patients as known in the art of the present disclosure to draw a feature evaluation 231a of medical imaging of the body represented by the planar medical imaging data 30a. Additionally, the 2D CNN 242a is operated by the salient image manager 43 via a correlation map to output 2D salient image data 236a (e.g., a 2D heat map, a 2D feature map, or an activation map) for the purpose of generating a planar salient image illustrating the feature evaluation 231a. In addition, the 2D CNN 242a operates according to the medical imaging interaction data 34c or the feature evaluation interaction data 35c via (one or more) GUIs as previously described in the present disclosure.

[0100] In such as Figure 3B In the second embodiment of the AI ​​engine 40 shown in , the 2D CNN / SVM 242b is trained on a variety of interpreted medical images of previous patients as known in the field of the present disclosure to draw a feature evaluation 231b of medical imaging of the body represented by the planar medical imaging data 30a. Additionally, the 2D CNN / SVM 242b is operated by the salient image manager 43 via a correlation map to output 2D salient image data 236b (e.g., a 2D heat map, a 2D feature map, or an activation map) for the purpose of generating a planar salient image illustrating the feature evaluation 231b. In addition, the 2D CNN / SVM 242b operates according to the medical imaging interaction data 34c or the feature evaluation interaction data 35c via (one or more) GUIs as previously described in the present disclosure.

[0101] In such as Figure 3C In the third embodiment of the AI ​​engine 40 shown in , a 3D CNN 242c is trained on a variety of interpreted medical images of previous patients as known in the art of the present disclosure to draw feature evaluations 231c of medical imaging of the body represented by the volumetric medical imaging data 30b. Additionally, the AI ​​engine 242c is operated by the salient image manager 43 via correlation mapping to output 2D salient image data 236c (e.g., a 3D heat map, a 3D feature map, or an activation map) for the purpose of generating a volumetric salient image illustrating the feature evaluations 231c. Furthermore, the 3D CNN 242c operates according to the medical imaging interaction data 34d or the feature evaluation interaction data 35d via (one or more) GUIs as previously described in the present disclosure.

[0102] In such as Figure 3DIn a fourth embodiment of the AI ​​engine 40 shown in , a 3D CNN / SVM 242d is trained on a variety of interpreted medical images of previous patients as known in the art of the present disclosure to draw feature evaluations 231d of medical imaging of a body represented by volumetric medical imaging data 30b. Additionally, the 3D CNN / SVM 242d is operated by a salient image manager 43 via a correlation map to output 2D salient image data 236d (e.g., a 3D heat map, a 3D feature map, or an activation map) for the purpose of generating a volumetric salient image illustrating the feature evaluations 231d. Furthermore, the 3D CNN / SVM 242d operates according to the medical imaging interaction data 34d or the feature evaluation interaction data 35d via (one or more) GUIs as previously described in the present disclosure.

[0103] In such as Figure 3E In the fifth embodiment of the AI ​​engine 40 shown in FIG. , the 2D CNN 242a ( Figure 3A ) acts as a secondary CNN that provides feature evaluations 231a and salient image data 236 as input to a 2D CNN 242e that acts as a primary CNN. The 2D CNN 242e is trained on a variety of interpreted medical images and feature evaluations 231a of previous patients as known in the art of the present disclosure to draw feature evaluations 231e of medical imaging of the body as represented by volumetric medical imaging data 30b. Additionally, the 2D CNN 242e is operated by the salient image manager 43 via correlation mapping to output 2D salient image data 236e (e.g., a 2D heat map, a 2D feature map, or an activation map) for the purpose of generating a planar salient image illustrating the feature evaluations 231a. Furthermore, the 2D CNN 242e operates according to the medical imaging interaction data 34e or the feature evaluation interaction data 35e via (one or more) GUIs as previously described in the present disclosure.

[0104] In such as Figure 3F In the sixth embodiment of the AI ​​engine 40 shown in FIG. , the 3D CNN 242c ( Figure 3C) acts as a secondary CNN that provides feature evaluations 231c and salient image data 23c as input to a 3D CNN 242f acting as a primary CNN. The 3D CNN 242f is trained on a variety of interpreted medical images of previous patients and feature evaluations 231c as known in the art of the present disclosure to draw feature evaluations 231f of medical imaging of a body represented by volumetric medical imaging data 30b. Additionally, the 3D CNN 242f operates via correlation mapping to output 3D salient image data 236f (e.g., a 3D heat map, a 3D feature map, or an activation map) for the purpose of generating a volumetric salient image illustrating the feature evaluations 231f. Furthermore, the 3D CNN 242f operates according to the medical imaging interaction data 34f or the feature evaluation interaction data 35f via (one or more) GUIs as previously described in the present disclosure.

[0105] In such as Figure 3G In the seventh embodiment of the AI ​​engine 40 shown in , a 2D CNN 242g is trained on a class of positively interpreted medical images of previous patients (e.g., images of cancer patients, images of the presence of surgical tools within anatomical regions, etc.) as known in the field of the present disclosure to draw a preliminary feature evaluation 231g of medical imaging of the body represented by the planar medical imaging data 30a. Additionally, the 2D CNN 242g is operated by the salient image manager 43 via correlation mapping to output 2D salient image data 236g (e.g., a 2D heat map, a 2D feature map, or an activation map) for the purpose of generating a planar salient image illustrating the feature evaluation 231g. In addition, the 2D CNN 242g is operated based on the medical imaging interaction data 34g or the feature evaluation interaction data 35g via (one or more) GUIs as previously described in the present disclosure.

[0106] Still reference Figure 3G , the 2D CNN 242h is trained as known in the art of the present disclosure on a class of negatively interpreted medical images of previous patients (e.g., images of non-cancer patients, images of the absence of surgical tools within anatomical regions, etc.) to draw a preliminary feature evaluation 231h of medical imaging of the body represented by the planar medical imaging data 30a. Additionally, the 2D CNN 242h is operated by the salient image manager 43 via correlation mapping to output 2D salient image data 236h (e.g., a 2D heat map, a 2D feature map, or an activation map) for the purpose of generating a planar salient image illustrating the feature evaluation 231h. Furthermore, the 2D CNN 242h is operated based on the medical imaging interaction data 34h or the feature evaluation interaction data 35h via (one or more) GUIs as previously described in the present disclosure.

[0107] Still reference Figure 3G , the classifier / predictor 242i is constructed to generate a final feature evaluation 231i of the medical imaging of the body represented by the planar medical imaging data 30a. In practice, the classifier / predictor 242i may implement any technique known in the art of the present disclosure for combining the preliminary feature evaluation 231g and the preliminary feature evaluation 231h to draw the final feature evaluation 231i. In one embodiment, the classifier / predictor 242i may average the risk scores associated with the preliminary feature evaluation 231g and the preliminary feature evaluation 231h to thereby draw the final feature evaluation 231i. In a second embodiment, the classifier / predictor 242i may implement a fully connected layer representing the networks 242g and 242h.

[0108] Still reference Figure 3G , classifier / predictor 242i is also configured to output salient image data 236i (e.g., a 2D heat map, a 2D feature map, or an activation map) for the purpose of generating a planar salient image illustrating feature evaluation 231i. In practice, classifier / predictor 242i may implement any technique known in the art of the present disclosure for combining the filter outputs of the convolutional layers of networks 242g and 242h to produce salient image data 236i (e.g., a 2D heat map, a 2D feature map, or an activation map) illustrating feature evaluation 231i. In one embodiment, classifier / predictor 242i may average the filter outputs of the last convolutional layers of networks 242g and 242h to produce salient image data 236i.

[0109] In such as Figure 3H In the eighth embodiment of the AI ​​engine 40 shown in , the 3D CNN 242j is trained on a class of positively interpreted medical images of previous patients (e.g., images of cancer patients, images of the presence of surgical tools within anatomical regions, etc.) as known in the field of the present disclosure to draw a preliminary feature evaluation 231j of medical imaging of the body represented by the volumetric medical imaging data 30b. Additionally, the 3D CNN 242j is operated by the salient image manager 43 via correlation mapping to output 3D salient image data 236j (e.g., a 3D heat map, a 3D feature map, or an activation map) for the purpose of generating a volumetric salient image illustrating the feature evaluation 231j. In addition, the 3D CNN 242j is operated according to the medical imaging interaction data 34j or the feature evaluation interaction data 35j via (one or more) GUIs as previously described in the present disclosure.

[0110] Still reference Figure 3H, the 3D CNN 242k is trained as known in the art of the present disclosure on a class of negatively interpreted medical images of previous patients (e.g., images of non-cancer patients, images of the absence of surgical tools within anatomical regions, etc.) to draw a preliminary feature evaluation 231k of medical imaging of the body represented by the volumetric medical imaging data 30b. Additionally, the 3D CNN 242k is operated by the salient image manager 43 via correlation mapping to output 3D salient image data 236k (e.g., a 3D heat map, a 3D feature map, or an activation map) for the purpose of generating a volumetric salient image illustrating the feature evaluation 231k. Furthermore, the 3D CNN 242k is operated based on the medical imaging interaction data 34k or the feature evaluation interaction data 35k via (one or more) GUIs as previously described in the present disclosure.

[0111] Still reference Figure 3H , the classifier / predictor 2421 is constructed to generate a final feature evaluation 2311 of the medical imaging of the body represented by the volumetric medical imaging data 30b. In practice, the classifier / predictor 2421 may implement any technique for combining the preliminary feature evaluation 231j and the preliminary feature evaluation 231k as known in the art of the present disclosure to draw the final feature evaluation 2311. In one embodiment, the classifier / predictor 2421 may average the risk scores associated with the preliminary feature evaluation 231j and the preliminary feature evaluation 231k to thereby draw the final feature evaluation 2311. In a second embodiment, the classifier / predictor 2421 may implement a fully connected layer representing the networks 242j and 242k.

[0112] Still reference Figure 3H , classifier / predictor 2421 is also configured to output salient image data 2361 (e.g., a 3D heat map, a 3D feature map, or an activation map) for the purpose of generating a volumetric salient image. In practice, classifier / predictor 2421 may implement any technique known in the art of the present disclosure for combining the filter outputs of the convolutional layers of networks 242j and 242k to produce salient image data 2361. In one embodiment, classifier / predictor 2421 may average the filter outputs of the last convolutional layers of networks 242j and 242k to produce salient image data 2361.

[0113] In such as Fig. 3I In the ninth embodiment of the AI ​​engine 40 shown in FIG. 1 , the classifier / predictor 242m is configured to generate a final feature evaluation 231m of the medical imaging of the body represented by the planar medical imaging data 30a. In practice, the classifier / predictor 242m may implement a method known in the art of the present disclosure for combining features from the 2D CNN 242a ( Figure 3A)’s preliminary feature evaluation 231c, from 2D CNN 242g( Figure 3G ) 231g and preliminary feature evaluation from 3D CNN 242h( Figure 3G ) to draw the final feature evaluation 231m. In one embodiment, the classifier / predictor 242m may average the risk scores associated with the preliminary feature evaluation 231c, the preliminary feature evaluation 231g, and the preliminary feature evaluation 231h to thereby draw the final feature evaluation 231m. In a second embodiment, the classifier / predictor 242m may implement a fully connected layer representing the networks 242a, 242g, and 242h.

[0114] Still reference Fig. 3I , classifier / predictor 241m is also configured to output salient image data 236m (e.g., a 2D heat map, a 2D feature map, or an activation map) for the purpose of generating a planar salient image illustrating feature evaluation 231m. In practice, classifier / predictor 241m may implement any technique known in the art of the present disclosure for combining the filter outputs of the convolutional layers of networks 242c, 242g, and 242h to produce salient image data 236m. In one embodiment, classifier / predictor 242i may average the filter outputs of the last convolutional layers of networks 242c, 242g, and 242h to produce salient image data 236m.

[0115] In such as Figure 3J In the tenth embodiment of the AI ​​engine 40 shown in FIG. 1 , the classifier / predictor 242n is configured to generate a final feature evaluation 231n of the medical imaging of the body represented by the volumetric medical imaging data 30b. In practice, the classifier / predictor 242n may implement a method known in the art of the present disclosure for combining features from the 3D CNN 242a ( Figure 3A ) preliminary feature evaluation 231c, from 3D CNN 242J ( Figure 3H ) and preliminary feature evaluation 231j from 3D CNN 242k (FIG. 3h) and preliminary feature evaluation 231K from 3D CNN 242k to draw final feature evaluation 231N. In one embodiment, classifier / predictor 242n may average the risk scores associated with preliminary feature evaluation 231c, preliminary feature evaluation 231j, and preliminary feature evaluation 231k to thereby draw final feature evaluation 231n. In a second embodiment, classifier / predictor 242n may implement a fully connected layer representing networks 242c, 242j, and 242k.

[0116] Still reference Figure 3J, classifier / predictor 241n is also configured to output salient image data 236n (e.g., a 3D heat map, a 3D feature map, or an activation map) for the purpose of generating a volumetric salient image illustrating feature evaluation 231n. In practice, classifier / predictor 242n may implement any technique known in the art of the present disclosure for combining the filter outputs of the convolutional layers of networks 242c, 242j, and 242k to produce salient image data 236n. In one embodiment, classifier / predictor 242n may average the filter outputs of the last convolutional layers of networks 242c, 242j, and 242k to produce salient image data 236n.

[0117] Return to reference Figure 2A The medical image display engine 50 controls the display of the medical image(s) 32 and the significant image(s) 33, as previously described with respect to Figure 1 To this end, the medical image display engine 50 includes a method for implementing the Figure 2C The flowchart 150 represents the image viewer 51 , the medical image generator 52 and the salient image generator 53 of the salient visualization method.

[0118] refer to Figure 2A and Figure 2C Both, during stage S152 of flowchart 150, planar medical imaging data 30a or volumetric medical imaging data 40a may be received in a viewable form by medical image display engine 50. After such reception, autonomously or via clinician activation, image viewer 51 proceeds to implement display of medical images represented by planar medical imaging data 30a or volumetric medical imaging data 40a, and also provides image navigation functions (e.g., zoom in, zoom out, rotate, etc.) and image annotation functions for clinicians viewing the displayed medical images. Additionally, medical image generator 52 may autonomously or via clinician activation implement: (1) volume rendering of a series of planar medical images represented by planar medical imaging data 30, (2) image segmentation of volumetric medical images represented by volumetric medical imaging data 40 or by volume rendering, or (3) any other technique known in the art for generating additional medical images.

[0119] During stage S152, the planar medical imaging data 30a or the volumetric medical imaging data 40a may alternatively be received in raw form by the medical image display engine 50. After such reception, the medical image generator 51 may implement image reconstruction techniques for the planar medical imaging data 30a or the volumetric medical imaging data 30b. After such imaging reconstruction, autonomously or via clinician activation, the image viewer 51 proceeds to implement display of the reconstructed medical image, and also provides image navigation functions (e.g., zoom in, zoom out, rotate, etc.) and image annotation functions for the clinician viewing the displayed reconstructed medical image. Additionally, the medical image generator 52 may autonomously or via clinician activation implement: (1) volume rendering of a series of reconstructed planar medical images, (2) image segmentation of a volumetric medical image represented by a reconstructed volumetric medical image or by a volume rendering, or (3) any other technique known in the art for generating additional medical images.

[0120] Subsequently or simultaneously with stage S152, during stage S154 of the flowchart, the feature evaluation data 31a and the significant image data 36a are received by the medical image display engine 50. After such reception, autonomously or via clinician activation, the image viewer 51 proceeds to display the feature evaluation data 31a in textual form or graphical form, and the significant image generator 53 processes the significant image data 36a to generate (one or more) significant images 33 for display by the image viewer 51.

[0121] Again, in practice, the salient image data 36a represents the relevance of each feature of the masked / corrected medical imaging data 30a / 30b to the new feature evaluation 31a. In one embodiment for any type of machine learning model, the salient image data 36a provides a scaling of each pixel represented by the planar medical imaging data 30a and each voxel of the volumetric medical imaging data 40a from the least relevant pixel(s) / voxel(s) (e.g., 0) to the most relevant pixel(s) / voxel(s) (e.g., 169). The scaling enables the salient image generator 51 to generate a heat map or feature map. In a second embodiment specifically for a neural network, the scaled image data 36a provides filter outputs ranging from the least relevant filter output(s) to the most relevant filter output(s).

[0122] In a convolutional neural network embodiment, the salient image manager 43 calculates the gradient of the species with respect to the last 3D convolutional layer of the CNN, and uses global average pooling to obtain the weights of each filter output, whereby the salient image generator 53 generates a heat map by applying the weights to the corresponding filter outputs and ReLU, and then normalizing and scaling the filter outputs to the desired size.

[0123] For example, Figure 4A An exemplary planar CT image 132a associated with an intensity scale 137a ranging from 0 to 255 and a corresponding planar heat map 133a associated with an importance scale 137b ranging from 0 to 169 are shown. Figure 4A , each pixel of the planar CT image 132a has been assigned an intensity value based on the level of correlation with the feature evaluation 31a, and the importance level of the pixel as displayed in the planar heat map 133a represents the level of correlation of the pixel with the feature evaluation 31a, such as, for example, a pixel shown at a level of 0 within the planar heat map 133a indicates that the pixel has the least correlation level with the feature evaluation 31a, and a pixel shown at a level of 169 within the planar heat map 133a indicates that the pixel has the most correlation level with the feature evaluation 31a.

[0124] Similarly, for example, Figure 4B An exemplary volumetric CT image 132b associated with an intensity scale 137a ranging from 0 to 255 and a corresponding volumetric heat map 133b associated with an importance scale 137b ranging from 0 to 169 are shown. Figure 4A , each voxel of the volume CT image 132b has been assigned an intensity value based on the level of correlation with the feature evaluation 31b, and the importance level of the voxel as displayed in the volume heat map 133b represents the level of correlation of the voxel with the feature evaluation 31b, such as, for example, a voxel shown at a level of 0 within the volume heat map 133b represents that the voxel has the least correlation level with the feature evaluation 31b, and a voxel shown at a level of 169 within the volume heat map 133b represents that the voxel has the most correlation level with the feature evaluation 31b.

[0125] Return to reference Figure 2C , for the heat map embodiment, stage S154 also includes a salient image generator 53 ( Figure 2A ) defines resliced ​​salient planes that intersect the volumetric heat map to expose the most relevant or descriptive information for feature evaluation 31a of the medical imaging 30, and the image viewer 51 provides user interactive functionality for manual exploration and navigation within the volumetric space of the medical imaging 30 by a clinician based on correlation-based guidance.

[0126] More specifically, the salient image generator 53 autonomously or via clinician activation performs: (1) correlation reslicing of a planar salient image from a volumetric salient image based on the level of correlation of each feature with the feature evaluation 31 of the volumetric medical image via (one or more) machine learning models, or (2) user-specified reslicing of a center point or orientation of a planar salient image from a volumetric salient image.

[0127] In practice, the correlation reslicing and user-specified reslicing of stage S154 are premised on the fact that the salient voxels of the volumetric salient image have a correlation level exceeding a correlation threshold and that a plurality of salient voxels will define a salient view of an evaluated feature of the volumetric medical imaging data 30b. For reslicing purposes, the center point and orientation of the resliced ​​planar salient image within the volumetric salient image are derived from the intersection of the resliced ​​salient plane with the volumetric salient image including one or more salient voxels.

[0128] In practice, correlation reslicing involves identification of salient voxels, delineation of plane intersections that include one or more salient voxels, and then optimizing selection of one of the plane intersections of a view of salient voxels, such as, for example, selection of the salient voxel with the highest spatial distribution, the highest sum of salient voxels, or the highest mean of salient voxels.

[0129] In practice, user-specified reslicing involves identification of salient voxels, user specification of a center point or orientation of a plane intersection, delineation of a plane intersection that includes one or more salient voxels, and then optimizing selection of one of the plane intersections of a view of salient voxels, such as, for example, selection of the salient voxels with the highest spatial distribution, the highest sum of salient voxels, or the highest average of salient voxels.

[0130] For example, if for clarity reasons Figure 5A , the volumetric saliency image 230 includes a set 231 of rows of salient voxels having correlation levels 232-235 of increasing intensity, wherein each correlation level exceeds a correlation threshold. For this example, the correlation threshold may be 200, the correlation level 232 may be 210, the correlation level 233 may be 225, the correlation level 234 may be 240, and the correlation level 235 may be 255.

[0131] Figure 5B An XY plane view of a row 231 of salient voxels is illustrated, wherein horizontal lines represent intersections of candidate resliced ​​salient YZ planes with a volumetric salient image 230 including one or more salient voxels, and vertical lines represent intersections of candidate resliced ​​salient XZ planes with a volumetric salient image 230 including one or more salient voxels. Figure 5BThe reslicing salient plane 236a or the reslicing salient plane 236b may be selected based on the highest spatially distributed salient voxels, the highest sum of salient voxels, or the highest average of salient voxels.

[0132] Figure 5C An XY plane view of a row 231 of salient voxels is illustrated, wherein the horizontal represents the intersection of a candidate re-sliced ​​salient YZ plane with a volumetric salient image 230 including one or more salient voxels. Figure 5C The re-slicing saliency plane 236c may be selected based on the highest spatially distributed salient voxels, the highest sum of salient voxels, or the highest average of salient voxels.

[0133] Figure 5D An XY plane view of a row 231 of salient voxels is illustrated, wherein black dots represent centroids of salient voxels and lines represent intersections of candidate resliced ​​salient YZ planes with a volumetric salient image 230 passing through the centroids including one or more salient voxels. Figure 5D The re-slicing salient plane 236d may be selected based on the highest spatially distributed salient voxel relative to the centroid, the highest sum of salient voxels, or the highest average of salient voxels.

[0134] In practice, the centroid can be calculated as all 3D points of salient voxels (p∈R 3 ) is the arithmetic mean of .

[0135] Figure 5E An XY plane view of a row 231 of salient voxels is illustrated, wherein a black dot represents a user-specified center point and a line represents the intersection of a candidate resliced ​​salient YZ plane with a volumetric salient image 230 passing through a user-specified location including one or more salient voxels. Figure 5E The re-slicing salient plane 236e may be selected based on the highest spatially distributed salient voxel, the highest sum of salient voxels, or the highest average of salient voxels relative to a user-specified center point.

[0136] Fig. 5F An XY plane view of a row 231 of salient voxels is illustrated, wherein the black line represents the intersection of a candidate resliced ​​salient YZ plane with the volumetric salient image 230 at a user-specified orientation including one or more salient voxels. Fig. 5F The re-slicing salient plane 236F may be selected based on the highest spatially distributed salient voxel, the highest sum of salient voxels, or the highest average of salient voxels relative to a user-specified center point.

[0137] Return to reference Figure 2A and Figure 2CDuring stage S154, the image viewer 51 may, for example, control the display of:

[0138] (1) A 3D volume rendering of the reconstructed medical image 132a and a corresponding salient visual volume image (e.g., as in Figure 4B 3D volume CT image 133b with associated intensity scale 1371 and corresponding 3D salient visual image 137b with associated relevance scale 234 shown in FIG.

[0139] (2) As in Fig. 6A Feature segmentation 338 from the medical image 132a shown in;

[0140] (3) Intelligent reslicing of the reconstructed medical image 132a and the corresponding salient visual volume image following the tilted plane (e.g., as in Figure 6B The re-slice 138a of the 3D volume CT image 132b at the first principal plane and the re-slice 139a of the 3D salient visual image 133b are shown in FIG. Figure 6C The re-slice 138b of the 3D volume CT image 132a at the second principal plane and the re-slice 139b of the 3D salient visual image 133b as shown in FIG. Fig.6D A reslice 138c of the 3D volume CT image 132a and a reslice 139c of the 3D salient visual image 133b at the third principal plane shown in FIG.

[0141] (4) intelligent reslicing in standard orthogonal planes (not shown) (e.g., sagittal, coronal, and axial); and / or

[0142] (5) Manually resliced ​​reconstructed medical images and corresponding salient visual volume images as defined by a user via a user input device (e.g., touch screen, mouse, keyboard, augmented reality glasses, etc.).

[0143] More specifically, the volume rendering is calculated using methods known in the art of the present disclosure (such as, for example, a ray casting algorithm) and is shown compared to the input image. Comparison with the input image can be achieved using side-by-side visualization or image blending. The importance map can be resampled to the size of the input image using common interpolation techniques known in the art of the present disclosure (such as, for example, linear-based interpolation or spline-based interpolation).

[0144] The reslicing plane orientation and position may be defined manually by a user via a GUI or automatically by the salient image generator 53. In one embodiment, the salient image generator 53 may position the reslicing plane in one of the standard axial, sagittal, coronal views, or in the most prominent salient features in the importance map (e.g., as in Fig. 6A The main plane is defined by the center point and the normal. The center point is located at the thermal map 133b ( Figure 4B ) is calculated as the centroid of all 3D points of the salient voxels (p∈R 3 ). The normal vectors to the principal planes are defined by the three eigenvectors with the largest eigenvalues ​​calculated on the most important significant features using a principal component analysis (PCA) method known in the art of the present disclosure. All re-sliced ​​planes can also be shown in a 3D context or as a 2D image compared to a 3D volume CT image 132a also re-sliced ​​at the same orientation.

[0145] More specific Figures 6A-6D , as in Fig. 6A The voxel of the salient feature 338 of the 3D volume CT image 132a shown in FIG is automatically extracted by the medical imaging display engine 50 as one of the features with the highest value in the importance map that is prioritized by the 3D artificial intelligence engine in the decision making process. The voxel of the salient feature 338 is automatically extracted from the importance map using a segmentation method known in the art of the present disclosure (such as, for example, instance intensity-based thresholding or model-based segmentation).

[0146] Figure 6B The diagram shows that the significant feature 338 ( Fig. 6A ) of the 3D volume CT image 132a at a first principal plane defined by a center point and a normal and a reslicing 139a of the 3D salient visual image 133b.

[0147] Figure 6C A reslice 138b of the 3D volume CT image 132a and a reslice 139b of the 3D salient visual image 133b at a second principal plane defined by a center point and a normal to a salient feature 338 are illustrated.

[0148] Fig.6DA reslicing 138c of the 3D volume CT image 132a and a reslicing 139c of the 3D salient visual image 133b at the third principal plane to the salient feature 338 are illustrated.

[0149] The center point of the reslices 139a-139c is located at the centroid calculated from the position of the salient feature 338 within the 3D salient visual image 133b. The normal to the first principal plane is defined by the eigenvector with the second highest eigenvalue calculated from the salient feature 338 using a principal component analysis method known in the art of the present disclosure. The normal to the second principal plane is defined by the eigenvector with the highest eigenvalue calculated from the salient feature 338 using a principal component analysis method known in the art of the present disclosure. The normal to the third principal plane is defined by the eigenvector with the lowest eigenvalue calculated from the salient feature 338 using a principal component analysis method known in the art of the present disclosure.

[0150] In practice, the user may modify the positions of the first principal plane, the second principal plane, and the third principal plane (eg, move the center point along its normal via the GUI).

[0151] Furthermore, in practice, the saliency image generator 53 may control the display of activation maps derived from feature elements evaluated by the AI ​​engine 40. For example, the saliency image generator 53 may display each individual filter output in an activation illustration 433 in the activation map, such as in Figure 7 By choosing a threshold of the total weight (e.g., 90%), the activation maps are ranked by the top n activations associated with those with the highest weights, to thereby list the activation maps in descending order. This embodiment is particularly useful for new clinicians early in their careers or medical students who are not very experienced, and can be a good reference in addition to textbooks.

[0152] Additionally, in practice, 3D salient visual images provide richer information about contours than 2D salient visual images. For example, from a 3D heat map, a user-specified grayscale pixel value visualizes the contour of a region of interest (e.g., a nodule), whereby the contour can be converted into clinical information that would otherwise require human annotation (e.g., area of ​​the volume of interest, maximum radius, axis direction of maximum radius, texture of the contour, etc.), and then a text output is generated. For example, as in Figure 8 As shown in , the clinician reporting aspect of the present disclosure provides automatic linking of generated text information 531 to images 30a, 133b, and 132b.

[0153] In order to promote a further understanding of the present disclosure, Fig. 9A and Fig. 9B The following description teaches various embodiments of the graphical user interface of the present disclosure. Fig. 9A and Fig. 9B With the description provided herein, those skilled in the art will recognize how to apply the present disclosure to many and various additional embodiments of making and using the graphical user interface of the present disclosure.

[0154] refer to Figure 2A and Figure 2B , the graphical user interface 70 is used to perform the previously combined Figure 1 GUI Home tab 71 , Image Mask tab 72 , Data Correction tab 73 , Relevance Threshold tab 74 , and Alternative Evaluation tab 75 of flowchart 170 of the described implementation of the Significant Manipulation stage S26 .

[0155] In this regard, stage S172 of flowchart 170 encompasses activation of GUI 70 for GUI 70 to thereby receive user input data via a user input device (e.g., mouse, keyboard, touch screen, augmented reality glasses, etc.) and optionally auxiliary patient information (e.g., patient age, medical history, etc.). Based on the input, GUI 70 processes through stages S174-S188 of flowchart 170 to thereby interact with AI engine 40 to manipulate the feature evaluation 31 saliency visualization of medical imaging 30 as specified by the user.

[0156] Specifically, during stage S174 of flowchart 170, GUI 70 confirms whether the input data is data mask data 34a received via image mask tag 72, whereby the clinician interacts directly with the planar medical image or volumetric medical image and sees how the artificial intelligence engine 40 responds. For example, the clinician can analyze (one or more) significant images 33 ( Figure 1 ) to confirm whether one or more specific regions of the planar medical image or the volumetric medical image are consistent with the current feature evaluation 31 ( Figure 1 If a clinician suspects that one or more specific regions of a planar medical image or a volumetric medical image are irrelevant to a current feature evaluation 31 of such image, the clinician can mask the irrelevant region(s) and view the salient image(s) to see the effect such masking has on the feature evaluation 31 of the medical imaging 30.

[0157] Fig. 10A An embodiment 72a of the image mask GUI 72 is shown, which is implemented by the AI ​​engine 40 ( Figure 2A) provides clinician specification of image masks to thereby see how the AI ​​engine 40 responds to masks of one or more regions of a 2D medical planar image or a 3D medical volume image. If the clinician suspects that a prominent region of a 2D medical planar image or a 3D medical volume image is irrelevant, the clinician can mask the irrelevant region and see the effect such a mask has on the prediction x or on the hypothesized prediction y. As in Figure 7 As shown in FIG. 4B , the image masking GUI 72a of the present disclosure may simultaneously display medical images 46a and 46b , whereby a clinician may utilize tool icons 273a , 273b , and 274c to mask regions of medical image 272 while using medical image 271 as an original reference.

[0158] If the GUI 70 confirms during stage S174 that the input data is for data masking, then the GUI 70 proceeds to stage S176 of the flowchart 170 to communicate the data mask data 34a to the AI ​​engine 40, whereupon the AI ​​engine 40 executes the flowchart 140 as previously described herein ( Figure 2B ), and the medical imaging display engine 50 performs stage S154 ( Figure 2C ).

[0159] If the GUI 70 confirms during stage S178 that the input data is not for a data mask, the GUI 70 proceeds to stage S178 of the flowchart 170 to confirm whether the input data is data correction data 34b received via the data correction label 73, whereby the clinician can test the effect of each input into the artificial intelligence engine on the feature evaluation 31 of the medical imaging 30 as visualized by the salient image(s) 30. Examples of data corrections include, but are not limited to: (1) enabling / disabling a medical imaging input or a combination of medical imaging inputs to the artificial intelligence engine, (2) increasing / decreasing pixel / voxel intensity values ​​of a medical imaging input or a combination of medical imaging inputs to the artificial intelligence engine, (3) enabling / disabling an auxiliary information input or a combination of auxiliary information inputs to the artificial intelligence engine, or (4) changing an auxiliary information input or a combination of auxiliary information inputs to the artificial intelligence engine.

[0160] Fig. 10BAn embodiment 73a of a data modification GUI 73 is shown that provides a clinician-specified configuration of data inputs to the AI ​​engine 40 to test the impact of each input on the resulting feature evaluation 31 of the 2D / 3D medical imaging 30. In one embodiment, the data manipulation GUI of the present disclosure enables the clinician to turn on and off one or a combination of image elements (e.g., pixels or voxels), as well as to change input values ​​via a slide bar, toggle button, etc. By observing the resulting salient visual image 30, the clinician can visually evaluate the impact of each input on the prediction accuracy, and therefore, eliminate all irrelevant information in the future or understand the importance of each input. More specifically, as shown in Fig. 10B As shown in , the AI ​​engine 40 can include both a primary input image element 370 (e.g., a pixel or voxel) and some auxiliary information (e.g., a nodule radius 371, a nodule sphericity 372, and an output from another machine learning model 373). The primary input can be a 2D image or a 3D image. Auxiliary inputs can also include a collection of words present in image annotations, patient age or medical history, an output or combination of outputs from an image processing or machine learning algorithm, and images from other modalities that are aligned to the primary input image. The data correction GUI 73a of the present disclosure provides corrections to the inputs 370-373, such as, for example, via a corrected 370'-373' having a correction characterized by blackening or whitening the input.

[0161] If the GUI 70 confirms during stage S178 that the input data is for data correction, then the GUI 70 proceeds to stage S180 of the flowchart 170 to communicate the data correction data 34a to the AI ​​engine 40, whereupon the AI ​​engine 40 executes the flowchart 140 ( Figure 2B ), and the medical imaging display engine 50 performs stage S154 ( Figure 2C ).

[0162] If the GUI 70 confirms during stage S178 that the input data is not for data correction, the GUI 70 proceeds to stage S182 of the flowchart 170 to confirm whether the input data is feature correlation threshold data 35a received via the correlation threshold tag 74, whereby the clinician sets the correlation level of the pixels of the planar medical image with the feature evaluation 31 of the planar medical image, or the clinician is able to set the correlation level of the voxels of the volumetric medical image with the feature evaluation 31 of the volumetric medical image. The insight is that not all pixels of a planar medical image or all voxels of a volumetric planar image are equally correlated with the feature evaluation 31, and clinicians typically focus on the (one or more) features of the image that they are most suspicious of. In this way, the correlation level of the pixels of the planar medical image or the voxels of the volumetric planar image can be set by the clinician to a value such that if the feature of the image reaches that correlation level, the feature will be more uniquely highlighted within the (one or more) significant images 33.

[0163] Fig. 10C An embodiment 74a of a correlation threshold GUI 74 is shown, which provides the AI ​​engine 40 with information on the level of correlation of pixels of 2D medical imaging and specific risk predictions / status classifications or voxels of 3D medical imaging and specific risk predictions / status classifications of anatomical regions / organs. Figure 1 ) is specified by a clinician of a configuration of a 2D medical planar image or a 3D medical volume image. The insight is that not all regions of a 2D medical planar image or a 3D medical volume image are equally relevant to a particular diagnostic decision, and clinicians often focus on the regions they are most suspicious of. In one embodiment, the image correlation GUI of the present disclosure controls the display of a 2D reconstructed medical image or a 3D medical volume image, whereby a clinician can set the level of correlation of one or more regions to a value such that if (one or more) of these regions reach that level of correlation, (one or more) regions in the 2D medical planar image or in the 3D medical volume image will be highlighted. For example, as in Fig. 10C As shown in , the image correlation GUI 74a of the present disclosure can display a 2D heat map 44a (or alternatively, a 3D heat map) from a feature evaluation, whereby the clinician can use the interface 475 to modify the correlation level (r) of the feature elements of the AI ​​engine 40 to control the display of the modified 2D heat map 44b (or alternatively, a modified 3D heat map) focused on the area considered most relevant to the feature evaluation. More specifically, as described with Figure 4A and Figure 4BRelatedly, a clinician may illustratively specify that any intensity below 135.2 may be the same color (e.g., blue) and any intensity above 135.2 may be the same color (e.g., red), or a clinician may illustratively specify that any intensity below 67.6 may be the same color (e.g., blue) and all intensities above 135.2 will follow a linear variation in color as shown.

[0164] If the GUI 70 confirms during stage S182 that the input data is for a relevance threshold designation, then the GUI 70 proceeds to stage S184 of the flowchart 170 to communicate the relevance threshold data 34a to the AI ​​engine 40, whereupon the AI ​​engine 40 executes the flowchart 140 as previously described herein ( Figure 2B ) of S144 and S146, and the medical imaging display engine 50 performs stage S154 ( Figure 2C ).

[0165] If the GUI 70 confirms during stage S182 that the input data is not for correlation threshold designation, the GUI 70 advances to stage S186 of the flowchart 170 to confirm whether the input data is alternative feature evaluation data 35b received via the alternative evaluation label 75, whereby the clinician can make assumptions between different predictions or different classifications (e.g., prediction / classification x and prediction / classification y) of the feature evaluation 31 of the medical imaging 30, and if the artificial intelligence engine draws prediction / classification x of the medical imaging 30 as visualized by (one or more) significant images 31 illustrating prediction / classification x of the medical imaging 30, then the clinician can select prediction / classification y via the GUI to see the features that are most correlated with prediction / classification y of the medical imaging 30a as visualized by (one or more) significant images 31 illustrating prediction / classification y of the medical imaging 30.

[0166] Fig. 10D An embodiment 74a of an alternative assessment GUI 75 is shown that provides user selection of alternative feature assessments, whereby if the AI ​​engine 40 draws the initial feature assessments, the clinician can see the areas that are most relevant to the alternative feature assessments. Fig. 10D As shown in , the alternative evaluation GUI 75a displays a planar heat map 575a (or alternatively a 3D heat map) of the initial feature evaluation 131a. The alternative evaluation GUI 75 enables the clinician to select one of the alternative feature evaluations 131b-131d to display a modified 2D heat map 575b illustrating image features associated with the selected alternative feature evaluation. By way of further example, the AI ​​engine 40( Figure 1) can be a multi-category lesion classification based on normal, glioblastoma, sarcoma, and brain metastasis. The initial feature evaluation 131a can be a sarcoma, and the 2D heat map 575b illustrates the sarcoma evaluation, whereby the clinician can select normal (131b), glioblastoma (131c), or brain metastasis (131d) to thereby see image features associated with the selected alternative feature evaluation.

[0167] If the GUI 70 confirms during stage S186 that the input data is for a relevance threshold designation, then the GUI 70 proceeds to stage S188 of the flowchart 170 to communicate the relevance threshold data 34a to the AI ​​engine 40, whereupon the AI ​​engine 40 executes the flowchart 140 as previously described herein ( Figure 2B ) of S144 and S146, and the medical imaging display engine 50 performs stage S154 ( Figure 2C ).

[0168] If the GUI 70 determines during stage S186 that the input data is not for an alternative evaluation designation, the GUI 70 returns to stage S172.

[0169] In order to promote a further understanding of the present disclosure, Fig.11 The following description of FIG. 12 teaches various embodiments of a significant medical imaging controller of the present disclosure, and the following description of FIG. 12 teaches various embodiments of a significant visualization system of the present disclosure. Fig.11 12 , those skilled in the art of the present disclosure will recognize how to apply the present disclosure to many and various additional embodiments of making and using the significant medical imaging controller of the present disclosure.

[0170] In practice, the significant medical imaging controller of the present disclosure may be embodied as hardware / circuitry / software / firmware for implementing the significant medical imaging method of the present disclosure as previously described herein. In addition, in practice, the significant medical imaging controller may be customized and installed in a server, workstation, etc. or programmed on a general-purpose computer.

[0171] exist Fig.11 In one embodiment shown in FIG. 8 , a medical imaging controller 80 includes a processor 81, a memory 82, a user interface 83, a network interface 84, and a storage device 85 interconnected via one or more system buses 86. In practice, the actual organization of the components 81-85 of the controller 80 may be more complex than illustrated.

[0172] Processor 81 can be any hardware device capable of executing instructions stored in a memory or storage device or otherwise processing data. Thus, processor 81 can include a microprocessor, a field programmable gate array (FPGA), an application specific circuit (ASIC) or other similar devices.

[0173] The memory 82 may include various memories, such as, for example, L1, L2 or L3 cache memory or system memory. Thus, the memory 82 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read-only memory (ROM) or other similar memory devices.

[0174] The user interface 83 may include one or more devices for realizing communication with a user (such as an administrator). For example, the user interface 83 may include a display, a mouse, and a keyboard for receiving user commands. In certain embodiments, the user interface 83 may include a command line interface or a graphical user interface that may be presented to a remote terminal via a network interface 84.

[0175] The network interface 84 may include one or more devices for implementing communication with other hardware devices. For example, the network interface 84 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. Additionally, the network interface 84 may implement a TCP / IP stack for communicating with the TCP / IP protocol. Various alternative or additional hardware or configurations for the network interface 84 will be apparent.

[0176] The storage device 85 may include one or more machine-readable storage media, such as a read-only memory (ROM), a random access memory (RAM), a disk storage medium, an optical storage medium, a flash memory device, or a similar storage medium. In various embodiments, the storage device 85 may store instructions for execution by the processor 81 or data that the processor 81 can operate on. For example, the storage device 85 stores a basic operating system (not shown) for controlling various basic operations of the hardware.

[0177] More specifically to the present disclosure, the storage device 85 may store the control module 87 in the form of the AI ​​engine 40a as the AI ​​engine 40( Figure 2A ) embodiment, the medical image display engine 50a is stored as the medical image display engine 50( Figure 2A ) embodiment, and storing (one or more) graphical user interfaces 70a as AI engine 70 ( Figure 8 A) embodiment.

[0178] refer to Fig. 12AIn practice, the significant medical imaging controller 80 can be installed / programmed on an application server 100 that can be accessed by multiple clients (e.g., client 101 and client 102 as shown), and / or installed / programmed within a workstation 103 that employs a monitor 104, a keyboard 105, and a computer 106.

[0179] In operation, the significant medical imaging controller 80 inputs medical imaging data 30 (plane or volume) from the medical imaging data source 90 during the training phase and phase. The medical imaging data source 90 may include any number and type of medical imaging machines (e.g., MRI machine 91, CT machine 93, X-ray machine 95, and ultrasound machine 97 as shown), and may also include a database management / file server (e.g., MRI database management server 92, CT server 94, X-ray database management server 96, and ultrasound database manager server 97 as shown). In practice, the application server 100 or workstation 103 (whichever is applicable) may be directly or network connected to the medical imaging data source 90 to thereby input the medical imaging data 30 for the significant medical imaging controller 80. Alternatively, the medical imaging data source 90 and the application server 100 or workstation 103 (whichever is applicable) may be directly integrated, whereby the significant medical imaging controller 80 has direct access to the medical imaging data 30.

[0180] Fig. 12B An alternative embodiment of a significant medical imaging controller 80 is illustrated, which is divided into an evaluation controller 80a for implementing the AI ​​engine 40a and the GUI 70a and a display controller 80b for the medical imaging display engine 50a. For this embodiment, the evaluation controller 80a and the display controller 80b can be installed / programmed within the application server 100 or the workstation 103, or alternatively can be installed / programmed distributedly between the application server and the workstation 103.

[0181] refer to Figure 1-1 2. Those skilled in the art will recognize the many benefits of the present disclosure, including but not limited to: visualization of (one or more) machine learning models of medical imaging of the body, human or animal by a clinician evaluating the features, the visualization facilitating clear and concise understanding by the clinician evaluating the features.

[0182] Furthermore, it will be apparent that various information described as being stored in a storage device may additionally or alternatively be stored in a memory. In this regard, the memory may also be considered to constitute a "storage device" and the storage device may also be considered to be a "memory". Various other arrangements will be apparent. Furthermore, both the memory and the storage device may be considered to be a "non-transitory machine-readable medium". The term "non-transitory" as used herein will be understood to exclude transient signals but include all forms of storage devices, including both volatile and non-volatile memory.

[0183] Although the device is shown as including one of each described component, various components may be replicated in various embodiments. For example, the processor may include multiple microprocessors configured to independently perform the methods described in the present disclosure, or configured to perform the steps or subroutines of the methods described in the present disclosure, so that the multiple processors cooperate to implement the functions described in the present disclosure. In addition, in the case of implementing the device in a cloud computing system, the various hardware components may belong to separate physical systems. For example, the processor may include a first processor in a first server and a second processor in a second server.

[0184] It is obvious from the foregoing description that various exemplary embodiments of the present invention can be implemented as hardware or firmware. In addition, various exemplary embodiments can be implemented as instructions stored on a machine-readable storage medium, and the instructions can be read and run by at least one processor to perform the operation described in detail herein. The machine-readable storage medium may include any mechanism for storing information in a machine-readable form, such as a personal computer or laptop computer, server or other computing device. Therefore, the machine-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium, an optical storage medium, a flash storage device and similar storage media.

[0185] Those skilled in the art will appreciate that any block diagrams herein represent conceptual views of illustrative circuits that implement the principles of the invention. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocodes, etc. represent various processes that can be substantially represented in a machine-readable medium and thus executed by a computer or processor, regardless of whether such a computer or processor is explicitly shown.

[0186] Although various exemplary embodiments have been described in detail with particular reference to specific exemplary aspects of the present invention, it will be appreciated that the present invention is capable of other embodiments and that its details are capable of modification in various obvious aspects. It will be apparent to those skilled in the art that changes and modifications can be made while remaining within the spirit and scope of the present invention. Therefore, the foregoing disclosure, description and drawings are intended for illustrative purposes only and do not limit the present invention in any way, which is defined solely by the claims.

Claims

1. A significant medical imaging controller (80), comprising: a memory (82) storing an artificial intelligence engine (40) and a graphical user interface (70), wherein the artificial intelligence engine (40) includes at least one machine learning model (42); and A processor (81) in communication with the memory (82), the processor (81) being configured to: applying the at least one machine learning model (42) to medical image data representing at least one feature of a volumetric medical image to render a feature evaluation of the volumetric medical image; and Controlling display of the graphical user interface (70) to provide user interaction with the artificial intelligence engine (40) to manipulate a saliency visualization of the feature evaluations of the volumetric medical image by the at least one machine learning model (42), wherein the saliency visualization comprises reslicing a planar saliency image from the volumetric medical image by the at least one machine learning model based on a level of correlation of each feature of the volumetric medical image with the feature evaluations.

2. The significant medical imaging controller (80) according to claim 1, wherein: The controlling of the display of the graphical user interface (70), thereby providing the user interaction with the artificial intelligence engine (40) to manipulate the saliency visualization of the feature evaluation of the volumetric medical image, comprises: The processor (81) is further configured to control the display of the graphical user interface (70) providing a user-specified correlation threshold of the significant visualization evaluated for each feature of the volumetric medical image with the feature of the volumetric medical image.

3. The significant medical imaging controller (80) according to claim 1, wherein: The controlling of the display of the graphical user interface (70), thereby providing the user interaction with the artificial intelligence engine (40) to manipulate the saliency visualization of the feature evaluation of the volumetric medical image, comprises: The processor (81) is also configured to control the display of the graphical user interface (70) providing a user-specified evaluation of candidate features of the volumetric medical image by the at least one machine learning model (42).

4. The salient medical imaging controller (80) according to claim 1, wherein: The controlling of the display of the graphical user interface (70), thereby providing the user interaction with the artificial intelligence engine (40) to manipulate the saliency visualization of the feature evaluation of the volumetric medical image, comprises: The processor (81) is further configured to control the display of the graphical user interface (70) providing a user specification of a mask for at least one of the features of the volumetric medical image.

5. The salient medical imaging controller (80) of claim 1, wherein: The controlling of the display of the graphical user interface (70), thereby providing the user interaction with the artificial intelligence engine (40) to manipulate the saliency visualization of the feature evaluation of the volumetric medical image, comprises: The processor (81) is further configured to control the display of the graphical user interface (70) providing a user specification of a modification of at least one of the features of the volumetric medical image.

6. The salient medical imaging controller (80) according to claim 1, in, The memory (82) also stores an image display engine; and Wherein, the processor (81) is further configured to: Controlling display of a saliency image generated by the image display engine, the saliency image illustrating the saliency visualization of the feature evaluation of the volumetric medical image.

7. The significant medical imaging controller (80) according to claim 6, wherein: The controlling of the display of the graphical user interface (70), thereby providing the user interaction with the artificial intelligence engine (40) to manipulate the saliency visualization of the feature evaluation of the volumetric medical image, comprises: The processor (81) is further configured to control the display of the graphical user interface (70) including at least one of the salient image and the volumetric medical image.

8. The salient medical imaging controller (80) according to claim 6, wherein: The saliency visualization of the feature evaluation of the volumetric medical image comprises at least one of a heat map, a feature segmentation, and an activation map.

9. A non-transitory machine-readable storage medium encoded with instructions for execution by at least one processor (81), the non-transitory machine-readable storage medium comprising instructions for: applying an artificial intelligence engine (40) including at least one machine learning model (42) to medical image data representing at least one feature of a volumetric medical image to render a feature evaluation of the volumetric medical image; and controlling display of a graphical user interface (70) to provide user interaction with the artificial intelligence engine (40) to manipulate a salient visualization of the feature evaluation of the volumetric medical image by the at least one machine learning model (42), wherein: The salient visualization includes reslicing a planar salient image from the volumetric medical image based on a correlation level of each feature of the volumetric medical image to the feature evaluation by the at least one machine learning model.

10. The non-transitory machine-readable storage medium of claim 9, wherein: The instructions for controlling the display of the graphical user interface (70) to provide the user interaction with the artificial intelligence engine (40) to manipulate the salient visualization of the feature assessment of the volumetric medical image include: Instructions for controlling the display of the graphical user interface (70) providing user-specified correlation thresholds of the significant visualization evaluated for each feature of the volumetric medical image with the feature of the volumetric medical image.

11. The non-transitory machine-readable storage medium of claim 9, wherein: The instructions for controlling the display of the graphical user interface (70) to provide the user interaction with the artificial intelligence engine (40) to manipulate the salient visualization of the feature assessment of the volumetric medical image include: Instructions for controlling the display of the graphical user interface (70) for providing a user-specified evaluation of candidate features of the volumetric medical image by the at least one machine learning model (42).

12. The non-transitory machine-readable storage medium of claim 9, wherein: The instructions for controlling the display of the graphical user interface (70) to provide the user interaction with the artificial intelligence engine (40) to manipulate the salient visualization of the feature assessment of the volumetric medical image include: Instructions for controlling the display of the graphical user interface (70) providing user specification of a mask for at least one of the features of the volumetric medical image.

13. The non-transitory machine-readable storage medium of claim 9, wherein: The instructions for controlling the display of the graphical user interface (70) to provide the user interaction with the artificial intelligence engine (40) to manipulate the salient visualization of the feature assessment of the volumetric medical image include: Instructions for controlling the display of the graphical user interface (70) providing user-specified modification of at least one of the features of the volumetric medical image.

14. The non-transitory machine-readable storage medium of claim 9, wherein: The non-transitory machine-readable storage medium also includes instructions for: Controlling display of a saliency image generated by an image display engine, the saliency image illustrating the saliency visualization of the feature evaluation of the volumetric medical image.

15. A significant medical imaging method comprising: applying an artificial intelligence engine (40) including at least one machine learning model (42) to medical image data representing at least one feature of a volumetric medical image to render a feature evaluation of the volumetric medical image; and Controlling display of a graphical user interface (70) to provide user interaction with the artificial intelligence engine (40) to manipulate a saliency visualization of the feature evaluations of the volumetric medical image by the at least one machine learning model (42), wherein the saliency visualization includes reslicing a planar saliency image from the volumetric medical image by the at least one machine learning model based on a level of correlation of each feature of the volumetric medical image with the feature evaluations.

Images