Medical images for physiological assessment and selection of model for medical image analysis

Abstract

Selection of a predictive model for medical image analysis includes a processor circuit receiving a clinical question and first patient case data, and accessing a plurality of second patient cases. Each second patient case includes second patient case data and a ground truth label. The processor circuit selects a subset of the second patient cases based on the first patient case data and the second patient case data, provides the subset to a first predictive model and a second predictive model, generates first model predictions by the first predictive model and second model predictions by the second predictive model. The processor circuit determines a selected model based on the ground truth labels of the subset, first model predictions, and the second model predictions. The processor circuit generates, by the selected model, a selected model prediction for the first patient case, and outputs the selected model prediction to a display.

Description

[0001] MEDICAL IMAGES FOR PHYSIOLOGICAL ASSESSMENT AND SELECTION OF

[0002] MODEL FOR MEDICAL IMAGE ANALYSIS

[0003] FIELD OF THE INVENTION

[0004] The subject matter described herein relates to physiological assessment of the patient using medical images (e.g., ultrasound, MRI, CT, PET, etc.). For example, a predictive model best suited to analyze medical image for a given patient case (e.g., identify anatomy, segment anatomy, classify disease stage, etc.) is identified by testing multiple predictive models using similar patient cases.

[0005] BACKGROUND OF THE INVENTION

[0006] Increasingly, artificial intelligence and various neural network-based models are being utilized by medical professionals, technicians, and staff in making diagnoses and analyzing patient records. Furthermore, a large and growing number of these models are available or coming available. These models are often tailored to specific use cases, such as image segmentation or diagnostic classification. The models may be trained on data which is no longer known or available. Thus, accurate selection of a model may be challenging. In addition, many models may be too large to be stored locally on a computer or server. Finally, the user of the model, e.g., doctors, often lack the expertise to select a model that will perform well for their use case.

[0007] The information included in this Background section of the specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and is not to be regarded as subject matter by which the scope of the disclosure is to be bound.

[0008] SUMMARY OF THE INVENTION

[0009] Disclosed in an automatic model selection and prediction system for evaluation of medical data (e.g., anatomical images) of a patient. A set of similar patient cases for identifying a well-suited predictive model are selected by comparing an incoming request to a dataset of patient cases and ground truth labels. After the set of similar cases is identified, then potential model may be tested on the set of similar cases. The model’s performance may be generated based on the ground truth labels in the set of similar cases compared to each model’s output. Then a model is selected based on the performances of each model, and a prediction for the incoming request may be generated and provided to a user. For example, a doctor may request for a particular anatomical feature to be detected in an ultrasound image, so the output of the selected model would include a medical image with a box surrounding the anatomical feature.

[0010] This automatic model selection and prediction system disclosed herein has particular, but not exclusive, utility for diagnosing disease, segmenting images, detecting anatomy, etc. The automatic model selection and prediction system selected a subset of a dataset automatically, automatically selects models, generates predictions from the model, and generates metrics representative of model performance. The automatic model selection and prediction system advantageously allows for selection of a best-suited model (if available) for the given clinical task from among many models. This allows even a user (e.g., a physician) who is not an expert in different types of predictive models, to use a well-suited model for assessment of a patient through model-based analysis of the medical image.

[0011] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0012] In one general aspect, the present disclosure is directed to a computer- implemented method. The computer-implemented method also includes receiving, for a first patient case, a clinical question and first patient case data. The computer-implemented method also includes accessing a plurality of second patient cases, where each second patient case that may include second patient case data and a ground truth label. The computer-implemented method also includes selecting a subset of the plurality of second patient cases based on the first patient case data and the second patient case data of the plurality of second patient cases. The computer-implemented method also includes providing the subset to a first predictive model and a second predictive model. The computer-implemented method also includes generating a plurality of first model predictions for the subset by the first predictive model and a plurality of second model predictions for the subset by the second predictive model. The computer- implemented method also includes determining a selected model based on the ground truth labels of the subset, the plurality of first model predictions, and the plurality of second model predictions, where the selected model may include at least one of the first predictive model or the second predictive model. The computer-implemented method also includes providing the clinical question to the selected model. The computer-implemented method also includes generating, by the selected model, a selected model prediction for the first patient case. The computer- implemented method also includes outputting the selected model prediction to a display.

[0013] In some aspects, implementations may include one or more of the following features. The clinical question may include a first medical image and a clinical task associated with the first medical image, and the selected model prediction may include a graphical representation overlaid on the first medical image and associated with the clinical task. The clinical task may include at least one of anatomy detection, anatomy segmentation, disease stage classification. The first predictive model and the second predictive model each may include a pre-trained convolutional neural network. Each second patient case further may include a second medical image, and the ground truth label is associated with the second medical image. The providing the subset to the first predictive model and the second predictive model may include: providing the second medical images of the subset to the first predictive model and the second predictive model. Selecting the subset may include: computing, using a similarity metric, a similarity value for each second patient case based on a comparison between the first patient case data and the second patient case data of the plurality of second patient cases; and identifying the subset based on the similarity value. Determining the selected model may include: computing, using a first plurality of outcome distance metrics, a first plurality of outcome distance values based on a comparison between the plurality of first model predications and the ground truth labels for subset; and computing, using a second plurality of outcome distance metrics, a second plurality of outcome distance values based on a comparison between the plurality of second model predictions and the ground truth labels for subset. Determining the selected model may include: computing, using a first reliability metric for the first predictive model, a first reliability value based on the first plurality of outcome distance values; computing, using a second reliability metric for the second predictive model, a second reliability value based on the second plurality of outcome distance values; and identifying the selected model based on the first reliability value and the second reliability value. Identifying the selected model is further based on the first robustness value and the second robustness value. Computing the first robustness value for the first predictive model is based on a further subset of the plurality of second patient cases, and computing the second robustness value for the second predictive model is based on a further subset of the plurality of second patient cases. The method may include outputting at least one of the first reliability value or the second reliability value to the display. The selected model may include the first predictive model and the second predictive model. The selected model prediction may include the combined output. The plurality of predictive models may include the first predictive model, the second predictive model, and a third predictive model, each of the plurality of predictive models may include model metadata representative of model training, pre-selecting that may include: performing a comparison between the model metadata of the plurality of predictive models and at least one of the clinical question or the first patient case data; and at least one of: identifying the first predictive model and the second predictive model based on the comparison; or eliminating the third predictive model based on the comparison. The filtering may include: performing a comparison between: at least one of the first patient case data or the clinical question; and at least one of the second patient case data or the ground truth labels of the plurality of second patient cases; identifying, based on the comparison, a filtered portion of the plurality of second patient cases, and the subset is selected from the filtered portion of the plurality of second patient cases. The method may include receiving user feedback representative of whether the selected model prediction is responsive to the clinical question; and updating, based on the user feedback, how the selected model is determined.

[0014] In one general aspect, the present disclosure is directed to a computer- implemented method. The computer-implemented method also includes receiving, for a first patient case, a clinical question and first patient case data; performing a comparison between the clinical question and a plurality of previous clinical questions; computing, using a similarity metric, similarity values for each of the plurality of previous clinical questions based on a comparison between the clinical question and the plurality of previous clinical questions. The computer-implemented method also includes determining a selected predictive model. The computer-implemented method also includes providing the clinical question to the selected model. The computer-implemented method also includes generating, by the selected model, a selected model prediction for the first patient case; and outputting the selected model prediction to a display. When the similarity values for the plurality of previous clinical questions does not satisfy a threshold, the computer-implemented method includes determining a selected model that may include: accessing a plurality of second patient cases, where each second patient case that may include second patient case data and a ground truth label; selecting a subset of the plurality of second patient cases based on the first patient case data and the second patient case data of the plurality of second patient cases; providing the subset to a first predictive model and a second predictive model; generating a plurality of first model predictions for the subset by the first predictive model and a plurality of second model predictions for the subset by the second predictive model, selecting at least one of at least one of the first predictive model or the second predictive model as the selected predictive model based on the ground truth labels of the subset, the plurality of first model predictions, and the plurality of second model predictions. When the similarity value for a previous clinical question of the plurality of clinical questions satisfies the threshold, the computer-implemented method includes determining the selected model that may include: selecting a predictive model previously determined for the previous clinical question as the selected model.

[0015] In one general aspect, the present disclosure is directed to a system including a processor circuit configured to: receive, for a first patient case, a clinical question and first patient case data; access a plurality of second patient cases, where each second patient case that may include second patient case data and a ground truth label; select a subset of the plurality of second patient cases based on the first patient case data and the second patient case data of the plurality of second patient cases; provide the subset to a first predictive model and a second predictive model; generate a plurality of first model predictions for the subset by the first predictive model and a plurality of second model predictions for the subset by the second predictive model; determine a selected model based on the ground truth labels of the subset, the plurality of first model predictions, and the plurality of second model predictions, where the selected model may include at least one of the first predictive model or the second predictive model; provide the clinical question to the selected model; generate, by the selected model, a selected model prediction for the first patient case; and output the selected model prediction to a display.

[0016] In one general aspect, the present disclosure is directed to a non-transitory machine-readable medium that may include a plurality of machine-executable instructions. The non-transitory machine-readable medium includes instructions for receiving, for a first patient case, a clinical question and first patient case data. The non-transitory machine-readable medium also includes instructions for accessing a plurality of second patient cases, where each second patient case that may include second patient case data and a ground truth label. The non- transitory machine-readable medium also includes instructions for selecting a subset of the plurality of second patient cases based on the first patient case data and the second patient case data of the plurality of second patient cases. The non-transitory machine-readable medium also includes instructions for providing the subset to a first predictive model and a second predictive model. The non-transitory machine-readable medium also includes instructions for generating a plurality of first model predictions for the subset by the first predictive model and a plurality of second model predictions for the subset by the second predictive model. The non-transitory machine-readable medium also includes instructions for determining a selected model based on the ground truth labels of the subset, the plurality of first model predictions, and the plurality of second model predictions, where the selected model may include at least one of the first predictive model or the second predictive model. The non-transitory machine-readable medium also includes instructions for providing the clinical question to the selected model. The non- transitory machine-readable medium also includes instructions for generating, by the selected model, a selected model prediction for the first patient case. The non-transitory machine- readable medium also includes instructions for outputting the selected model prediction to a display.

[0017] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. A more extensive presentation of features, details, utilities, and advantages of the automatic measurement point detection system, as defined in the claims, is provided in the following written description of various aspects of the disclosure and illustrated in the accompanying drawings.

[0018] BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Illustrative aspects of the present disclosure will be described with reference to the accompanying drawings, of which:

[0020] Fig. l is a schematic diagram of a networked system for model selection and prediction, according to aspects of the present disclosure.

[0021] Fig. 2 is a schematic diagram of a processor circuit, according to aspects of the present disclosure.

[0022] Fig. 3 is a schematic diagram of a deep learning network configuration, according to aspects of the present disclosure.

[0023] Fig. 4A is an example ultrasound image of a kidney, according to aspects of the present disclosure.

[0024] Fig. 4B is an example ultrasound image of a kidney with object detection box, according to aspects of the present disclosure.

[0025] Fig. 5 is a schematic diagram of at least a portion of a model selection and prediction system, according to aspects of the present disclosure.

[0026] Fig. 6A is a first schematic sequence diagram of a process for model selection of one or more models, according to aspects of the present disclosure. Fig. 6B is a second schematic sequence diagram of a process for model prediction and update, according to aspects of the present disclosure.

[0027] Fig. 7 is a schematic diagram of an incoming request, according to aspects of the present disclosure.

[0028] Fig. 8 is a schematic diagram of a patient case and a second patient case, according to aspects of the present disclosure.

[0029] Fig. 9 is a first schematic graph of patient cases in a ground truth dataset, according to aspects of the present disclosure.

[0030] Fig. 10 is a first schematic diagram of generating model metric values from patient cases, according to aspects of the present disclosure.

[0031] Fig. 11 is a second schematic diagram of generating model metric values from patient cases, according to aspects of the present disclosure.

[0032] Fig. 12 is a second schematic graph of patient cases in a ground truth dataset, according to aspects of the present disclosure.

[0033] Fig. 13 is a third schematic graph of patient cases in a ground truth dataset, according to aspects of the present disclosure.

[0034] Fig. 14 is a schematic diagram of at least a portion of a model selection and prediction system, according to aspects of the present disclosure.

[0035] Fig. 15 is a schematic flow diagram of a method for model selection and prediction, according to aspects of the present disclosure.

[0036] Fig. 16 is a schematic diagram of a model selection and prediction system with pre-selected models, according to aspects of the present disclosure.

[0037] Fig. 17 is a schematic flow diagram of a method for model selection and prediction with pre-selected models, according to aspects of the present disclosure.

[0038] Fig. 18 is a schematic diagram of a model selection and prediction system with question comparison, according to aspects of the present disclosure.

[0039] Fig. 19 is a schematic flow diagram of a method for model selection and prediction with question comparison, according to aspects of the present disclosure.

[0040] Fig. 20 is an example display including selected model and model result, according to aspects of the present disclosure. DETAILED DESCRIPTION OF EMBODIMENTS

[0041] In accordance with at least one aspect of the present disclosure, a model selection and prediction system is provided that automatically selects a model and generates a prediction in response to a user request. This may allow, for example, minimally trained users (including general practitioners, paramedics, and even patients) to use models whose performance they would otherwise lack the expertise to judge due to either or both lack of model transparency or requiring subject matter expertise in non-medical fields, such as machine learning, data, science, etc. Furthermore, for multiple models capable of completing a task, the system provides an automatic way of selecting between the models based on their performance on patient cases similar to the case in the user request.

[0042] These descriptions are provided for exemplary purposes only, and should not be considered to limit the scope of the model selection and prediction system. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.

[0043] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the aspects illustrated in the drawings, and specific language will be used to describe the same. It is nevertheless understood that no limitation to the scope of the disclosure is intended. Any alterations and further modifications to the described devices, systems, and methods, and any further application of the principles of the present disclosure are fully contemplated and included within the present disclosure as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one aspect may be combined with the features, components, and / or steps described with respect to other aspects of the present disclosure. For the sake of brevity, however, the numerous iterations of these combinations will not be described separately. As used herein, accessing can include querying, retrieving, sorting, etc.

[0044] The systems and methods disclosed herein provide a number of benefits. Some Al image analysis models are narrowly tailored so that they can be reliable for a given task. For example, Al image models can be specific to an age group, a particular body part (e.g., an organ), a particular disease, etc. Thus, there are a large number of models available with new models coming online regularly. No medical professional can be aware of all available models, especially for a general practitioner covering a large number of medical areas, so the systems and methods described herein allow medical professionals access to a wider body of models than they would otherwise know to use. Al in general and Al image analysis in particular continue to grow in popularity do to their increasing domains of applicability and sophistication. Physicians are becoming aware of Al-based image analysis, and seek to use Al-based image analysis to help their patients. But physicians are experts in the physiological subject matter areas, not Al models. While physicians may know the clinical question they want answered, they do not know how to efficiently and effectively choose models (from among many models) so that their clinical question can be reliably answered with Al image processing. Thus, the system and methods disclosed herein remove the need for a physician to choose between models, facilitating improved medical care of their patients.

[0045] Wanting to use an Al model, physicians must choose a model that is relevant to their clinical question. In some instances, there may be one or more Al models that support answering the clinical question, but they may not have been specifically designed to answer that type of clinical question. For example, a physician may want to do tumor staging in a 50-year- old alcoholic’s liver, but there may be no models that are specifically trained for that. The present disclosure helps the doctor identify a model that works sufficiently well and calculates and outputs a reliability value that numerically represents what sufficiently well means. For example, the systems and methods disclosed herein can determined that a model trained on people 30- to 40-years-old would work, a model trained on pancreas images would work, or a model trained on nonalcoholic liver disease may work sufficiently well.

[0046] Fig. 1 is a schematic diagram of a networked system 100 for model selection and prediction, according to aspects of the present disclosure. The networked system 100 may for example may be used to receive clinical questions from a user. A clinical question may be transmitted to different components of the network system 100 to facilitate model selection and predictions. The networked system 100 may provide a user with a prediction in the form of a diagnosis which was output from a selected model.

[0047] The networked system 100 is used for selecting one or more models and making predictions with the selected models. The networked system 100 may include a network / cloud computers 110,140, user computers 160, medical imaging console 170, and medical imaging device 180. The network / cloud computers 110, 140 may be in communication with each, each sending and / or receiving data and information from the other. User computer 160 is depicted in Fig. 1 as communication with network / cloud computer 110. However, in some instances, user computer 160 may be in communication with either or both of network / cloud computers 110, 140. User computer 160 may be in communication with a medical imaging console 170 which is in communication with a medical imaging device 180. As described herein, communication between the different components may be accomplished by any numbers of connections, e.g., wired and / or wireless.

[0048] Network / cloud computer 110 may include a processor 112, input device 114, display 116, communication interface 117, and memory 118.

[0049] Processor 112 may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 112 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 112 is configured to process the instructions stored in memory 118. The processor 112 is connected to the communication interface 117.

[0050] Input device 114 allows a user to make selections or provide instructions to the network / cloud computer 110. The input device, may be a mouse, touch screen, touch pad etc.

[0051] The display 116 is coupled to the processor 112. The display 116 may be a monitor or any suitable display. The display 116 is configured to display model A 124 output, model B 128 output, model metadata 126,130, network performance, or any other system diagnostic information.

[0052] The communication interface 117 is coupled to the processor 112. The communication interface 117 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 117 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 117 can be referred to as a communication device or a communication interface module.

[0053] The memory 118 is coupled to the processor 112. The memory 118 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 112), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and nonvolatile memory, or a combination of different types of memory. The memory 118 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to ultrasound images, ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical scan window, a probe orientation, and / or the subject position during an imaging procedure. The memory 118 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting / segmenting anatomy, image quantification algorithms, and / or image acquisition guidance algorithms, including those described herein.

[0054] In some instances, memory 118 includes data and stored instructions for modules, including model selector 120, ground truth dataset 122, Al model A 124, Al model B 128, a reference location to Al Model C 132. Model selector 120 may select similar cases to a request from a user and select a model to generate a prediction for a user request.

[0055] The predictive models 124, 128, 152 may be any neural network-based model or rule-based programming model, e.g., as depicted in Fig. 3. For example, models 124, 128, 152 can be Convolutional Neural Networks (CNN), decision tree models, support vector machines (SVM), generative image-to-image transformers, generative image-to-text transformers (GIT), or other transformer-based models. A 124 may include model metadata 126. Al model B may include model metadata 130. A reference location to model C 132 details the location of Al model C, located on network / user computer 140. The metadata may include the data format type for inputs and outputs to a model, description of task model was trained on, etc. Ground truth dataset 122 includes a plurality of patient case data and information. Furthermore, the ground truth dataset is not the same as a training dataset 195. Training dataset 195 includes the data that was used to train each of the models 126, 130, and model C 150 on network / cloud computer 140. For example, training dataset may include pairs of anatomical features with segmented images identifying the anatomical features. Ground truth dataset includes annotation and ground truth labels 190 for the patient case data and information. For example, ground truth labels may be disease stage classification, objection detection boxes for anatomy identification detection, anatomy contours for anatomy segmentation, etc. Network / cloud computer 140 may include a processor 142, input device 144, display 146, communication interface 147, and memory 148.

[0056] Processor 142 may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 142 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 142 is configured to process the instructions stored in memory 148. The processor 142 is connected to the communication interface 147.

[0057] Input device 144 allows a user to make selections or provide instructions to the network / cloud computer 140. The input device, may be a mouse, touch-screen, touch pad etc.

[0058] The display 146 is coupled to the processor 142. The display 146 may be a monitor or any suitable display. The display 146 is configured to display model C 150 output, model metadata 152 network performance, or any other system diagnostic information.

[0059] The communication interface 147 is coupled to the processor 142. The communication interface 147 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 147 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 147 can be referred to as a communication device or a communication interface module.

[0060] The memory 148 is coupled to the processor 142. The memory 148 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 142), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and nonvolatile memory, or a combination of different types of memory.

[0061] The memory 148 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to ultrasound images, ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical scan window, a probe orientation, and / or the subject position during an imaging procedure. The memory 148 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting anatomy, segmenting anatomy (e.g., defining contours of the anatomy), image quantification algorithms, and / or image acquisition guidance algorithms, including those described herein.

[0062] In some instances, memory 148 includes data and stored instructions for modules, including model selector Al model C 132 and associated model C metadata 152. As depicted in Fig. 1, model C 132 is located on a separate network / cloud computer from models A and B 124, 128. During model selection and prediction user requests and patient cases from the ground truth dataset 122 may be sent to the network / cloud computer 140. Model C 150 may generate output and predictions for the received data which may then be sent back to one or more of the computers and devices of networked system 100.

[0063] User computer 160 may include a processor 162, input device 164, display 166, communication interface 167, and memory 168.

[0064] Processor 162 may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 162 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 162 is configured to process the instructions stored in memory 168. The processor 162 is connected to the communication interface 167.

[0065] Input device 164 allows a user to make selections or provide instructions to the user computer 160. The input device, may be a mouse, touch-screen, touch pad etc.

[0066] The display 166 is coupled to the processor 162. The display 166 may be a monitor or any suitable display. The display 166 is configured to display model A 124 output, model B 128 output, model C 150 output, model metrics and information as described herein. Furthermore, the display 166 may be configured to display the images from the data generated by medical imaging device 180.

[0067] The communication interface 167 is coupled to the processor 162. The communication interface 167 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 167 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 167 can be referred to as a communication device or a communication interface module.

[0068] The memory 168 is coupled to the processor 162. The memory 168 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 162), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and nonvolatile memory, or a combination of different types of memory.

[0069] The memory 168 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to ultrasound images, ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical scan window, a probe orientation, and / or the subject position during an imaging procedure. The memory 168 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting anatomy, segmenting anatomy, image quantification algorithms, and / or image acquisition guidance algorithms, including those described herein.

[0070] Medical imaging console 170 may include a processor 172, input device 174, display 176, communication interface 177, and memory 178

[0071] Processor 172 may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 172 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 172 is configured to process the instructions stored in memory 178. The processor 172 is connected to the communication interface 177.

[0072] Input device 174 allows a user to make selections or provide instructions to the user computer 170. The input device, may be a mouse, touch-screen, touch pad etc.

[0073] The display 176 is coupled to the processor 172. The display 176 may be a monitor or any suitable display. The display 176 is configured to display model A 124 output, model B 128 output, model C 150 output, model metrics and information as described herein. Furthermore, the display 176 may be configured to display the images from the data generated by medical imaging device 180.

[0074] The communication interface 177 is coupled to the processor 172. The communication interface 177 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 177 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 177 can be referred to as a communication device or a communication interface module.

[0075] The memory 178 is coupled to the processor 172. The memory 178 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 172), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and nonvolatile memory, or a combination of different types of memory.

[0076] The memory 178 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to ultrasound images, ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical scan window, a probe orientation, and / or the subject position during an imaging procedure. The memory 178 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting anatomy, segmenting anatomy, image quantification algorithms, and / or image acquisition guidance algorithms, including those described herein.

[0077] Medical imaging device 180 may generate medical images 185. The medical imaging device is in communication with medical imaging console 170, e.g., through the communication interface 177. The medical images generated by data from the medical imaging device 180 may be shown on the display 176. Medical images 185 may also be communicated to any of the computers, networks / clouds 110, 140, 160.

[0078] In some aspects, aspects of the present disclosure can be implemented with medical images 185 of subjects obtained using any suitable medical imaging device 180 and / or modality. Examples of medical images and medical imaging devices include x-ray images (angiographic images, fluoroscopic images, images with or without contrast) obtained by a medical imaging device such as an ultrasound imaging device, X-ray imaging device, computed tomography (CT) images obtained by a CT imaging device, positron emission tomography- computed tomography (PET-CT) images obtained by a PET-CT imaging device, magnetic resonance images (MRI) obtained by an MRI imaging device, single-photon emission computed tomography (SPECT) images obtained by a SPECT imaging device, optical coherence tomography (OCT) images obtained by an OCT imaging device, and intravascular photoacoustic (IVPA) images obtained by an IVPA imaging device. The medical imaging device 180 can obtain the medical images while positioned outside the subject body, spaced from the subject body, adjacent to the subject body, in contact with the subject body, and / or inside the subject body.

[0079] Before continuing, it should be noted that the examples described above are provided for purposes of illustration, and are not intended to be limiting. Other devices and / or device configurations may be utilized to carry out the operations described herein.

[0080] Fig. 2 is a schematic diagram of a processor circuit 250, according to aspects of the present disclosure. The processor circuit 250 may be implemented in the network / cloud computer 110, 140, user computer 160, imaging console 170, or other devices or workstations (e.g., third-party workstations, network routers, etc.), or on a cloud processor or other remote processing unit, as necessary to implement the method. As shown, the processor circuit 250 may include a processor 260, a memory 264, and a communication interface 268. These elements may be in direct or indirect communication with each other, for example via one or more buses.

[0081] The processor 260 may include a central processing unit (CPU), a digital signal processor (DSP), an ASIC, a controller, or any combination of general-purpose computing devices, reduced instruction set computing (RISC) devices, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other related logic devices, including mechanical and quantum computers. The processor 260 may also comprise another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 260 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0082] The memory 264 may include a cache memory (e.g., a cache memory of the processor 260), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, other forms of volatile and non-volatile memory, or a combination of different types of memory. In an aspect, the memory 264 includes a non-transitory computer-readable medium. The memory 264 may store instructions 266. The instructions 266 may include instructions that, when executed by the processor 260, cause the processor 260 to perform the operations described herein. Instructions 266 may also be referred to as code. The terms “instructions” and “code” should be interpreted broadly to include any type of computer-readable statement(s). For example, the terms “instructions” and “code” may refer to one or more programs, routines, sub-routines, functions, procedures, etc. “Instructions” and “code” may include a single computer-readable statement or many computer-readable statements.

[0083] The communication interface 268 can include any electronic circuitry and / or logic circuitry to facilitate direct or indirect communication of data between the processor circuit 250, and other processors or devices. In that regard, the communication interface 268 can be an input / output (VO) device. In some instances, the communication interface 268 facilitates direct or indirect communication between various elements of the processor circuit 250 and / or computers of networked system 100. The communication interface 268 may communicate within the processor circuit 250 through numerous methods or protocols. Serial communication protocols may include but are not limited to United States Serial Protocol Interface (US SPI), Inter-Integrated Circuit (I2C), Recommended Standard 232 (RS-232), RS-485, Controller Area Network (CAN), Ethernet, Aeronautical Radio, Incorporated 429 (ARINC 429), MODBUS, Military Standard 1553 (MIL-STD-1553), or any other suitable method or protocol. Parallel protocols include but are not limited to Industry Standard Architecture (ISA), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI), Institute of Electrical and Electronics Engineers 488 (IEEE-488), IEEE-1284, and other suitable protocols. Where appropriate, serial and parallel communications may be bridged by a Universal Asynchronous Receiver Transmitter (UART), Universal Synchronous Receiver Transmitter (USART), or other appropriate subsystems.

[0084] External communication (including but not limited to software updates, firmware updates, model sharing between the processor and central server, or outputs generated by the model selection and prediction system described herein) may be accomplished using any suitable wireless or wired communication technology, such as a cable interface such as a universal serial bus (USB), micro USB, Lightning, or FireWire interface, Bluetooth, Wi-Fi, ZigBee, Li-Fi, or cellular data connections such as 2G / GSM (global system for mobiles) , 3G / UMTS (universal mobile telecommunications system), 4G, long term evolution (LTE), WiMax, or 5G. For example, a Bluetooth Low Energy (BLE) radio can be used to establish connectivity with a cloud service, for transmission of data, and for receipt of software patches. The controller may be configured to communicate with a remote server, or a local device such as a laptop, tablet, or handheld device, or may include a display capable of showing status variables and other information. Information may also be transferred on physical media such as a USB flash drive or memory stick.

[0085] Fig. 3 is a schematic diagram of a deep learning network configuration, according to aspects of the present disclosure. The configuration 300 can be implemented by a deep learning network. The configuration 300 includes a deep learning network 310 including one or more CNNs 312. The CNN 312 is one example of a type of predictive model, such as Al model A 124, Al model B 128, and Al model C 150. For simplicity of illustration and discussion, Fig. 3 illustrates one CNN 312. However, the embodiments can be scaled to include any suitable number of CNNs 312 (e.g., about 2, 3 or more). The configuration 300 can be trained for identification of various anatomy (organs, tissue, bone) and / or other features (natural and / or man-made) within a patient anatomy. The configuration 300 can be further trained for segmenting human anatomy, diagnosis medical conditions or any number of other diagnostic or medical tasks. The CNN 312 may include a set of N convolutional layers 320 followed by a set of K fully connected layers 330, where N and K may be any positive integers. The convolutional layers 320 are shown as 320(1) to 320(N). The fully connected layers 330 are shown as 330(1) to 330(K). Each convolutional layer 320 may include a set of filters 322 configured to extract features from an input 302 (e.g., x-ray venogram images or other additional data). The values N and K and the size of the filters 322 may vary depending on the embodiments. In some instances, the convolutional layers 320(1) to 320(N) and the fully connected layers 330(1) to 330(K-l) may utilize a leaky rectified non-linear (ReLU) activation function and / or batch normalization. The fully connected layers 330 may be non-linear and may gradually shrink the high-dimensional output to a dimension of the prediction result 340 (e.g., location for an object detection box or other the classification output). The fully connected layers 330 may also be referred to as a classifier. In some embodiments, the fully convolutional layers 320 may additionally be referred to as perception or perceptive layers.

[0086] When the prediction result 340 takes the form of classification output, it may indicate a confidence score for each class 342 based on the input image 302. The classes 342 are shown as 342a, 342b, . . . , 342c. For example, when the CNN 312 is trained for regions of stenosis or general venous compression, the classes 342 may indicate an inguinal ligament class 342a, a crossover class 342b, a pelvic bone notch class 342c, a region of blood flow restriction class 342d, or any other suitable class. A class 342 indicating a high confidence score indicates that the input image 302 or a section or pixel of the image 302 is likely to include an anatomical object / feature of the class 342. Conversely, a class 342 indicating a low confidence score indicates that the input image 302 or a section or pixel of the image 302 is unlikely to include an anatomical object / feature of the class 342.

[0087] The deep learning network 310 may implement or include any suitable type of learning network. For example, in some embodiments and as described in relation to Fig. 3, the deep learning network 310 could include a convolutional neural network 312. In addition, the convolutional neural network 310 may additionally or alternatively be or include a multi-class classification network, an encoder-decoder type network, or any suitable network or means of identifying features within an image.

[0088] In an embodiment in which the deep learning network 310 includes an encoderdecoder network, the network may include two paths. One path may be a constricting path, in which a large image, such as the image 302, may be convolved by several convolutional layers 320 such that the size of the image 302 changes in relation to the depth of the network layer. The image 302 may then be represented in a low dimensional space, or a flattened space. From this flattened space, an additional path may expand the flattened space to the original size of the image 302. In some embodiments, the encoder-decoder network implemented may also be referred to as a principal component analysis (PCA) method. In some embodiments, the encoderdecoder network may segment the image 302 into patches. In an additional embodiment of the present disclosure, the deep learning network 310 may include a multi-class classification network. In such an embodiment, the multi-class classification network may include an encoder path. For example, the image 302 may be of a high dimensional image. The image 302 may then be processed with the convolutional layers 320 such that the size is reduced. The resulting low dimensional representation of the image 302 may be used to generate the feature vector 350 shown in Fig. 3. The low dimensional representation of the image 302 may additionally be used by the fully connected layers 330 to regress and output one or more classes 342. In some regards, the fully connected layers 330 may process the output of the encoder or convolutional layers 320. The fully connected layers 330 may additionally be referred to as task layers or regression layers, among other terms.

[0089] Any suitable combination or variations of the deep learning network 310 described is fully contemplated. For example, the deep learning network may include fully convolutional networks or layers or fully connected networks or layers or a combination of the two. In addition, the deep learning network may include a multi-class classification network, an encoder-decoder network, or a combination of the two.

[0090] Fig. 4A is an example ultrasound image 400 of a kidney, according to aspects of the present disclose. As shown, ultrasound image 400 depicts human anatomy including a kidney and neighboring tissue. Image 400 may be an example of an image captured by a user operating a medical imaging device 180 (e.g., an ultrasound probe). A user might provide image 500 to the model selection and prediction systems described herein with a request for anatomy detection of the image. Image 400 may be included in a ground truth dataset 122 or a training dataset 195, and, in some instances, the image may only be contained in one of the two datasets.

[0091] Fig. 4B is an example ultrasound image 450 of a kidney with object detection box, according to aspects of the present disclosure. As shown, ultrasound image 450 depicts human anatomy including a kidney and neighboring tissue. In addition, ultrasound image 450 includes an object detection box 460. The object detection box 460 may be generated by an objection detection model (e.g., one of Al Model A, B, or C in Fig. 1) from an image, e.g., image 400 of Fig. 4A. The object detection box 460 surrounds anatomy, e.g., a kidney in image 450, which may be specified by a user for location and / or identification, or a user may ask an objection detection model to identify objects in the image 400. In some instances, various characteristics 470 are associated with the objection detection box 460. For example, characteristics 470 may include the object detected by the objection detection model, the objection detection model’s confidence in the generated detection, an objection detection box area, or the location of the objection detection box (e.g., given by a coordinate for the centroid of the segmentation box). In some instances, the characteristics 470 may be output to a user and displayed (e.g., on display 166 of user computer 160 in Fig. 1) with the segmented image 450.

[0092] Fig. 5 is a schematic diagram of at least a portion of a model selection and prediction system 500, according to aspects of the present disclosure. Model selection and prediction system 500 selects a model based on various conditions and metrics and then uses the selected model to make a prediction given user input. User input may take the form of medical images and / or clinical task and the output may be a response to the clinical task. A user may be any of a number of medical professionals. Alternatively, user input may not require a user to generate and send the input, programmed instructions may automatically generate a user input based on a received image. The model selection and prediction system 500 may include model selector 120, ground truth dataset 122, model A 124, model B 128, and model C 150.

[0093] The model selection and prediction system 500 may receive an incoming request 504 from a user computer 160 or medical imaging console 170. Incoming request 504 may comprise a clinical question 506 and first patient case data 508. In some instances, a clinical question may include a medical image and / or a clinical task, further described with respect to Fig. 7.

[0094] Model selector 120 may include a similar case selection module 510 and a model metrics module 530. The similar case selection module 510 identifies a set of cases that are similar, according to one or more metrics, to the received information contained in the incoming request 504. The model metric module 530 identifies one or more models for selection according to one or more metrics, user selections, model metadata, etc.

[0095] The similar case selection module 510 may include a similarity metric 512 and weights 514. The similarity metric 512 measures how close one patient case is to another. The similarity metric 512 may depend on a patient’s age, weight, height, and other characteristics. The weights 514 may be parameters used in the similarity metric to generate a numerical value for the similarity metric 510 given two or more cases. For example, a similarity metric may be defined by a function, S, represented by a mathematical expression: where oq may represent the weights 514, x£may present case parameters for patient cases in the ground truth dataset 122, and x0;£represent the case parameters for the incoming request 504. A case similarity metric may be defined as the distance of two points in a multi-dimensional space, where the dimensions represent different aspects of the case as numeric values or binary (one-hot) encodings (e.g., patient age, type of disease, body part, clinical question, disease progression time, etc.) and each point represents one patient case.

[0096] The model metrics module 530 may include an outcome distance metric 532, a reliability metric 534, a robustness metric 536, and weights 538. The outcome distance metric 532 determines a distance between a ground truth label of the cases in the subset of similar cases and the label contained in the model outputs 525 generated by the models 124, 128, 150. The reliability metric 534 measures how reliably each model based on its performance on generating output for subset of similar cases 520 that match ground truth labels (i.e., 190). The robustness metric 536 measures how robust the performance of models across different subsets of ground truth data. In other words, the robustness metric indicates the sensitivity of a model to inputs which are close in the sense described herein with respect to the similarity metric. The weights are used in the outcome distance metric 532, reliability metric 534, and robustness metric 536 as parameters to generate numerical values for the metrics given the model outputs and ground truth labels.

[0097] The similar case selection module 510 may receive the incoming request 504 and generate a subset of similar cases 520 in the ground truth dataset 122 that are similar to the patent case described in the incoming request 504. For example, similar case selection module 510 may select all of those case in the ground truth dataset 122 which have similarity metric values less than a maximum value. The maximum value may be pre-programmed into the module or determined directly or indirectly from the incoming request 504.

[0098] Each of model A 124, model B 128, and model C 150 may receive the cases in the subset of similar cases 520 and generate model outputs 525. For example, model outputs 525 may take the form of segmentations for images. Model outputs 525 may take any form depending on the clinical task the model is tasked with, such as diagnosis, segmentation, classification tasks, etc. The model selection and prediction system 500 may include varying numbers of models. For example, selection may be between two models, or it may between four to twenty models, and / or other values both larger and smaller. The model metrics module 530 may receive the model outputs 525 and generate selected model output 540, reliability value 545, and / or robustness value 550. The model metrics module 530 may select a model, e.g., one or more of model A 124, model B 128, and model C 150. The model metric module may also generate, using the selected model, the selected model output 540 based on the incoming request. The model metrics 530 module may generate reliability values 545 using the reliability metric 534 and the weights 538. The model metrics module 530 may generate a robustness value 550 using the robustness metric 534 and the weights 538.

[0099] User computer 160 or medical imaging console 170 may receive the selected model output 540, reliability value 545, and / or robustness value 550. The user computer 160 or medical imaging console 170 may display and / or post-process the information from the model selection and prediction system 500. A user, such as a doctor, interacting with the display may evaluate the results of the model selection and prediction system generated from the incoming request 504. The user may provide user feedback 560 to the model selection and prediction system 500, in some instances, to the model selector 120, that evaluate the performance of the system. User feedback 560 may also include parameter or other selections to modify, directly or indirectly, the weights 514, 538 or metrics 512, 532, 534, or 536 in the model selector 120. A user may act iteratively with the model selection and prediction system 500 to arrive at a desired clinical result. In addition, model may be finetuned with user feedback on performance.

[0100] Fig. 6A and 6B are schematic sequence diagrams of a process for model selection and prediction system 600, according to aspects of the present disclosure. The model selection and prediction system can include a user computer 160 or medical imaging console 170, a model selector 120, a ground truth dataset 122, a model A 124, and a model B 128.

[0101] At step 602, the user computer 160 or the medical imaging device 170 may send a clinical question and first patient case data to model selector 120.

[0102] At step 604, the model selector 120 may access ground truth dataset 122.

[0103] At step 606, second patient case data for second patient cases may be received by the model selector 120 from the ground truth dataset.

[0104] At step 608, the model selector 120 may determine if the second patient case is part of a subset of similar cases 520 to the clinical question and first patient data received by the model selector 120 at step 602.

[0105] At step 610, the subset of cases in the ground truth dataset 122 are determined. For example, subset of similar cases 520 in Fig. 5.

[0106] At step 612, the subset of similar cases 520 is received by model A 124 as input. At step 614, the subset of similar cases 520 is received by model B 128 as input.

[0107] At step 616, the ground truth labels for the subset of similar cases are retrieved by the model selector 120 from the ground truth dataset 122.

[0108] At step 618, model A 124 outputs its predictions to model selector 120.

[0109] At step 620, model B 128 outputs its predictions to model selector 120.

[0110] At step 622, model selector 120 determines model metric values. For example, the model metric values may be determined from the predictions of model’s A and B, as well as the ground truth labels for the subset of similar cases using the outcome distance metric 532, reliability metric 534, robustness metric 536, and weights 538.

[0111] At step 624, model selector 624 determines selected model using model metric values. For example, model A 124 may have the highest reliability and robustness values and be selected on that basis.

[0112] At step 626, model selector 120 provides clinical question to selected model, model A 124. Selection of model A is merely an example, one or more other models, including models not shown in Fig. 6B, may be selected.

[0113] At step 628, selected model, model A 124, outputs a prediction based on the clinical question to model selector 120. For example, model A 124 may generate an image with an anatomy detection, e.g., the image in Fig. 4B.

[0114] At step 630, model selector 120 transmits the selected model prediction and model metric value(s) for the selected model to a user computer 160 or medical imaging device 170.

[0115] At step 632, user computer 160 or medical imaging device 170 displays selected model prediction and model metric value(s) for selected model 632. Fig. 20 depicts an example user display.

[0116] At step 634, user computer 160 or medical imaging device 170 transmits user feedback 560 to model selector 120.

[0117] At step 636, model selector 120 may update the determination of the selected model. For example, model metrics 530 may be updated.

[0118] Fig. 7 is a schematic diagram of an incoming request 504, according to aspects of the present disclosure. An incoming request may contain information provided by a user, including doctors, nurses, medical technicians, and other medical professional and support staff. An incoming request may include one or more clinical question 506 and patient case data 508.

[0119] A clinical question 506 may comprise a medical image 710 and a clinical task 720. Medical image 710 may comprise any image from various imaging modalities / sy stems, e.g., ultrasound, MRI, CT, PET, x-ray, etc. In addition, a medical image may be a digital image taken by a traditional camera. In some instances, associated with the medical image 710 may be image metadata 712, which may include a timestamp, date, imaging system settings, etc. A clinical task 720 may comprise a task type 722 and a task inquiry 724. Task type 722 may be a category for the task. Some examples of a task type are anatomy detection, anatomy segmentation, disease stage classification, etc. Task inquiry 724 indicates what the user of the model selection and prediction system is trying to determine. For example, a user may want a kidney identified from an unmarked ultrasound image. In that example, a user provides an ultrasound image of human anatomy (e.g., the medical image 710) and an inquiry “Identify a kidney in the image.” An inquiry 724 may take any number of forms, including affirmative command or a question, e.g., “What stage is the disease?”

[0120] Patient case data 508 may include any number of patient details, including biomarkers, e.g., name, date of birth, age, weight, medical history, current medications, allergies, etc. Patient case data 508 may include a class of information called case parameters. As used herein, case parameters may be used by the model selection and prediction system to generate subsets of similar cases. For example, as depicted in Figs. 9 and 12-13 case parameters, such as age and weight, are used to structure patient cases for determining their similarity to the case data contained in clinical question 506.

[0121] Fig. 8 is a schematic diagram of a patient case and a second patient case, according to aspects of the present disclosure. As shown in Fig. 8, first patient case includes the medical images 805 and patient case data 810 that a user may submit, in some instances, with a clinical task (e.g., 720 in Fig. 7), to a model prediction and selection system for processing. In comparison to the first patient case 800, second patient case 850 is a case in the ground truth dataset 122. While both the first patient case 800 and second patient case 850 include medical images 805,855, patient case data 810, 860, and related image metadata 815, 870 and case parameters 820, 875, second patient case 850 may also include annotation / ground truth labels 865 associated with the medical images (e.g., 190 in Fig. 1). Annotations / ground truth labels may include anatomy detection boxes, segmentation contours, anatomy classification labels, disease stage classification labels, etc.

[0122] Fig. 9 is a first schematic graph 900 of patient cases in a ground truth dataset 122, according to aspects of the present disclosure. Each marker in graph 900 represents a patient case in the ground truth dataset 122. As depicted, the graph has a horizontal axis representing case parameter A (e.g., age) and a vertical axis representing case parameter B (e.g., weight). An example subset of similar cases 905 are identified with patterned markers. The subset of similar cases 905 surround a first patient case 910, where the first patient case may be a case submitted by a user to the model selection and prediction system. The first patient case 920 has a first patient case vector 915 whose components are case parameter A and case parameter B for the first patient case 910. Similarly, second patient case 920 has a second patient case vector 925 whose components are case parameter A and case parameter B for the second patient case.

[0123] In some instances, between a case depicted in graph 900 and the first patient case 910 may, at least in part, determine whether the case is selected to be in a subset of similar cases (e.g., 520 in Fig. 5), A threshold distance 930 may be determined from previous similar subset selections, calculated based on one or more parameters (e.g., quality of a medical image provided by a user, the task type, the task inquiry, the image metadata, etc.), or provided by a user. Cases which have a small enough distance from first patient case 910 may be selected for inclusion in the subset of similar cases. For example, distance 935 indicates that a case should be included in the subset of similar cases, whereas distance 940 indicated that a case should not be included in the subset of similar cases.

[0124] Fig. 10 is a first schematic diagram of generating model metric values for a model from a subset of similar cases 905, according to aspects of the present disclosure. Generating model metrics values for each model allows model performances to be compared and for the model selection and prediction system to select one or more models to use for prediction based on the user incoming request. Generating model metric values may include a subset of similar cases 905 and model A 124. By way of example, the subset of similar cases 905 may include case 1 1002, case II 1004, and case III 1006. Each of the cases may include a medical image 1008, 1012, 1016 and ground truth 1010, 1014, 1018, respectively.

[0125] In some instances, model A 124 receives the medical images 1008, 1012, 1016 of each case 1002, 1004, 1006 and generates output 1022, 1024, 1026. For example, the output may be a segmentation of the medical images (e.g., identification of a contour of anatomy in the image). Model A 124 may be any number of models as described herein with respect to any model. By way of example, model A 124 may comprise a neural network such as a convolution neural network, transformer network, recurrent neural network, or other model as described herein.

[0126] An outcome distance value 1032, 1034, 1036 may be computed based on the outputs 1022, 1024, 1026 of model A and the ground truths 1010, 1014, 1018 of each case. The outcome distance metric may take different forms. For example, an outcome distance metric for a segmentation task may be the percentage of the area overlapping between the segmented regions of the ground truth segmentation and the output segmentation from a model, while the outcome distance metric for a task with numerical label may comprise the square of the difference between the ground truth label and the predicted label. If the result is a binary decision (e.g., disease detected yes / no), the outcome distance metric can be defined as 0 if equal and 1 if unequal. If the result consists of a numeric value (e.g., number of lesions), the metric can be defined as the difference of the numeric values derived from the model and the ground truth. A reliability value 1040 and robustness value 1050 of model A may be computed from the outcome distance values 1032, 1034, 1036, as described herein. For example, the reliability value may be the average, weighted or unweighted, of the outcome distance values 1032, 1034, 1036, and the robustness value may be the standard deviation of the outcome distance values 1032, 1034, 1036. The reliability value may also be the inverse of the average distance, the inverse of the median distance, the inverse of a weighted sum of mean and standard deviation of the distances, or a similar definition.

[0127] Fig. 11 is a second schematic diagram of generating model metric values from patient cases, according to aspects of the present disclosure. As depicted in Fig. 11, model metric values may be generated from a plurality of different subsets of similar cases. One or more of the subsets may be subsets of another, larger subset. Alternatively, two or more of subsets may not contain a single common patient case, i.e., they do not intersect. Generating model metric values for each model allows model performances to be compared and to allow selection of one or more models for making predictions based on the user incoming request. Generating model metric values may include a plurality of subsets of similar cases. Fig. 11 depicts three subsets: subset I of similar cases 1104, subset II of similar cases 1114, and subset III of similar cases 1124. By way of example, the subset I of similar cases 1104 may include case I 1106, case II 1108; subset II of similar cases 1114 may include case I’ 1116, case IF 1118; and the subset III of similar cases may include case F ’ 1126, case IF ’ 1128.

[0128] In some instances, subset II of similar cases 1114 and subset III of similar cases 1124 may be generated using alternative subset selection logic 1110. For example, the threshold distance, as depicted in Fig. 9, may be varied to generate new subsets. As another example, cases may be selected within a second threshold distance less than a first threshold distance of a point contained within a subset corresponding to the first threshold distance. This may be extended to a plurality of different threshold distances centered around a plurality of different points. As another example, a new subset of similar cases may be generated by randomly sampling a first subset of similar cases. Two of these examples are depicted in, and described with respect to, Figs. 12 and 13. Similar to Fig. 10, model A (not shown) may generate output from the cases 1106, 1108, 1116, 1118, 1126, 1128. For example, the output may be a segmentation of the medical images. Model A 124 may be any number of models as described herein with respect to any model.

[0129] An outcome distance value may be computed from an outcome distance metric, as described herein, and correspond to subset I of similar cases 1104, i.e., outcome distance values 1136, 1138; correspond to subset II of similar cases 1114, i.e., outcome distance values 1146, 1148; and correspond to subset III of similar cases 1124, i.e., outcome distance values 1156, 1158. The outcome distance metric may take mathematical form as described herein. For example, an outcome distance metric for a segmentation task may be the percentage of the area overlapping between the segmented regions of the ground truth segmentation and the output segmentation from a model. A reliability value 1160 and robustness value 1170 of model A may be computed from the outcome distance values 1136, 1138, 1146, 1148, 1156, 1158, as described herein. A reliability value 1160 may be computed for each subset of similar cases by averaging the outcome distance values computed for that subset. A robustness value may be computed in a number of ways. For example, the robustness value 1170 may be computed as the average, weighted or unweighted, of the reliability values for each subset. Alternatively, the robustness value may be computed as the standard deviation of the reliability values for each subset.

[0130] Fig. 12 is a second schematic graph 900 of patient cases in a ground truth dataset 122, according to aspects of the present disclosure. Each marker in graph 900 represents a patient case in the ground truth dataset 122. As depicted, the graph has a horizontal axis representing case parameter A (e.g., age) and a vertical axis representing case parameter B (e.g., weight). The graph 900 depicts three subsets of similar cases 1205, 1210, 1215.

[0131] In some instances, the three subsets of similar cases 1205, 1210, 1215 correspond to subset I-III 1104, 1114, 1124 of Fig 11, where the subsets 1210,1215 are generated using the alternative subset selection logic 1110 of Fig. 11. A subset of similar cases 1205 are identified with patterned markers (similar to the subset depicted in Fig. 9). The first subset of similar cases 1205 surround a first patient case 910, where the first patient case may be a case submitted by a user to the model selection and prediction system. In some instances, subsets 1210, 1215 are generated as described with respect to Fig. 11, i.e., by using a second and third threshold distance, both less than a first threshold distance associated with subset 1205. It should be appreciated that the shape of the subsets need not be a disk (or even approximately a disk), any geometric shape may be used. Alternatively, subset of similar cases may be generated by randomly sampling according to a probability distribution with higher probability nearer to a first patient case, e.g., 910, received from a user.

[0132] Fig. 13 is a third schematic graph 900 of patient cases in a ground truth dataset 122, according to aspects of the present disclosure. Each marker in graph 900 represents a patient case in the ground truth dataset 122. As depicted, the graph has a horizontal axis representing case parameter A (e.g., age) and a vertical axis representing case parameter B (e.g., weight). The graph 900 depicts three subsets of similar cases 1305, 1310, 1315.

[0133] In some instances, the three subsets of similar cases 1305, 1310, 1315 correspond to subset I-III 1104, 1114, 1124 of Fig 11, where the subsets 1305, 1310,1315 are generated using the alternative subset selection logic 1110 of Fig. 11. Each of the subsets 1305, 1310, 1315 is centered around a first patient case 910, where the first patient case may be a case submitted by a user to the model selection and prediction system. In some instances, subsets 1305, 1310, 1315 are generated as described with respect to Fig. 11, i.e., by using a plurality of different threshold distances measured for the first patient case 910. It should be appreciated that the shape of the subsets need not be a disk (or approximately a disk), any geometric shape may be used. Alternatively, subset of similar cases may be generated by randomly sampling according to a probability distribution with higher probability nearer to a first patient case from a user.

[0134] Fig. 14 is a schematic block diagram of at least a portion of a model selection and prediction system 1400, according to aspects of the present disclosure. Model selection and prediction system 1400 may select one or more models for use in generating a prediction in response to a clinical question. Model selection and prediction system 1400 may include models 124, 128, 150, model selection logic 1432, combination block 1434, and display 1450.

[0135] As depicted in, and described with respect to, Fig. 10, model A 124, model B 128, and model C 150 generate predict! on(s) based on a subset of similar cases. Reliability values 1405, 1415, 1425 and robustness values 1410, 1420, 1430 may be determined for each model A, B, and C, respectively. In some instances, fewer models may be used, and in other instances, more models may be used.

[0136] Model selection logic 1432 may receive reliability values 1405, 1415, 1425 and robustness values 1410, 1420, 1430. From the reliability values and robustness values, model selection logic 1432 selects one or more models. By way of illustration, Fig. 14 depicts two configurations. In the first configuration, model selection logic 1432 selects a single model, e.g., model A 124. In the second configuration, model selection logic selects two models, model A 124 and model B 128. In the first configuration, model A 124 receives the clinical question 506 and generated output 1435, as described herein. The output 1435 of model A may be displayed to user via a display 1450.

[0137] In the second configuration, both model A 124 and model B 128 receive the clinical question 506 and generate respective output 1435 and 1440. The second configuration, where more than one model is selected may be referred to as coadjutant. Next, the output 1435 of model A and output 1440 of model B may be received by a combination block 1434. Combination block 1434 generates an output through a combination of the outputs 1435, 1440. In some instances, combination block 1434 may comprise at least a portion of a neural network trained using ensemble learning. Thus, combination block 1434 may internally generate weights that are used in the combination of outputs 1435,1440. In other instances, combination block 1434 may compute on average of the outputs 1435, 1440. For example, for an object detection task, combination block may use linear interpolation to generate a single detection from each of the output detection 1435, 1440. In another example, if the task is disease stage classification, then combining the output of the two models may be accomplished by a weighted average. If model A predicted that the disease stage was 3 out of 5 while model B predicted disease stage was 4 out of 5, then combination block might generate a disease stage classification of 3.5 out of 5, an equally weighted average. However, in some instances, in may be preferable to weight one model higher than another. For example, model B’s prediction might be weighted by 0.8 and model A’s prediction weighted by 0.2. Using the previous disease stage classifications, combination block 1434 would generate a disease stage classification of 0.2 * 3 + 0.8 * 4 = 3.8.

[0138] The output of combination block 1434 may be sent to a display 1450, viewable by a user.

[0139] Fig. 15 is a schematic flow diagram of a method 1500 for model selection and prediction, according to aspects of the present disclosure. It is understood that the steps of method 1500 may be performed in a different order than shown in Fig. 15, additional steps can be provided before, during, and after the steps, and / or some of the steps described can be replaced or eliminated in other aspects. One or more of steps of the method 1500 can be carried by one or more devices and / or systems described herein, such as components of the networked system 100 and / or processor circuit 250.

[0140] In step 1502, the method 1500 includes receiving, for first patient case, incoming request (e.g., 504, including a clinical question and first patient case data). For example, incoming request 504 sent from user computer 160 or medical imaging console 170 to model selector 120 as depicted in Fig. 5. In step 1504, the method 1500 includes accessing ground truth dataset (second patient cases, each with second patient case data and ground truth label associated with clinical question). For example, model selector 120 accessing ground truth dataset 122 as depicted in Fig. 5.

[0141] In step 1506, the method 1500 includes selecting a subset of similar cases based on first patient case data and second patient case data. For example, selection is done based on the similarity metric 512 and weights 514 as depicted in Fig. 5.

[0142] In step 1508, the method 1500 includes providing the subset to first predictive model and second predictive model. For example, the subset of similar cases 520 is sent to models 124, 128 in Fig. 5. Providing the subset to a model is not the same as training the model. In some instances, the models are already trained.

[0143] In step 1510, the method 1500 includes generating first model predictions for subset by first predictive model and second model predictions for subset by second predictive model. For example, the first model predictions and second model predictions comprising model outputs 525 as depicted in Fig. 5.

[0144] In step 1512, the method 1500 includes determining selected model(s) based on ground truth labels of the subset, first model predictions, and second model predications. For example, selecting a model based on the model metrics 530 as depicted in Fig. 5.

[0145] In step 1514, the method 1500 includes providing clinical question to selected model. For example, model selector 120 providing the clinical question to model A 124 as depicted in Fig. 6B.

[0146] In step 1516, the method 1500 includes generating, by selected model, selected model prediction for first patient case. For example, model A 124 generating selected model output 540 as depicted in Fig. 5.

[0147] In step 1518, the method 1500 includes outputting selected model prediction to a display. For example, sending the selected model output 540 to the user computer 160 or medical imaging console 170 as depicted in Fig. 5.

[0148] In step 1520, the method 1500 includes receiving user feedback associated with selected model prediction. For example, user feedback sent from the user computer 160 or medical imaging console 170 to model selector 120 as depicted in Fig. 5.

[0149] In step 1522, the method 1500 includes updating, based on user feedback, determination of selected model. For example, by updating the model metrics 530 as depicted in Fig. 5. In some instances, updating the model selector’s determination of selected model affects subsequent model selections, not the selection for the current incoming request 504. Fig. 16 is a schematic diagram of a model selection and prediction system 1600 patient case filtering and model pre-selection, according to aspects of the present disclosure. Model selection and prediction system 1600 selects a model based on various conditions and metrics and then uses the selected model to make a prediction given user input. Model selection and prediction system 1600 may use information from a user input to pre-filter patient cases in a dataset and to remove models from consideration that are either not designed to perform the necessary task or would be expected to perform poorly on that task. User input may take the form of medical images and / or clinical task and the output may be a response to the clinical task. A user may be any of a number of medical professionals. Alternatively, user input may not require a user to generate and send the input, programmed instructions may automatically generate a user input based on a received image. The model selection and prediction system 1600 may include model selector 1605, ground truth dataset 122, Model A 124, Model B 128, and Model C 150.

[0150] The model selector 1605 may receive an incoming request 504 from a user computer 160 or medical imaging console 170. Incoming request 504 may comprise a clinical question 506 and first patient case data 508. In some instances, a clinical question may include a medical image and / or a clinical task, further described with respect to Fig. 7. Incoming request 504 may be generated from a user’s input into the user computer 160 or medical imaging console 170.

[0151] Model selector 1605 may include a similar case selection module 1610 and a model metrics module 1630. The similar case selection module 1610 identifies a set of cases that are similar, according to one or more metrics, to the received information contained in the incoming request 504. The model metric module 1630 identifies one or more models for selections according to one or more metrics, user selections, model metadata, and / or other information.

[0152] The similar case selection module 1610 may include a similarity metric 1612 and weights 1614. The similarity metric 1612 measures how close one patient case is to another. The similarity metric 1612 may depend on a patient’s age, weight, height, and other characteristics or biomarkers. The weights 1614 may be parameters used in the similarity metric to generate a numerical value for the similarity metric 1612 from two or more cases. For example, a similarity metric may be defined by a function, S, represented by a Eq. 1.

[0153] The model metrics module 1630 may include an outcome distance metric 1632, a reliability metric 1634, a robustness metric 1636, and weights 1638. The outcome distance metric 1632 determines a distance between a ground truth label of the cases in the subset of similar cases and the label contained in the model outputs 1660 generated by the models 124, 128. The reliability metric 534 measures how reliably each model based on its performance on generating output for subset of similar cases 520 that match ground truth labels (i.e., 190). The robustness metric 536 measures how robust the performance of models across different subsets of ground truth data. In other words, the robustness metric indicates the sensitivity of a model to inputs which are close in the sense described herein with respect to the similarity metric. The weights are used in the outcome distance metric 532, reliability metric 534, and robustness metric 536 as parameters to generate numerical values for the metrics given the model outputs and ground truth labels.

[0154] The similar case selection module 1610 may receive the incoming request 504 and pre-filter cases from the ground truth dataset 122 based on the incoming request. In some instances, the task type, task inquiry, image metadata, etc., may be used by the similar case selection module 1610 to filter, or pre-filter, cases in the ground truth dataset. Thus, only a filtered portion 1640 of cases in the ground truth dataset 122 may be selected from. For example, the incoming request may seek object detection of a uterus, thus similar case selection module 1610 may pre-filter all the cases with male as the biological sex identifier in the patient case data in the ground truth dataset. Pre-filtering may be carried out by other devices and systems described herein.

[0155] The similar case selection module 1610 may generate a subset of similar cases 1645 from the filtered portion 1640 of cases that are similar to the patient case described in the incoming request 504. For example, similar case selection module 510 may select all of those case in the filtered portion 1640 of cases which have similarity metric values less than a maximum value. The maximum value may be pre-programmed into the module or determined directly or indirectly from the incoming request 504.

[0156] In some instances, model selector 1605 may pre-select the models that will receive the subset of similar cases 1645. In some instances, the task type, task inquiry, image metadata, etc., may be used by the model selector 1605 to pre-select models for testing. For example, some models may not be designed for object detection, thus generating output from such a model would generate output unrelated to the task. For sake of example, Fig. 16 depicts a configuration where model A 124 and model B 128 have been pre-selected by the model selector 1605.

[0157] As an example of model pre-selection, there may be 15 models available to choose from: 5 are trained with ultrasound images, 5 are trained with MRI images, and 5 are trained with CT images. If the incoming requests includes an MRI image, then the 5 models trained on ultrasound images and the 5 models trained with CT images can be excluded. Thus, the 5 models trained with MRI images can be pre-selected. Pre-selection allows models to be removed from consideration without requiring extensive computational resources.

[0158] Both model A 124 and model B 128 may receive the cases in the subset of similar cases 1645 and generate respective model outputs 1660. For example, model outputs 1660 may take the form of segmentations for images, i.e., because the task was segmentation. Model outputs 1660 may take any form depending on the clinical task the model is asked to solve, such diagnosis, segmentation, classification tasks, etc. The model selection and prediction system 1600 may include varying numbers of models. For example, selection may be between two models, four to twenty models, and / or other values both larger and smaller.

[0159] The model metrics module 1630 may receive the model outputs 1660 and generate selected model output 1662, reliability value 1665, and / or robustness value 1670. The model metrics module 1630 may select a model, e.g., one or more of Model A 124, Model B 128. The model selection and prediction system 1600 may generate, using the selected model, the selected model output 1662 based on the incoming request. The model metrics module 1630 may generate reliability values 1665 using the reliability metric 1634 and the weights 1638. The model metrics module 1605 may generate robustness values 1670 using the robustness metric 1636 and the weights 1638.

[0160] User computer 160 or medical imaging console 170 may receive the selected model output 1662, reliability value 1665, and / or robustness value 1670. The user computer 160 or medical imaging console 170 may display and / or post-process the information from the model selection and prediction system 1600. A user, such as a doctor, interacting with the display may evaluate the results of the model selection and prediction system generated from the incoming request 504. The user may provide user feedback 1680 to the model selection and prediction system 1600, in some instances, to the model selector 1605, that evaluates the performance of the system. User feedback 1680 may also include parameter or other selections to modify, directly or indirectly, the weights 1614, 1638 or metrics 1612, 1632, 1634, or 1636 in the model selector 1605. A user may act iteratively with the model selection and prediction system 1600 to arrive at a desired clinical result. In addition, the model may be finetuned with user feedback on performance.

[0161] Fig. 17 is a schematic flow diagram of a method 1700 for model selection and prediction with patient case filtering and model pre-selection, according to aspects of the present disclosure. It is understood that the steps of method 1700 may be performed in a different order than shown in Fig. 17, additional steps can be provided before, during, and after the steps, and / or some of the steps described can be replaced or eliminated in other aspects. One or more of steps of the method 1700 can be carried by one or more devices and / or systems described herein, such as components of the networked system 100 and / or processor circuit 250.

[0162] In step 1702, the method 1700 includes receiving, for first patient case, incoming request (clinical question and first patient case data). For example, incoming request 504 sent from user computer 160 or medical imaging console 170 to model selector 1605 as depicted in Fig. 16.

[0163] In step 1704, the method 1700 includes accessing ground truth dataset (second patient cases, each with second patient case data and ground truth label associated with clinical question). For example, model selector 1605 accessing ground truth dataset 122 as depicted in Fig. 16.

[0164] In step 1706, the method 1700 includes filtering second patient cases based on comparison between clinical questions and ground truth labels of second patient cases. In some instances, filtering may also be based on image metadata. For example, filter portion of cases 1640 depicted in Fig. 16.

[0165] In step 1708, the method 1700 includes selecting a subset of similar cases from filtered potion of second patient cases based on first patient case data and second patient case data. For example, selection is done based on similarity metric 1612 and weights 1614 as depicted in Fig. 16.

[0166] In step 1710, the method 1700 includes pre-selecting first predictive model and second predictive model from multiple predictive models. For example, pre-selected model 1650 as depicted in Fig. 16.

[0167] In step 1712, the method 1700 includes providing subset to first predictive model and second predictive model. For example, subset of similar cases 1645 is sent to models 124, 128 as depicted in Fig. 16. Providing the subset to a model is not the same as training the model. In some instances, the models are already trained.

[0168] In step 1714, the method 1700 includes generating first model predictions for subset by first predictive model and second model predictions for subset by second predictive model. For example, the first model predictions and second model predictions in model outputs 1660 as depicted in Fig. 16.

[0169] In step 1716, the method 1700 includes determining selected model(s) based on ground truth labels of the subset, first model predictions, and second model predictions. For example, selecting a model based on model metrics 1630 as depicted in Fig. 16. In step 1718, the method 1700 includes providing clinical question to selected model. For example, model selector 120 providing the clinical question to model A 124 as depicted in Fig. 6B

[0170] In step 1720, the method 1700 includes generating, by selected model, selected model prediction for the first patient case. For example, model A 124 generating selected model output 1662 as depicted in Fig. 16.

[0171] In step 1722, the method 1700 includes outputting selected model prediction to display. For example, sending the selected model output 1662 to the user computer 160 or medical imaging console 170 as depicted in Fig. 16.

[0172] Fig. 18 is a schematic diagram of a model selection and prediction system 1800 with request comparison, according to aspects of the present disclosure. Model selection and prediction system 1800 compares a new request from a user with previous requests to determine if the new request is similar to old request. If a new request is determined to be similar enough to an old request, then the same model that was selected for the old request may be selected for the new request. Thus, a significant savings in computation time and resources may be realized. Model selection and prediction system may include a user computer 160 or medical imaging console 170, model selector 1805, previous incoming request dataset 1820, ground truth dataset 122, and models A 124, model B 128, and model C 150.

[0173] The model selector 1805 may receive an incoming request 504 from a user computer 160 or medical imaging console 170. Incoming request 504 may comprise a clinical question 506 and first patient case data 508. In some instances, a clinical question may include a medical image and / or a clinical task, further described with respect to Fig. 7. Incoming request 504 may be generated from a user’s input into the user computer 160 or medical imaging console 170.

[0174] Model selector 1805 may include a previous request comparison module 1810, similar case selection module 510, and model metrics module 530. The previous request comparison may determine whether the incoming request 504 and a previous request in the previous requests dataset 1820 are similar enough to use the previous selected model for the incoming request 504. Previous requests dataset 1820 includes the previous clinical questions 1822, previous first patient case data 1824, and previous selected models 1826

[0175] Previous request comparison module 1810 may use any of the information provided in the incoming request 504 to determine if a selected model may be chosen based on a previous request. In some instances, the comparison may be based on age, weight, and other biomarkers, as well as the task type, task inquiry, image metadata, etc. If the previous request comparison module determines the incoming request is sufficiently close to one of the previous requests, then the previously selected model is chosen as the model for evaluating the incoming request. In that case, it may be unnecessary to utilize the similar case selection module 510 or the model metrics module 530. By way of example, Fig. 18 depicts selection of model A to generate selected model output 1830 from the incoming request 504. Selected model output 1830 may be provided to the user computer 160 or medical imaging console 170 and displayed to a user.

[0176] As an example of an incoming request comparison to previous requests, an incoming request may seek a disease stage classification for cirrhosis of a 60-y ear-old’s liver. The selection and prediction system 1800 may search through previous requests that have disease stage classification of cirrhosis of the liver as the clinical task and requests with similarly aged patients. For example, the system would ignore previous requests for image segmentation of the lungs of a patient. If a close enough match is found, then the model used for the identified previous request is used on the current incoming request.

[0177] Fig. 19 is a schematic flow diagram of a method 1900 for model selection and prediction with incoming request comparison, according to aspects of the present disclosure. It is understood that the steps of method 1900 may be performed in a different order than shown in Figure 19, additional steps can be provided before, during, and after the steps, and / or some of the steps described can be replaced or eliminated in other aspects. One or more of steps of the method 1900 can be carried by one or more devices and / or systems described herein, such as components of the networked system 100 and / or processor circuit 250.

[0178] In step 1902, the method 1900 includes receiving, for a first patient case, incoming request (clinical question and first patient case data). For example, incoming request 504 sent from user computer 160 or medical imaging console 170 to model selector 1805 as depicted in Fig. 18.

[0179] In step 1904, the method 1900 includes performing a comparison between an incoming request and previous incoming requests stored in memory. For example, comparison is made by previous request comparison 1810 with the incoming request 504 and previous request for the previous requests dataset 1820 as depicted in Fig. 18. For example, the comparison can include calculating a similarity metric (like the similarity metric 512) by using the first patient case data 508 and the previous first patient case data 1824.

[0180] In step 1906, the method 1900 includes a decision step to determine if the incoming request is similar enough to a previous incoming request. For example, if the value of the similarity metric satisfies a threshold value (e.g., one of greater than the threshold value or less than the threshold value), then the incoming request is sufficiently similar to at least one incoming request. If the value of the similarity metric does not satisfy the threshold value (e.g., the other of greater than the threshold value or less than the threshold value), then incoming request is not sufficiently similar to at least one incoming request. If similar enough, then the method proceeds to step 1907, and if it is not similar enough, then method proceeds to step 1908. In some instances, determining if two incoming requests are close enough may be accomplished via a threshold score, e.g., a percentage representing equivalency of two requests. Alternatively, the model selection and prediction system could require the task type to match and for case parameters to lie within a pre-defined range of each other. For example, an incoming request for an 80-year-old patient may be similar to one for a 75-year-old patient but not a 55-year-old patient.

[0181] In step 1907, the method 1900 includes identifying previously determined selected model for previous incoming request as selected model. For example, the previously selected model in the previous selected models 1826 associated with the previous request as depicted in Fig. 18. As depicted in Fig. 18. Model A 124 was the previously selected model.

[0182] In step 1908, the method 1900 includes accessing ground truth data set (second patient cases, each with second patient case data and ground truth label associated with clinical question). For example, model selector 120 accessing ground truth dataset 122 as depicted in Fig. 5.

[0183] In step 1910, the method 1900 includes selecting subset of similar cases based on first patient case data and second patient case data. For example, selection is done based on the similarity metric 512 and weights 514 as depicted in Fig. 5.

[0184] In step 1912, the method 1900 includes providing subset to first predictive model and second predictive model. For example, the subset of similar cases 520 is sent to models 124, 128, and 150 as depicted in Fig. 5. Providing the subset to a model is not the same as training the model. In some instances, the models are already trained.

[0185] In step 1914, the method 1900 includes generating first model predictions for subset by first predictive model and second model predictions for subset by second predictive model. For example, the first model predictions and second model predictions comprising model outputs 525 as depicted in Fig. 5

[0186] In step 1916, the method 1900 includes determining selected model(s) based on ground truth labels of the subset, first model predictions, and second model predictions. For example, selecting a model based on model metrics 530 as depicted in Fig. 5. In step 1918, the method 1900 includes providing clinical question to selected model. For example, model selector 120 providing clinical question to model A 124 as depicted in Fig. 6B.

[0187] In step 1920, the method 1900 includes generating, by selected model, selected model prediction for first patient case. For example, model A 124 generating selected model output 540 as depicted in Fig. 5.

[0188] In step 1922, the method 1900 includes outputting selected model prediction to a display. For example, sending the selected model output 540 to the user computer 160 or medical imaging console 170 as depicted in Fig. 5.

[0189] Fig. 20 is an example user interface on a display including selected model and model result, according to aspects of the present disclosure. A user may both input information, such as a first patient case via the user interface on a display, and receive information, such as results from model selection and prediction. Example user interface may include patient data 2005, clinical question interface 2010, case parameter interface 2015, feedback interface 2020, and result interface 2025.

[0190] Patient data 2005 may include patient identifying information, such as name, date of birth, record identifier, patient identifier, medical facility where patient is located, etc. Patient data 2005 may include hyperlinks to a database containing patient medical records.

[0191] Clinical question interface 2010 may provide functionality to select or upload an image file and to enter a clinical task. Clinical task input may include the type of task, e.g., object detection and inquiry, e.g., what object is sought to be detected. The task type and / or inquiry

[0192] Case parameter interface 2015 depicts various values and ranges for case parameters used in the model selection and prediction. For example, an age of the patient / subject and the maximum and minimum age of patient’s used in the subset of similar cases, as described herein, may be marked on a line. In some instances, the markers may be configurable by a user to select wider, narrow, or shifted ranges for age to be used in the model selection and prediction system. By making new selections for the age ranges, the model selection and prediction system may be rerun to update the results based on the user’s input. Similar functionality may be used for other parameters, e.g., weight, blood pressure, and other biomarkers.

[0193] Feedback interface 2020 may allow a user to evaluate the results generated by the model selection and prediction system. Feedback may be as simple a yes / no question, “Did selected Al model successfully answer clinical question.” In some instances, the feedback may seek ratings, such a numerical input from 1-10. Feedback interface 2020 may include an option to rerun the model selection and prediction system considering any changes to the input in interfaces 2010, 2015, 2020.

[0194] As depicted in Fig. 20, result interface 2025 displays results for detection of a kidney in an ultrasound image. Results interface 2025 includes the ultrasound image with an objection detection box generated as output of the model selection and prediction system. Results interface 2025 may include various metrics including the reliability and robustness metric as described herein, as well as model confidence for the result. The confidence from the model is a distinct metric from the reliability and robustness metrics. In some instances, the name and / or source of the model used to generate the result may be listed. In addition, the results interface may include a list of models whose performance during model selection was also high.

[0195] It should be appreciated that the icons may be rearranged as needed and that various degrees of technical information may be depicted on the user interface. For example, more sophisticated users may be shown highly detailed information regarding results and case parameters, as well as a greater degree of ability to provide feedback. Certain users may need permissions to provide feedback, thus ensuring only users with the desired level of expertise are able to contribute to any model training or updating based on feedback.

[0196] Furthermore, the technology disclosed herein is also applicable to other medical imaging modalities obtained from a medical imaging device where 3D data is available, such as other ultrasound applications, camera-based videos, X-ray videos, and 3D volume images, such as computer aided tomography (CT) scans, magnetic resonance imaging (MRI) scans, optical coherence tomography (OCT) scans, or intravenous ultrasound (IVUS) pullback sequences. The technology described herein can be used in a variety of settings including emergency department, intensive care, inpatient, and out-of-hospital settings.

[0197] Accordingly, the logical operations making up the aspects of the technology described herein are referred to variously as operations, steps, objects, layers, elements, components, algorithms, or modules. Furthermore, it should be understood that these may occur or be performed or arranged in any order, unless explicitly claimed otherwise or a specific order is inherently necessitated by the claim language.

[0198] All directional references e.g., upper, lower, inner, outer, upward, downward, left, right, lateral, front, back, top, bottom, above, below, vertical, horizontal, clockwise, counterclockwise, proximal, and distal are only used for identification purposes to aid the reader’s understanding of the claimed subject matter, and do not create limitations, particularly as to the position, orientation, or use of the Automatic measurement point detection system. Connection references, e.g., attached, coupled, connected, joined, or “in communication with” are to be construed broadly and may include intermediate members between a collection of elements and relative movement between elements unless otherwise indicated. As such, connection references do not necessarily imply that two elements are directly connected and in fixed relation to each other. The term “or” shall be interpreted to mean “and / or” rather than “exclusive or.” The word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. Unless otherwise noted in the claims, stated values shall be interpreted as illustrative only and shall not be taken to be limiting.

[0199] The above specification, examples and data provide a complete description of the structure and use of exemplary aspects of the model selection and prediction system as described herein. Although various aspects of the claimed subject matter have been described above with a certain degree of particularity, or with reference to one or more individual aspects, those skilled in the art could make numerous alterations to the disclosed aspects without departing from the spirit or scope of the claimed subject matter.

[0200] Still other aspects are contemplated. It is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative only of particular aspects and not limiting. Changes in detail or structure may be made without departing from the basic elements of the subject matter as defined in the following claims.

[0201] In particular, an advantageous embodiment is a computer-implemented method that comprises the step of receiving, for a first patient case, a clinical question and first patient case data. It comprises the step of performing a comparison between the clinical question and a plurality of previous clinical questions. It comprises the step of computing, using a similarity metric, similarity values for each of the plurality of previous clinical questions based on a comparison between the clinical question and the plurality of previous clinical questions. It comprises the step of determining a selected predictive model. It comprises the step of providing the clinical question to the selected model. It comprises the step of generating, by the selected model, a selected model prediction for the first patient case. It comprises the step of outputting the selected model prediction to a display, wherein, when the similarity values for the plurality of previous clinical questions does not satisfy a threshold, determining a selected model comprises: accessing a plurality of second patient cases, wherein each second patient case comprises second patient case data and a ground truth label; selecting a subset of the plurality of second patient cases based on the first patient case data and the second patient case data of the plurality of second patient cases; providing the subset to a first predictive model and a second predictive model. The advantageous embodiment further comprises the step of generating a plurality of first model predictions for the subset by the first predictive model and a plurality of second model predictions for the subset by the second predictive model. It comprises the step of selecting at least one of at least one of the first predictive model or the second predictive model as the selected predictive model based on the ground truth labels of the subset, the plurality of first model predictions, and the plurality of second model predictions, wherein, when the similarity value for a previous clinical question of the plurality of clinical questions satisfies the threshold, determining the selected model comprises: selecting a predictive model previously determined for the previous clinical question as the selected model.

Claims

CLAIMS:

1. A computer-implemented method, comprising: receiving, for a first patient case, a clinical question (506) and first patient case data (508); accessing a plurality of second patient cases (122), wherein each second patient case (850) comprises second patient case data (860) and a ground truth label (865); selecting a subset (520) of the plurality of second patient cases based on the first patient case data (508) and the second patient case data (860) of the plurality of second patient cases; providing the subset (520) to a first predictive model (124) and a second predictive model (128); generating a plurality of first model predictions (618) for the subset by the first predictive model (124) and a plurality of second model predictions (620) for the subset by the second predictive model (128); determining a selected model based on the ground truth labels of the subset, the plurality of first model predictions (618), and the plurality of second model predictions (620), wherein the selected model comprises at least one of the first predictive model (124) or the second predictive model (128); providing the clinical question (506) to the selected model; generating, by the selected model, a selected model prediction for the first patient case; and outputting the selected model prediction (540) to a display (166).

2. The method of claim 1, wherein the clinical question (506) comprises a first medical image (710) and a clinical task (720) associated with the first medical image, and wherein the selected model prediction comprises a graphical representation (460) overlaid on the first medical image (450) and associated with the clinical task.

3. The method of claim 2, wherein the clinical task (720) comprises at least one of anatomy detection, anatomy segmentation, disease stage classification.

4. The method according to any of the preceding claims, wherein the first predictive model (124) and the second predictive model (128) each comprise a pre-trained convolutional neural network (CNN) (312) .

5. The method according to any of the preceding claims, wherein each second patient case (850) further comprises a second medical image (855), wherein the ground truth label (865) is associated with the second medical image.

6. The method of claim 5, wherein the providing the subset to the first predictive model and the second predictive model comprises: providing the second medical images (855) of the subset (122) to the first predictive model (124) and the second predictive model (128).

7. The method according to any of the preceding claims, wherein the selecting the subset comprises: computing, using a similarity metric (512), a similarity value for each second patient case based on a comparison between the first patient case data (508) and the second patient case data (860) of the plurality of second patient cases; and identifying the subset (122) based on the similarity value.

8. The method according to any of the preceding claims, wherein the determining the selected model comprises: computing, using a first plurality of outcome distance metrics (512), a first plurality of outcome distance values (1032) based on a comparison between the plurality of first model predications (1022) and the ground truth labels (1010) for subset; and computing, using a second plurality of outcome distance metrics, a second plurality of outcome distance values based on a comparison between the plurality of second model predictions and the ground truth labels for subset.

9. The method of claim 8, wherein the determining the selected model comprises:computing, using a first reliability metric for the first predictive model (124), a first reliability value (1405) based on the first plurality of outcome distance values; computing, using a second reliability metric for the second predictive model (128), a second reliability value (1415) based on the second plurality of outcome distance values; and identifying the selected model based on the first reliability value and the second reliability value.

10. The method according to any of the preceding claims, further comprising: computing, using a first robustness metric for the first predictive model, a first robustness value (1410) based on the first plurality of outcome distance values; computing, using a second robustness metric for the second predictive model, a second robustness value (1420) based on the second plurality of outcome distance values; wherein identifying the selected model is further based on the first robustness value and the second robustness value.

11. The method of claim 10, wherein computing the first robustness value (1410) for the first predictive model (124) is based on a further subset (1210) of the plurality of second patient cases, and wherein computing the second robustness value (1420) for the second predictive model (128) is based on a further subset (1215) of the plurality of second patient cases.

12. The method according to any of the claims 9-11, further comprising outputting at least one of the first reliability value (1405) or the second reliability value (1415) to the display (166).

13. The method according to any of the preceding claims, wherein the selected model comprises the first predictive model (124) and the second predictive model (128).

14. The method according to any of the preceding claims, further comprising generating a combined output (1434) based on the selected model prediction (1435) by the first predictive model (124) for the first patient case and the selected model prediction (1440) by the second model (128) for the first patient case; wherein the selected model prediction comprises the combined output.

15. The method according to any of the preceding claims, further comprising pre-selecting the first predictive model and the second predictive model (1650) from a plurality of predictive models, wherein the plurality of predictive models comprises the first predictive model (124), the second predictive model (128), and a third predictive model (150), wherein each of the plurality of predictive models comprises model metadata (126) representative of model training, wherein pre-selecting comprising: performing a comparison between the model metadata of the plurality of predictive models and at least one of the clinical question (506) or the first patient case data (508); and at least one of: identifying the first predictive model (124) and the second predictive model based (128) on the comparison; or eliminating the third predictive model (150) based on the comparison.

16. The method according to any of the preceding claims, further comprising filtering the plurality of second patient cases (122), wherein the filtering comprises: performing a comparison between: at least one of the first patient case data (508) or the clinical question (506); and at least one of the second patient case data (860) or the ground truth labels (865) of the plurality of second patient cases; identifying, based on the comparison, a filtered portion (1640) of the plurality of second patient cases (122), wherein the subset (1645) is selected from the filtered portion (1640) of the plurality of second patient cases (122).

17. The method according to any of the preceding claims, receiving user feedback (560) representative of whether the selected model prediction is responsive to the clinical question (506); and updating, based on the user feedback, how the selected model is determined.

18. A system (100), comprising: a processor circuit (112) configured to: receive, for a first patient case, a clinical question (506) and first patient case data (508); access a plurality of second patient cases (122), wherein each second patient case (850) comprises second patient case data (860) and a ground truth label (865); select a subset (520) of the plurality of second patient cases based on the first patient case data (508) and the second patient case data (860) of the plurality of second patient cases; provide the subset (520) to a first predictive model (124) and a second predictive model (128); generate a plurality of first model predictions (618) for the subset by the first predictive model (124) and a plurality of second model predictions (620) for the subset by the second predictive model (128); determine a selected model based on the ground truth labels of the subset, the plurality of first model predictions (618), and the plurality of second model predictions (620), wherein the selected model comprises at least one of the first predictive model (124) or the second predictive model (128); provide the clinical question (506) to the selected model; generate, by the selected model, a selected model prediction for the first patient case; and output the selected model prediction (540) to a display (166).

19. A non-transitory machine-readable medium comprising a plurality of machineexecutable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising: receiving, for a first patient case, a clinical question (506) and first patient case data (508); accessing a plurality of second patient cases (122), wherein each second patient case (850) comprises second patient case data (860) and a ground truth label (865); selecting a subset (520) of the plurality of second patient cases based on the first patient case data (508) and the second patient case data (860) of the plurality of second patient cases;providing the subset (520) to a first predictive model (124) and a second predictive model (128); generating a plurality of first model predictions for the subset by the first predictive model and a plurality of second model predictions for the subset by the second predictive model; determining a selected model based on the ground truth labels of the subset, the plurality of first model predictions (618), and the plurality of second model predictions (620), wherein the selected model comprises at least one of the first predictive model (124) or the second predictive model (128); providing the clinical question (506) to the selected model; generating, by the selected model, a selected model prediction for the first patient case; and outputting the selected model prediction (540) to a display (166).

Images