Selection of medical images for physiological assessment and models for medical image analysis

Abstract

The selection of a predictive model for medical image analysis includes: a processor circuit receiving a clinical question and first patient case data, and accessing multiple second patient cases. Each second patient case contains second patient case data and a baseline truth label. The processor circuit selects a subset of the second patient cases based on the first and second patient case data, provides the subset to a first predictive model and a second predictive model, generates a first model prediction result from the first predictive model, and generates a second model prediction result from the second predictive model. The processor circuit determines the selected model based on the baseline truth label of the subset, the first model prediction result, and the second model prediction result. The processor circuit generates the selected model prediction result for the first patient case using the selected model and outputs the selected model prediction result to a display.

Description

Technical Field

[0001] The topics described in this article involve using medical images (e.g., ultrasound, MRI, CT, PET, etc.) to perform physiological assessments of patients. For example, for a given patient case, multiple predictive models are tested using similar patient cases to identify the most suitable predictive model for analyzing medical images (e.g., identifying anatomical structures, segmenting anatomical structures, classifying disease stages, etc.). Background Technology

[0002] Increasingly, medical professionals, technicians, and staff are leveraging artificial intelligence and various neural network-based models for diagnosing and analyzing patient records. Furthermore, a large and growing number of such models are already available or will soon be available. These models are often tailored to specific use cases, such as image segmentation or diagnostic classification. These models may be trained on data that is no longer known or available. Therefore, accurately selecting a model can be challenging. Additionally, many models may be too large to store on a local computer or server. Finally, users of the models (such as physicians) often lack the expertise to select models that perform well in their use cases.

[0003] The information contained in the background section of this specification, including any references cited herein and their descriptions or discussions, is included for technical reference only and should not be considered as the subject matter defining the scope of this disclosure. Summary of the Invention

[0004] An automated model selection and prediction system for evaluating patient medical data (e.g., anatomical images) is disclosed. A set of similar patient cases is selected to identify suitable predictive models by comparing an incoming request with a dataset of patient cases and baseline ground truth labels. After identifying the set of similar cases, potential models can be tested on the set. Model performance can be generated based on comparing the baseline ground truth labels in the set of similar cases with the output of each model. A model is then selected based on the performance of each model, and a prediction for the incoming request can be generated and provided to the user. For example, a physician might request the detection of a specific anatomical feature in an ultrasound image; therefore, the output of the selected model will include the medical image with a bounding box defining the anatomical feature.

[0005] The automated model selection and prediction system disclosed herein has specific uses (but is not limited to) for diagnosing diseases, segmenting images, detecting anatomical structures, etc. The automated model selection and prediction system automatically selects a subset of a dataset, automatically selects a model, generates predictions based on the model, and generates metrics representing model performance. The automated model selection and prediction system advantageously allows the selection of the most suitable model (if any) for a given clinical task from a multitude of models. This allows even users (e.g., physicians) who are not proficient in different types of prediction models to assess patients using appropriate models through model-based analysis of medical images.

[0006] A system of one or more computers can be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which, during operation, cause the system to perform the actions. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0007] In one general aspect, this disclosure relates to a computer-implemented method. The computer-implemented method further includes receiving a clinical question and first patient case data for a first patient case. The computer-implemented method further includes accessing a plurality of second patient cases, wherein each second patient case may include second patient case data and a baseline truth label. The computer-implemented method further includes selecting a subset of the plurality of second patient cases based on the first patient case data and the second patient case data from the plurality of second patient cases. The computer-implemented method further includes providing the subset to a first prediction model and a second prediction model. The computer-implemented method further includes: generating a plurality of first model prediction results by the first prediction model for the subset; and generating a plurality of second model prediction results by the second prediction model for the subset. The computer-implemented method further includes: determining a selected model based on the baseline truth label of the subset, the plurality of first model prediction results, and the plurality of second model prediction results, wherein the selected model may include at least one of the first prediction model or the second prediction model. The computer-implemented method further includes providing a clinical question to the selected model. The computer-implemented method further includes generating a selected model prediction result for the first patient case by the selected model. The computer-implemented method further includes outputting the selected model prediction result to a display.

[0008] In some aspects, the implementation may include one or more of the following features. The clinical problem may include a first medical image and a clinical task associated with the first medical image, and the selected model prediction result may include a graphical representation superimposed on the first medical image and associated with the clinical task. The clinical task may include at least one of anatomical structure detection, anatomical structure segmentation, and disease stage classification. Both the first prediction model and the second prediction model may include a pre-trained convolutional neural network. Each second patient case may also include a second medical image, and the baseline ground truth label is associated with the second medical image. Providing the subset to the first prediction model and the second prediction model may include: providing the second medical images in the subset to the first prediction model and the second prediction model. Selecting the subset may include: using a similarity metric, calculating a similarity value for each second patient case based on a comparison between the first patient case data and the second patient case data in the plurality of second patient cases; and identifying the subset based on the similarity value. Determining the selected model may include: calculating a first plurality of result distance values ​​based on a comparison between the plurality of first model prediction results and the baseline ground truth labels of the subset, using a first subset result distance metric; and calculating a second plurality of result distance values ​​based on a comparison between the plurality of second model prediction results and the baseline ground truth labels of the subset, using a second plurality of result distance metric. Determining the selected model may include: calculating a first reliability value based on the first plurality of result distance values ​​using a first reliability metric for the first prediction model; calculating a second reliability value based on the second plurality of result distance values ​​using a second reliability metric for the second prediction model; and identifying the selected model based on the first reliability value and the second reliability value. Identifying the selected model is also based on the first robustness value and the second robustness value. Calculating the first robustness value for the first prediction model is based on another subset of the plurality of second patient cases, and calculating the second robustness value for the second prediction model is based on another subset of the plurality of second patient cases, and the method may include outputting at least one of the first reliability value or the second reliability value to the display. The selected model may include the first prediction model and the second prediction model. The selected model prediction results can include a combination of output results.The plurality of predictive models may include a first predictive model, a second predictive model, and a third predictive model, each of the plurality of predictive models may include model metadata representing model training, and pre-selection may include: performing a comparison of the model metadata of the plurality of predictive models with at least one of: the clinical question or the first patient case data; and at least one of: identifying the first predictive model and the second predictive model based on the comparison; or excluding the third predictive model based on the comparison. The filtering may include: performing a comparison among: at least one of the first patient case data or the clinical question; at least one of the second patient case data or the baseline ground truth labels of the plurality of second patient cases; and identifying a red-filtered subset of the plurality of second patient cases based on the comparison, the subset being selected from the filtered subset of the plurality of second patient cases. The method may include: receiving user feedback indicating whether the prediction results of the selected model are responsive to the clinical question; and updating how the selected model was determined based on the user feedback.

[0009] In one general aspect, this disclosure relates to a computer-implemented method. The computer-implemented method further includes: receiving a clinical question and first patient case data for a first patient case; performing a comparison between the clinical question and a plurality of prior clinical questions; and calculating a similarity value for each of the plurality of prior clinical questions based on the comparison of the clinical question and the plurality of prior clinical questions using a similarity metric. The computer-implemented method further includes determining a selected prediction model. The computer-implemented method further includes providing the clinical question to the selected model. The computer-implemented method further includes: generating a selected model prediction result for the first patient case using the selected model, and outputting the selected model prediction result to a display. When the similarity values ​​of the plurality of prior clinical questions do not meet a threshold, the computer-implemented method includes determining a selected model, which may include: accessing a plurality of second patient cases, wherein each second patient case may include second patient case data and a baseline ground truth label; selecting a subset of the plurality of second patient cases based on the first patient case data and the second patient case data in the plurality of second patient cases; providing the subset to a first predictive model and a second predictive model; generating a plurality of first model prediction results for the subset by the first predictive model and generating a plurality of second model prediction results for the subset by the second predictive model; and selecting at least one of the first predictive model or the second predictive model as the selected predictive model based on the baseline ground truth label of the subset, the plurality of first model prediction results, and the plurality of second model prediction results. When the similarity values ​​of the prior clinical questions among the plurality of clinical questions meet a threshold, the computer-implemented method includes determining a selected model, which may include: selecting a previously determined predictive model for the prior clinical questions as the selected model.

[0010] In one aspect, this disclosure relates to a system including processor circuitry configured to: receive a clinical question and first patient case data for a first patient case; access a plurality of second patient cases, wherein each second patient case may include second patient case data and a baseline truth label; select a subset of the plurality of second patient cases based on the first patient case data and the second patient case data from the plurality of second patient cases; provide the subset to a first prediction model and a second prediction model; generate a plurality of first model prediction results for the subset by the first prediction model and generate a plurality of second model prediction results for the subset by the second prediction model; determine a selected model based on the baseline truth label of the subset, the plurality of first model prediction results, and the plurality of second model prediction results, wherein the selected model may include at least one of the first prediction model or the second prediction model; provide the clinical question to the selected model; generate the selected model prediction results for the first patient case using the selected model; and output the selected model prediction results to a display.

[0011] In one general aspect, this disclosure relates to a non-transient machine-readable medium that may include a plurality of machine-executable instructions. The non-transient machine-readable medium includes instructions for receiving a clinical question and first patient case data for a first patient case. The non-transient machine-readable medium also includes instructions for accessing a plurality of second patient cases, wherein each second patient case may include second patient case data and a baseline truth label. The non-transient machine-readable medium further includes instructions for selecting a subset of the plurality of second patient cases based on the first patient case data and the second patient case data from the plurality of second patient cases. The non-transient machine-readable medium further includes instructions for providing the subset to a first prediction model and a second prediction model. The non-transient machine-readable medium further includes instructions for generating a plurality of first model prediction results for the subset by the first prediction model and for generating a plurality of second model prediction results for the subset by the second prediction model. The non-transient machine-readable medium further includes instructions for determining a selected model based on the baseline truth label of the subset, the plurality of first model prediction results, and the plurality of second model prediction results, wherein the selected model may include at least one of the first prediction model or the second prediction model. The non-transient machine-readable medium also includes instructions for providing the clinical problem to the selected model. The non-transient machine-readable medium further includes instructions for generating a selected model prediction result for the first patient case from the selected model. The non-transient machine-readable medium also includes instructions for outputting the selected model prediction result to a display.

[0012] This abstract is provided to introduce some concepts in a simplified form, which will be further described in the detailed description section below. This abstract is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. The features, details, uses, and advantages of the automated measurement point detection system as defined in the claims are more fully described in the following written description of various aspects of this disclosure, illustrated in the accompanying drawings. Attached Figure Description

[0013] The illustrative aspects of this disclosure will be described with reference to the accompanying drawings, in which,

[0014] Figure 1 This is a schematic diagram of a network system for model selection and prediction based on aspects of this disclosure.

[0015] Figure 2 This is a schematic diagram of a processor circuit according to aspects of this disclosure.

[0016] Figure 3 This is a schematic diagram of a deep learning network configuration based on aspects of this disclosure.

[0017] Figure 4A The following is an example of a kidney ultrasound image based on an aspect of this disclosure.

[0018] Figure 4B This is an example kidney ultrasound image with a target detection box, based on aspects of this disclosure.

[0019] Figure 5 This is a schematic diagram of at least a portion of the model selection and prediction system based on aspects of this disclosure.

[0020] Figure 6A This is a first schematic sequence diagram of the selection process for one or more models based on aspects of this disclosure.

[0021] Figure 6B This is a second schematic sequence diagram of model prediction results and update processes based on aspects of this disclosure.

[0022] Figure 7 This is a schematic diagram illustrating an incoming request based on aspects of this disclosure.

[0023] Figure 8 This is a schematic diagram of a patient case and a second patient case based on aspects of this disclosure.

[0024] Figure 9 This is a first schematic diagram of patient cases in a benchmark truth dataset based on aspects of this disclosure.

[0025] Figure 10This is a first schematic diagram illustrating the generation of model metrics based on patient cases according to aspects of this disclosure.

[0026] Figure 11 This is a second schematic diagram illustrating the generation of model metrics based on patient cases in accordance with aspects of this disclosure.

[0027] Figure 12 This is a second schematic diagram of patient cases in a benchmark truth dataset based on aspects of this disclosure.

[0028] Figure 13 This is a third schematic diagram of patient cases in a benchmark truth dataset based on aspects of this disclosure.

[0029] Figure 14 This is a schematic diagram of at least a portion of the model selection and prediction system based on aspects of this disclosure.

[0030] Figure 15 This is a schematic flowchart of a method for model selection and prediction based on aspects of this disclosure.

[0031] Figure 16 This is a schematic diagram of a model selection and prediction system with a pre-selected model, based on aspects of this disclosure.

[0032] Figure 17 This is a schematic flowchart of a method for model selection and prediction using a pre-selected model, based on aspects of this disclosure.

[0033] Figure 18 This is a schematic diagram of a model selection and prediction system with problem comparison based on aspects of this disclosure.

[0034] Figure 19 This is a schematic flowchart of a method for model selection and prediction with problem comparison based on aspects of this disclosure.

[0035] Figure 20 This disclosure includes examples of the selected model and model results. Detailed Implementation

[0036] According to at least one aspect of this disclosure, a model selection and prediction system is provided, capable of automatically selecting a model and generating a prediction in response to a user's request. For example, this can allow minimally trained users (including general practitioners, nurses, and even patients) to use the models, who otherwise lack the expertise to judge them due to a lack of model transparency or the need for expertise in one or more non-medical fields such as machine learning, data, or science. Furthermore, for multiple models capable of performing the task, the system provides an automated method for selecting models based on their performance on patient cases similar to those requested by the user.

[0037] These descriptions are provided for illustrative purposes only and should not be construed as limiting the scope of model selection and prediction systems. Specific features may be added, deleted, or changed without departing from the spirit of the claimed subject matter.

[0038] For the purpose of facilitating an understanding of the principles of this disclosure, reference will now be made to the aspects illustrated in the accompanying drawings, and they will be described using specific language. However, it should be understood that this is not intended to limit the scope of this disclosure. Any alterations and further modifications to the described apparatus, systems, and methods, and to the principles of this disclosure, are fully contemplated and included in this disclosure, as would normally occur to those skilled in the art. In particular, it is fully contemplated that features, components, and / or steps described with respect to one aspect may be combined with features, components, and / or steps described with respect to other aspects of this disclosure. However, for the sake of brevity, various iterations of these combinations will not be described separately. The term "access" as used herein may include querying, retrieving, sorting, etc.

[0039] The systems and methods disclosed in this paper offer numerous advantages. Some AI image analysis models are rigorously customized to reliably perform specific tasks. For example, AI image models can be age-specific, specific body parts (e.g., organs), specific diseases, etc. Therefore, a large number of models are available, with new models being added regularly. No medical professional can be familiar with all available models, especially for general practitioners covering numerous medical fields; therefore, the systems and methods described in this paper allow medical professionals to access a wider range of models than they would otherwise know to use.

[0040] AI (especially AI image analysis) is gaining popularity due to its expanding applications and increasing complexity. Physicians are becoming increasingly aware of and seeking to utilize AI-based image analysis to help their patients. However, physicians are experts in their own physiological domains, not in the specific AI models they use. While they may know the clinical questions they want answered, they lack the expertise to efficiently and effectively select the appropriate model from a multitude of options to reliably address their clinical needs using AI image processing. Therefore, the systems and methods disclosed in this paper eliminate the need for physicians to choose between different models, facilitating improved patient healthcare.

[0041] Physicians wishing to use AI models must select models relevant to their specific clinical problems. In some instances, one or more AI models may be able to answer clinical questions, but they may not be specifically designed for that type of question. For example, a physician might want to stage a tumor in the liver of a 50-year-old alcoholic, but there may not be a model specifically trained for this purpose. This disclosure helps physicians identify models that work well enough and calculates and outputs a reliability value, which numerically represents what "well enough" means. For example, the systems and methods disclosed herein can determine that a model trained on a 30- to 40-year-old population will work, a model trained on pancreatic images will work, or a model trained on non-alcoholic liver disease is good enough.

[0042] Figure 1 This is a schematic diagram of a network system 100 for model selection and prediction according to aspects of this disclosure. For example, the network system 100 can be used to receive clinical questions from users. The clinical questions can be transmitted to different components of the network system 100 to facilitate model selection and prediction. The network system 100 can provide the user with a prediction result in the form of a diagnosis output from the selected model.

[0043] Network system 100 is used to select one or more models and make predictions using the selected models. Network system 100 may include network / cloud computers 110 and 140, user computer 160, medical imaging console 170, and medical imaging device 180. Network / cloud computers 110 and 140 can communicate with each other and send and / or receive data and information. Figure 1User computer 160 is described as communicating with network / cloud computer 110. However, in some cases, user computer 160 may communicate with one or both of network / cloud computers 110 and 140. User computer 160 may communicate with medical imaging console 170, which in turn communicates with medical imaging device 180. As described herein, communication between different components can be achieved through any number of connections, such as wired and / or wireless connections.

[0044] The network / cloud computer 110 may include a processor 112, an input device 114, a display 116, a communication interface 117, and a memory 118.

[0045] Processor 112 may include a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), controller, field-programmable gate array (FPGA) device, other hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor 112 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Processor 112 is configured to process instructions stored in memory 118. Processor 112 is connected to communication interface 117.

[0046] Input device 114 allows the user to make selections or provide commands to the network / cloud computer 110. Input devices can be mice, touchscreens, touchpads, etc.

[0047] Display 116 is coupled to processor 112. Display 116 can be a monitor or any suitable display. Display 116 is configured to display model A 124 output, model B 128 output, model metadata 126, 130, network performance, or any other system diagnostic information.

[0048] Communication interface 117 is coupled to processor 112. Communication interface 117 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. Communication interface 117 may include hardware and / or software components implementing a specific communication protocol suitable for transmitting signals to other devices in network system 100. Communication interface 117 may be referred to as a communication device or communication interface module.

[0049] Memory 118 is coupled to processor 112. Memory 118 can be any suitable storage device, such as cache memory (e.g., cache memory of processor 112), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory device, hard disk drive, solid-state drive, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0050] Memory 118 may be configured to store object information, measurements, data, or documents relating to the object's medical history, history of surgeries performed, anatomical or biological features, characteristics, or medical conditions associated with the object, computer-readable instructions such as code, software, or other applications, and any other suitable information or data. Object information may include measurements, data, documents, other forms of medical history, such as, but not limited to, ultrasound images, ultrasound videos, and / or any imaging information relating to the object's anatomy. Object information may include parameters relating to the imaging process, such as the anatomical scanning window, probe orientation, and / or the object's position during the imaging process. Memory 118 may also be configured to store information relating to the training and implementation of machine learning algorithms (e.g., neural networks), and / or information relating to the implementation of image recognition algorithms, image quantization algorithms, and / or image acquisition guidance algorithms for detecting / segmenting anatomical structures, including those described herein.

[0051] In some instances, memory 118 includes data and storage instructions for the module, including a model selector 120, a benchmark truth dataset 122, reference locations for AI model A 124, AI model B 128, and AI model C 132. The model selector 120 can select cases similar to the user's request and choose a model to generate the prediction requested by the user.

[0052] Prediction models 124, 128, and 152 can be any neural network-based model or rule-based programming model, for example, Figure 3As shown in the diagram. For example, models 124, 128, and 152 can be convolutional neural networks (CNNs), decision tree models, support vector machines (SVMs), generative image-to-image transformers, generative image-to-text transformers (GITs), or other transformer-based models. A 124 may contain model metadata 126. AI model B may contain model metadata 130. The reference location to model C 132 details the location of AI model C on the network / user computer 140. Metadata may include the data format types of the inputs and outputs to the model, a description of the task on which the model is trained, etc. The benchmark ground truth dataset 122 includes multiple patient case data and information. Furthermore, the benchmark ground truth dataset is not the same as the training dataset 195. The training dataset 195 includes data used to train models 126, 130, and model C 150 on the network / cloud computer 140. For example, the training dataset may contain pairs of anatomical features and segmented images that identify those anatomical features. The benchmark ground truth dataset includes annotations and benchmark ground truth labels 190 for patient case data and information. For example, the baseline ground truth label can be a disease stage classification, a target detection box for anatomical structure recognition and detection, or an anatomical structure contour for anatomical structure segmentation.

[0053] The network / cloud computer 140 may include a processor 142, an input device 144, a display 146, a communication interface 147, and a memory 148.

[0054] Processor 142 may include a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), controller, field-programmable gate array (FPGA) device, other hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor 142 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Processor 142 is configured to process instructions stored in memory 148. Processor 142 is connected to communication interface 147.

[0055] Input device 144 allows the user to make selections or provide instructions to network / cloud computer 140. Input device may be a mouse, touch screen, touchpad, etc.

[0056] Display 146 is coupled to processor 142. Display 146 can be a monitor or any suitable display. Display 146 is configured to display model C 150 output, model metadata 152 network performance, or any other system diagnostic information.

[0057] Communication interface 147 is coupled to processor 142. Communication interface 147 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. Communication interface 147 may include hardware and / or software components implementing a specific communication protocol suitable for transmitting signals to other devices in network system 100. Communication interface 147 may be referred to as a communication device or communication interface module.

[0058] Memory 148 is coupled to processor 142. Memory 148 can be any suitable storage device, such as cache memory (e.g., cache memory of processor 142), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory device, hard disk drive, solid-state drive, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0059] Memory 148 may be configured to store object information, measurements, data, or documents relating to the object's medical history, history of surgeries performed, anatomical or biological features, characteristics, or medical conditions associated with the object, computer-readable instructions such as code, software, or other applications, and any other suitable information or data. Object information may include measurements, data, documents, other forms of medical history, such as, but not limited to, ultrasound images, ultrasound videos, and / or any imaging information relating to the object's anatomy. Object information may include parameters relating to the imaging process, such as the anatomical scanning window, probe orientation, and / or the object's position during the imaging process. Memory 148 may also be configured to store information relating to the training and implementation of machine learning algorithms (e.g., neural networks), and / or information relating to the implementation of image recognition algorithms, image quantization algorithms, and / or image acquisition guidance algorithms for detecting anatomical structures, segmenting anatomical structures (e.g., defining the contours of anatomical structures), including those described herein.

[0060] In some instances, memory 148 contains data and storage instructions for the module, including a model selector AI model C 132 and associated model C metadata 152. For example... Figure 1As shown, model C 132 resides on a different network / cloud computer than models A and B 124, 128. During model selection and prediction, user requests and patient cases from the baseline ground truth dataset 122 can be sent to network / cloud computer 140. Model C 150 can generate outputs and predictions based on the received data, and can then send these outputs and predictions back to one or more computers and devices in network system 100.

[0061] User computer 160 may include processor 162, input device 164, display 166, communication interface 167, and memory 168.

[0062] Processor 162 may include a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), controller, field-programmable gate array (FPGA) device, other hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor 162 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Processor 162 is configured to process instructions stored in memory 168. Processor 162 is connected to communication interface 167.

[0063] Input device 164 allows the user to make selections or provide instructions to the user computer 160. Input device may be a mouse, touch screen, touchpad, etc.

[0064] Display 166 is coupled to processor 162. Display 166 can be a monitor or any suitable display. Display 166 is configured to display the outputs of model A 124, model B 128, model C 150, model metrics, and information, as described herein. Furthermore, display 166 can be configured to display images from data generated by medical imaging device 180.

[0065] Communication interface 167 is coupled to processor 162. Communication interface 167 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. Communication interface 167 may include hardware and / or software components implementing a specific communication protocol suitable for transmitting signals to other devices in network system 100. Communication interface 167 may be referred to as a communication device or communication interface module.

[0066] Memory 168 is coupled to processor 162. Memory 168 can be any suitable storage device, such as cache memory (e.g., cache memory of processor 162), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory device, hard disk drive, solid-state drive, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0067] Memory 168 may be configured to store object information, measurements, data, or documents relating to the object's medical history, history of surgeries performed, anatomical or biological features, characteristics, or medical conditions associated with the object, computer-readable instructions such as code, software, or other applications, and any other suitable information or data. Object information may include measurements, data, documents, other forms of medical history, such as, but not limited to, ultrasound images, ultrasound videos, and / or any imaging information relating to the object's anatomy. Object information may include parameters relating to the imaging process, such as the anatomical scanning window, probe orientation, and / or the object's position during the imaging process. Memory 168 may also be configured to store information relating to the training and implementation of machine learning algorithms (e.g., neural networks), and / or information relating to the implementation of image recognition algorithms, image quantization algorithms, and / or image acquisition guidance algorithms for detecting anatomical structures, segmenting anatomical structures, and / or the algorithms described herein.

[0068] The medical imaging console 170 may include a processor 172, an input device 174, a display 176, a communication interface 177, and a memory 178.

[0069] Processor 172 may include a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), controller, field-programmable gate array (FPGA) device, other hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor 172 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Processor 172 is configured to process instructions stored in memory 178. Processor 172 is connected to communication interface 177.

[0070] Input device 174 allows the user to make selections or provide instructions to the user computer 170. Input device may be a mouse, touch screen, touchpad, etc.

[0071] Display 176 is coupled to processor 172. Display 176 may be a monitor or any suitable display. Display 176 is configured to display the outputs of model A 124, model B 128, model C 150, model metrics, and information, as described herein. Furthermore, display 176 may be configured to display images from data generated by medical imaging device 180.

[0072] Communication interface 177 is coupled to processor 172. Communication interface 177 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. Communication interface 177 may include hardware and / or software components implementing a specific communication protocol suitable for transmitting signals to other devices in network system 100. Communication interface 177 may be referred to as a communication device or communication interface module.

[0073] Memory 178 is coupled to processor 172. Memory 178 can be any suitable storage device, such as cache memory (e.g., cache memory of processor 172), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory device, hard disk drive, solid-state drive, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0074] Memory 178 may be configured to store object information, measurements, data, or documents relating to the object's medical history, history of surgeries performed, anatomical or biological features, characteristics, or medical conditions associated with the object, computer-readable instructions such as code, software, or other applications, and any other suitable information or data. Object information may include measurements, data, documents, other forms of medical history, such as, but not limited to, ultrasound images, ultrasound videos, and / or any imaging information relating to the object's anatomy. Object information may include parameters relating to the imaging process, such as the anatomical scanning window, probe orientation, and / or the object's position during the imaging process. Memory 178 may also be configured to store information relating to the training and implementation of machine learning algorithms (e.g., neural networks), and / or information relating to the implementation of image recognition algorithms, image quantization algorithms, and / or image acquisition guidance algorithms for detecting anatomical structures, segmenting anatomical structures, and / or the algorithms described herein.

[0075] Medical imaging device 180 can generate medical images 185. The medical imaging device communicates with medical imaging console 170, for example, via communication interface 177. Medical images generated from data produced by medical imaging device 180 can be displayed on display 176. Medical images 185 can also be transmitted to any computer, network / cloud 110, 140, 160.

[0076] In some aspects, aspects of this disclosure can be achieved by obtaining medical images 185 of the object using any suitable medical imaging device 180 and / or modality. Examples of medical images and medical imaging devices include: X-ray images (angiographic images, fluoroscopic images, images with or without contrast agents) obtained by medical imaging devices such as ultrasound imaging devices and X-ray imaging devices; computed tomography (CT) images obtained by CT imaging devices; PET-CT images obtained by positron emission tomography-computed tomography (PET-CT) imaging devices; magnetic resonance images (MRI) obtained by MRI imaging devices; single-photon emission computed tomography (SPECT) images obtained by SPECT imaging devices; optical coherence tomography (OCT) images obtained by OCT imaging devices; and intravascular photoacoustic (IVPA) images obtained by IVPA imaging devices. The medical imaging device 180 can obtain medical images when located outside the object's body, at a distance from the object's body, adjacent to the object's body, in contact with the object's body, and / or inside the object's body.

[0077] Before proceeding, it should be noted that the examples described above are for illustrative purposes only and are not intended to be limiting. Other devices and / or device configurations can be used to perform the operations described herein.

[0078] Figure 2 This is a schematic diagram of processor circuitry 250 according to an embodiment of the present disclosure. Processor circuitry 250 can be implemented in network / cloud computers 110, 140, user computer 160, imaging console 170, or other devices or workstations (e.g., third-party workstations, network routers, etc.), or on a cloud processor or other remote processing unit, to implement the method as needed. As shown, processor circuitry 250 may include processor 260, memory 264, and communication interface 268. These components can communicate directly or indirectly with each other, for example, via one or more buses.

[0079] Processor 260 may include a central processing unit (CPU), a digital signal processor (DSP), an ASIC, a controller, or any combination of general-purpose computing devices, reduced instruction set computing (RISC) devices, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other related logic devices, including mechanical computers and quantum computers. Processor 260 may also include other hardware devices, firmware devices, or any combination thereof configured to perform the operations described herein. Processor 260 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0080] Memory 264 may include cache memory (e.g., the cache memory of processor 260), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state storage devices, hard disk drives, other forms of volatile and non-volatile memory, or combinations of different types of memory. In one aspect, memory 264 includes a non-transitory computer-readable medium. Memory 264 may store instructions 266. Instructions 266 may include instructions that, when executed by processor 260, cause processor 260 to perform the operations described herein. Instructions 266 may also be referred to as code. The terms “instruction” and “code” should be interpreted broadly to include any type of computer-readable statement(s). For example, the terms “instruction” and “code” may refer to one or more programs, routines, subroutines, functions, procedures, etc. “Instruction” and “code” may include a single computer-readable statement or multiple computer-readable statements.

[0081] Communication interface 268 may contain any electronic and / or logic circuitry to facilitate direct or indirect data communication between processor circuit 250 and other processors or devices. In this regard, communication interface 268 may be an input / output (I / O) device. In some instances, communication interface 268 facilitates direct or indirect communication between various components of processor circuit 250 and / or the computer of network system 100. Communication interface 268 can communicate within processor circuit 250 using various methods or protocols. Serial communication protocols may include, but are not limited to, US Serial Protocol Interface (US SPI), Inter-Integrated Circuit Protocol (IIP), etc. 2C) Recommended standards RS-232, RS-485, Controller Area Network (CAN), Ethernet, ARINC 429, MODBUS, MIL-STD-1553, or any other suitable method or protocol. Parallel protocols include, but are not limited to, Industry Standard Architecture (ISA), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI), IEEE 488, IEEE 1284, and other suitable protocols. Where appropriate, serial and parallel communications can be bridged using a Universal Asynchronous Receiver / Transmitter (UART), a Universal Synchronous Receiver / Transmitter (USART), or other suitable subsystems.

[0082] External communication (including but not limited to software updates, firmware updates, model sharing between the processor and the central server, or output generated by the model selection and prediction system described herein) can be implemented using any suitable wireless or wired communication technology, such as cable interfaces (e.g., Universal Serial Bus (USB), micro USB, Lightning, or FireWire interfaces), Bluetooth, Wi-Fi, ZigBee, Li-Fi, or cellular data connections (e.g., 2G / GSM (Global System for Mobile Communications), 3G / UMTS (Universal Mobile Communications System), 4G, LTE, WiMax, or 5G). For example, Bluetooth Low Energy (BLE) radios can be used to establish connections with cloud services, transmit data, and receive software patches. The controller can be configured to communicate with remote servers or local devices (e.g., laptops, tablets, or handheld devices), or may include a display capable of showing state variables and other information. Information can also be transmitted via physical media, such as USB flash drives or memory sticks.

[0083] Figure 3 This is a schematic diagram of a deep learning network configuration according to aspects of this disclosure. Configuration 300 can be implemented using a deep learning network. Configuration 300 includes a deep learning network 310, which includes one or more CNNs 312. CNN 312 is an example of a prediction model, such as AI model A 124, AI model B 128, and AI model C 150. For ease of illustration and discussion, Figure 3A CNN 312 is illustrated. However, these embodiments can be extended to include any suitable number of CNNs 312 (e.g., approximately two, three, or more). Configuration 300 can be trained to recognize various anatomical structures (organs, tissues, bones) and / or other features (natural and / or artificial) in a patient's anatomy. Configuration 300 can also be trained to segment human anatomy, diagnose medical conditions, or perform any number of other diagnostic or medical tasks.

[0084] CNN 312 may include a set of N convolutional layers 320, followed by a set of K fully connected layers 330, where N and K can be any positive integers. The convolutional layers 320 are shown as 320(1) to 320(N). The fully connected layers 330 are shown as 330(1) to 330(K). Each convolutional layer 320 may include a set of filters 322 configured to extract features (e.g., X-ray venography images or other supplementary data) from the input 302. Depending on the embodiment, the values ​​of N and K and the size of the filters 322 may vary. In some instances, the convolutional layers 320(1) to 320(N) and the fully connected layers 330(1) to 330(K-1) may use a leak-corrected nonlinear (ReLU) activation function and / or batch normalization. The fully connected layers 330 may be nonlinear and may progressively reduce the high-dimensional output to the dimension of the prediction result 340 (e.g., the location of the object detection box or other classification output). The fully connected layers 330 may also be referred to as classifiers. In some embodiments, the fully convolutional layer 320 may also be referred to as a perception layer or a perceptual layer.

[0085] When the prediction result 340 appears as a classification output, it can indicate the confidence score of each category 342 based on the input image 302. Categories 342 are shown as 342a, 342b, ..., 342c. For example, when the CNN 312 is trained for narrow areas or general venous compression, category 342 can indicate the inguinal ligament category 342a, the crossing category 342b, the pelvic bone notch category 342c, the blood flow restriction area category 342d, or any other suitable category. A category 342 indicating a high confidence score indicates that the input image 302, or a portion of image 302, or a pixel of image 302, is likely to contain the anatomical target / feature of category 342. Conversely, a category 342 indicating a low confidence score indicates that the input image 302, or a portion of image 302, or a pixel of image 302, is unlikely to contain the anatomical target / feature of category 342.

[0086] The deep learning network 310 can implement or include any suitable type of learning network. For example, in some embodiments, and as per [reference to...] Figure 3As described, the deep learning network 310 may include a convolutional neural network 312. Furthermore, the convolutional neural network 310 may also, or alternatively, be or include a multi-class classification network, an encoder-decoder type network, or any suitable network or method for recognizing features within an image.

[0087] In an embodiment where the deep learning network 310 includes an encoder-decoder network, the network may include two paths. One path may be a shrinking path, where a large image (e.g., image 302) is convolved through multiple convolutional layers 320 such that the size of image 302 varies with the depth of the network layers. Image 302 can then be represented in a low-dimensional space or a flattened space. From this flattened space, an additional path can expand the flattened space to the original size of image 302. In some embodiments, the implemented encoder-decoder network may also be referred to as a principal component analysis (PCA) method. In some embodiments, the encoder-decoder network can segment image 302 into blocks. In an additional embodiment of this disclosure, the deep learning network 310 may include a multi-class classification network. In this embodiment, the multi-class classification network may include an encoder path. For example, image 302 may be a high-dimensional image. Image 302 can then be processed using convolutional layers 320 to reduce its size. The resulting low-dimensional representation of image 302 can be used to generate... Figure 3 The feature vector 350 is shown. The low-dimensional representation of image 302 can also be used by fully connected layer 330 for regression and outputting one or more categories 342. In some respects, fully connected layer 330 can process the output of encoder or convolutional layer 320. Among other terms, fully connected layer 330 can also be called task layer or regression layer.

[0088] Full consideration has been given to any suitable combinations or variations of the described deep learning network 310. For example, the deep learning network may include a fully convolutional network or layers, or a fully connected network or layers, or a combination of both. Furthermore, the deep learning network may include a multi-class classification network, an encoder-decoder network, or a combination of both.

[0089] Figure 4A Example of a kidney ultrasound image 400 according to an aspect of this disclosure. As shown, ultrasound image 400 depicts human anatomical structures, including the kidney and adjacent tissues. Image 400 may be an example of an image captured by a user-operated medical imaging device 180 (e.g., an ultrasound probe). A user may provide image 500 to the model selection and prediction system described herein and request anatomical structure detection on that image. Image 400 may be included in a benchmark ground truth dataset 122 or a training dataset 195, and in some cases, the image may be included in only one of these two datasets.

[0090] Figure 4B This is an example kidney ultrasound image 450 with a target detection box, according to aspects of this disclosure. As shown, ultrasound image 450 depicts human anatomical structures, including the kidney and adjacent tissues. Furthermore, ultrasound image 450 also includes a target detection box 460. The target detection box 460 can be generated by a target detection model (e.g., Figure 1 (One of the AI ​​models A, B, or C in the image) from the image (e.g., Figure 4A Image 400 in the image is generated. Object detection bounding boxes 460 surround anatomical structures, such as the kidney in image 450, and can be specified by the user for localization and / or recognition, or the user can instruct the object detection model to recognize objects in image 400. In some instances, various features 470 are associated with object detection bounding boxes 460. For example, features 470 may include objects detected by the object detection model, the confidence level of the object detection model in the generated detection results, the area of ​​the object detection box, or the location of the object detection box (e.g., given by the coordinates of the centroid of the segmentation box). In some instances, features 470 may be output to the user and displayed along with the segmented image 450 (e.g., in...). Figure 1 On the monitor 166 of the user computer 160.

[0091] Figure 5 This is a schematic diagram of at least a portion of a model selection and prediction system 500 according to aspects of this disclosure. Given user input, the model selection and prediction system 500 selects a model based on various conditions and metrics and then uses the selected model to make a prediction. The user input may take the form of a medical image and / or a clinical task, and the output may be a response to the clinical task. The user may be any of several medical professionals. Alternatively, the user input may not require the user to generate and send the input; program instructions may automatically generate the user input based on received images. The model selection and prediction system 500 may include a model selector 120, a benchmark ground truth dataset 122, model A 124, model B 128, and model C 150.

[0092] The model selection and prediction system 500 can receive an incoming request 504 from a user computer 160 or a medical imaging console 170. The incoming request 504 may include a clinical question 506 and first patient case data 508. In some instances, the clinical question may include medical images and / or clinical tasks, such as... Figure 7 Further description.

[0093] The model selector 120 may include a similar case selection module 510 and a model metric module 530. The similar case selection module 510 identifies a set of cases similar to the received information contained in the incoming request 504 based on one or more metrics. The model metric module 530 identifies one or more models for selection based on one or more metrics, user selection, model metadata, etc.

[0094] The similar case selection module 510 may include a similarity metric 512 and weights 514. The similarity metric 512 measures how closely one patient case is similar to another. The similarity metric 512 may depend on the patient's age, weight, height, and other characteristics. The weights 514 may be parameters used in the similarity metric to generate a numerical value for the similarity metric 510 given two or more cases. For example, the similarity metric may be derived from a function... Defined, and expressed by the following mathematical expression:

[0095] in, This can represent a weight of 514. It can represent the case parameters of patient cases in the benchmark truth dataset 122, and This represents the case parameters for an incoming 504 request. Case similarity metrics can be defined as the distance between two points in a multidimensional space, where the dimensions represent different aspects of a case, represented by numerical or binary (one-hot) encoding (e.g., patient age, disease type, body part, clinical problem, disease progression time, etc.), and each point represents a patient case.

[0096] The model metrics module 530 may include an outcome distance metric 532, a reliability metric 534, a robustness metric 536, and weights 538. The outcome distance metric 532 determines the distance between the baseline ground truth labels of cases in subsets of similar cases and the labels contained in the model outputs 525 generated by models 124, 128, and 150. The reliability metric 534 measures the reliability (i.e., 190) of each model's performance in generating outputs matching the baseline ground truth labels for subsets of similar cases 520. The robustness metric 536 measures the robustness of the model's performance on different subsets of the baseline ground truth data. In other words, the robustness metric indicates the model's sensitivity to inputs that are close in the sense of similarity metrics described herein. These weights are used as parameters for the outcome distance metric 532, the reliability metric 534, and the robustness metric 536 to generate numerical values ​​for the metrics based on the model outputs and the baseline ground truth labels.

[0097] The similar case selection module 510 can receive the incoming request 504 and generate a subset 520 of similar cases in the benchmark truth dataset 122 that are similar to the patient cases described in the incoming request 504. For example, the similar case selection module 510 can select all cases in the benchmark truth dataset 122 whose similarity metric values ​​are less than the maximum value. The maximum value can be pre-programmed into the module or determined directly or indirectly based on the incoming request 504.

[0098] Each of Model A 124, Model B 128, and Model C 150 can receive cases from a subset 520 of similar cases and generate a model output 525. For example, the model output 525 can take the form of image segmentation. The model output 525 can take any form, depending on the clinical task undertaken by the model, such as diagnosis, segmentation, classification, etc. The model selection and prediction system 500 can contain different numbers of models. For example, selection can be made between two models, or it can be made between four and twenty models, and / or other larger or smaller values ​​can be selected.

[0099] The model measurement module 530 can receive model output 525 and generate selected model output 540, reliability value 545, and / or robustness value 550. The model measurement module 530 can select a model, such as one or more of model A 124, model B 128, and model C 150. The model measurement module can also use the selected model to generate the selected model output 540 based on an incoming request. The model measurement module 530 can generate a reliability value 545 using a reliability metric 534 and weights 538. The model measurement module 530 can generate a robustness value 550 using a robustness metric 534 and weights 538.

[0100] User computer 160 or medical imaging console 170 can receive the selected model output 540, reliability value 545, and / or robustness value 550. User computer 160 or medical imaging console 170 can display and / or post-process information from model selection and prediction system 500. When a user (e.g., a physician) interacts with the display, they can evaluate the results generated by the model selection and prediction system based on incoming request 504. The user can provide user feedback 560 to model selection and prediction system 500 (in some cases, to model selector 120) to evaluate the system's performance. User feedback 560 may also include parameters or other options to directly or indirectly change the weights 514, 538 or metrics 512, 532, 534, or 536 in model selector 120. The user can operate iteratively using model selection and prediction system 500 to obtain the desired clinical outcome. Furthermore, the model can be fine-tuned based on user feedback on performance.

[0101] Figure 6A and 6B This is a schematic sequence diagram of the process of a model selection and prediction system 600 according to the present disclosure. The model selection and prediction system may include a user computer 160 or a medical imaging console 170, a model selector 120, a benchmark truth dataset 122, model A 124, and model B 128.

[0102] In step 602, the user computer 160 or medical imaging device 170 may send clinical questions and first patient case data to the model selector 120.

[0103] In step 604, the model selector 120 can access the benchmark truth dataset 122.

[0104] In step 606, the model selector 120 can receive second patient case data from the benchmark truth dataset.

[0105] In step 608, model selector 120 can determine whether the second patient case belongs to a subset 520 of similar cases to the clinical questions and first patient data received by model selector 120 in step 602.

[0106] In step 610, a subset of cases in the benchmark truth dataset 122 is determined. For example, Figure 5 A subset of 520 similar cases.

[0107] In step 612, model A 124 receives a subset 520 of similar cases as input.

[0108] In step 614, model B 128 receives a subset 520 of similar cases as input.

[0109] In step 616, the model selector 120 retrieves the baseline truth labels of a subset of similar cases from the baseline truth dataset 122.

[0110] In step 618, model A 124 outputs its prediction results to model selector 120.

[0111] In step 620, model B 128 outputs its prediction results to model selector 120.

[0112] In step 622, the model selector 120 determines the model metric. For example, the model metric can be determined using the result distance metric 532, the reliability metric 534, the robustness metric 536, and the weights 538, based on the prediction results of models A and B and the baseline truth labels of a subset of similar cases.

[0113] In step 624, the model selector 624 uses model metrics to determine the selected model. For example, model A124 may have the highest reliability and robustness values ​​and is selected based on these values.

[0114] In step 626, model selector 120 provides a clinical question to the selected model (model A 124). The selection of model A is just an example; one or more other models can also be selected, including... Figure 6B The model is not shown in the figure.

[0115] In step 628, the selected model, model A 124, outputs a prediction to model selector 120 based on the clinical question. For example, model A 124 can generate images with anatomical structure detection, such as... Figure 4B The image in the image.

[0116] In step 630, the model selector 120 transmits the selected model prediction results and one or more model metrics for the selected model to the user computer 160 or the medical imaging device 170.

[0117] In step 632, the user computer 160 or medical imaging device 170 displays the selected model prediction results and one or more model metrics for the selected model 632. Figure 20 An example of what the user displays is shown.

[0118] In step 634, the user computer 160 or medical imaging device 170 transmits user feedback 560 to the model selector 120.

[0119] In step 636, the model selector 120 can update the determination of the selected model. For example, the model metric 530 can be updated.

[0120] Figure 7 This is a schematic diagram of incoming request 504 drawn according to various aspects of this disclosure. An incoming request may contain information provided by a user (including physicians, nurses, medical technicians, and other medical professionals and support staff). An incoming request may contain one or more clinical questions 506 and patient case data 508.

[0121] Clinical problem 506 may include a medical image 710 and a clinical task 720. Medical image 710 may include any image from various imaging modalities / systems, such as ultrasound, MRI, CT, PET, X-ray, etc. Furthermore, medical images may be digital images taken by conventional cameras. In some instances, image metadata 712 may be associated with medical image 710, which may include timestamps, dates, imaging system settings, etc. Clinical task 720 may include task type 722 and task query 724. Task type 722 may be a category of the task. Some examples of task types include anatomical structure detection, anatomical structure segmentation, disease stage classification, etc. Task query 724 instructs the user of the model selection and prediction system what they are trying to determine. For example, a user may want to identify a kidney from an unlabeled ultrasound image. In this example, the user provides an ultrasound image of a human anatomy (e.g., medical image 710) and asks, “Identify the kidney in the image.” Query 724 can take various forms, including affirmative commands or questions such as, “What stage is the disease in?”

[0122] Patient case data 508 can contain any number of detailed patient information, including biomarkers such as name, date of birth, age, weight, medical history, current medications, and allergies. Patient case data 508 can also contain a type of information called case parameters. As used in this paper, case parameters can be used by model selection and prediction systems to generate subsets of similar cases. For example, ... Figure 9 and Figure 12-13 As shown, case parameters such as age and weight are used to construct patient cases to determine their similarity to the case data contained in Clinical Question 506.

[0123] Figure 8 This is a diagram illustrating patient cases and second patient cases based on aspects of this disclosure. (See diagram for example.) Figure 8 As shown, the first patient case includes medical images 805 and patient case data 810, which users can use in some cases to perform clinical tasks (e.g., Figure 7 The second patient case 850 is a case in the baseline ground truth dataset 122, submitted together with the model prediction results and selection system for processing. Compared to the first patient case 800, the second patient case 850 is a case in the baseline ground truth dataset 122. While both the first and second patient cases 800 contain medical images 805, 855, patient case data 810, 860, and associated image metadata 815, 870, and case parameters 820, 875, the second patient case 850 may also contain annotations / benchmark ground truth labels 865 associated with the medical images (e.g., ...). Figure 1 (190 in the text). Annotation / benchmark truth labels can include anatomical structure detection boxes, segmentation contours, anatomical structure classification labels, disease stage classification labels, etc.

[0124] Figure 9 This is a first schematic diagram 900 of patient cases in a benchmark ground truth dataset 122 according to some aspects of this disclosure. Each marker in Figure 900 represents a patient case in the benchmark ground truth dataset 122. As shown, the horizontal axis of the graph represents case parameter A (e.g., age), and the vertical axis represents case parameter B (e.g., weight). A subset 905 of similar cases is identified using pattern markers. The subset 905 of similar cases surrounds a first patient case 910, where the first patient case may be a case submitted by a user to a model selection and prediction system. The components of the first patient case vector 915 of the first patient case 920 are the case parameters A and B of the first patient case 910. Similarly, the second patient case 920 has a second patient case vector 925, the components of which are the case parameters A and B of the second patient case.

[0125] In some instances, the distance between the case depicted in Figure 900 and the first patient case 910 determines, at least to some extent, whether that case is selected into a subset of similar cases (e.g., Figure 5 (520 in the original text). The threshold distance 930 can be determined based on previous similarity subset selection, calculated based on one or more parameters (e.g., user-provided medical image quality, task type, task query, image metadata, etc.), or provided by the user. Cases that are sufficiently close to the first patient case 910 can be selected for inclusion in the similar cases subset. For example, a distance of 935 indicates that the case should be included in the similar cases subset, while a distance of 940 indicates that the case should not be included in the similar cases subset.

[0126] Figure 10 This is a first schematic diagram illustrating the generation of model metrics for a model based on a subset 905 of similar cases, according to aspects of this disclosure. Generating model metrics for each model allows for comparison of model performance and allows the model selection and prediction system to select one or more models for prediction based on user-input requests. The generated model metrics may include the subset 905 of similar cases and model A 124. For example, the subset 905 of similar cases may include case I 1002, case II 1004, and case III 1006. Each case may respectively include medical images 1008, 1012, 1016 and baseline ground truth values ​​1010, 1014, 1018.

[0127] In some instances, model A 124 receives medical images 1008, 1012, and 1016 for each case 1002, 1004, and 1006, and generates outputs 1022, 1024, and 1026. For example, the output could be a segmentation of the medical image (e.g., identifying anatomical structure outlines in the image). Model A 124 can be any number of models described herein with respect to any model. For example, model A 124 can include neural networks, such as convolutional neural networks, Transformer networks, recurrent neural networks, or other models described herein.

[0128] The outcome distance values ​​1032, 1034, and 1036 can be calculated based on the outputs 1022, 1024, and 1026 of model A and the baseline ground truth values ​​1010, 1014, and 1018 for each case. The outcome distance metric can take different forms. For example, for a segmentation task, the outcome distance metric could be the percentage of the segmented region of the baseline value to the overlapping area of ​​the segments from the model output, while for a task with numerical labels, the outcome distance metric could include the squared difference between the baseline ground truth label and the predicted label. If the outcome is a binary decision (e.g., disease detected yes / no), the outcome distance metric can be defined as 0 when equal and 1 when unequal. If the outcome consists of numerical values ​​(e.g., number of lesions), the metric can be defined as the difference between the numerical values ​​derived from the model and the baseline ground truth. As described in this paper, the reliability value 1040 and robustness value 1050 of model A can be calculated based on the outcome distance values ​​1032, 1034, and 1036. For example, the reliability value could be a weighted or unweighted average of the resulting distance values ​​1032, 1034, and 1036, and the robustness value could be the standard deviation of the resulting distance values ​​1032, 1034, and 1036. The reliability value could also be the reciprocal of the mean distance, the reciprocal of the median distance, the reciprocal of the weighted sum of the mean and standard deviation of the distances, or a similar definition.

[0129] Figure 11 This is a second schematic diagram illustrating the generation of model metrics based on patient cases according to aspects of this disclosure. For example... Figure 11 As shown, model metrics can be generated from multiple distinct subsets of similar cases. One or more of these subsets can be subsets of another larger subset. Alternatively, two or more subsets may not contain any common patient cases, i.e., they have no overlap. Generating model metrics for each model allows for comparison of model performance and allows for the selection of one or more models for prediction based on user-input requests. Generating model metrics can include multiple subsets of similar cases. Figure 11Three subsets are described: subset I 1104 of similar cases, subset II 1114 of similar cases, and subset III 1124 of similar cases. For example, subset I 1104 of similar cases may include case I 1106 and case II 1108; subset II 1114 of similar cases may include case I' 1116 and case II' 1118; and subset III of similar cases may include case I'' 1126 and case II'' 1128.

[0130] In some instances, alternative subset selection logic 1110 can be used to generate subset II 1114 of similar cases and subset III 1124 of similar cases. For example, as... Figure 9 As shown, the threshold distance can be varied to generate new subsets. As another example, cases can be selected within a second threshold distance, where the distance is less than the first threshold distance, and the points in the subset corresponding to the first threshold distance are included. This can be extended to multiple different threshold distances centered at multiple distinct points. As yet another example, a new subset of similar cases can be generated by randomly sampling from a first subset of similar cases. Regarding... Figure 12 and Figure 13 Two of these examples are depicted and described in the text.

[0131] and Figure 10 Similarly, model A (not shown) can generate outputs based on cases 1106, 1108, 1116, 1118, 1126, and 1128. For example, the output could be a segmentation of a medical image. Model A 124 can be any number of models described herein with respect to any model.

[0132] As described herein, result distance values ​​can be calculated based on the result distance metric, corresponding to subsets I 1104 of similar cases, i.e., result distance values ​​1136 and 1138; subset II 1114 of similar cases, i.e., result distance values ​​1146 and 1148; and subset III 1124 of similar cases, i.e., result distance values ​​1156 and 1158. The result distance metric can take the mathematical form described herein. For example, the result distance metric for a segmentation task can be the percentage of overlap between the true segmented region and the model output segmented region. The reliability value 1160 and robustness value 1170 of model A can be calculated based on the result distance values ​​1136, 1138, 1146, 1148, 1156, and 1158 described herein. For each subset of similar cases, the reliability value 1160 can be calculated by averaging the result distance values ​​calculated for that subset. The robustness value can be calculated in several ways. For example, the robustness value 1170 can be calculated as a weighted or unweighted average of the reliability values ​​for each subset. Alternatively, the robustness value can be calculated as the standard deviation of the reliability value for each subset.

[0133] Figure 12 This is a second schematic diagram 900 of patient cases in a benchmark truth dataset 122 according to some aspects of this disclosure. Each label in Figure 900 represents a patient case in the benchmark truth dataset 122. As shown, the horizontal axis of the figure represents case parameter A (e.g., age), and the vertical axis represents case parameter B (e.g., weight). Figure 900 depicts three subsets 1205, 1210, and 1215 of similar cases.

[0134] In some instances, the three subsets of similar cases, 1205, 1210, and 1215, correspond to Figure 11 The subsets I-III1104, 1114, and 1124 are used, among which subsets 1210 and 1215 are used. Figure 11 The alternative subset selection logic 1110 in the code generates the subset. Similar case subsets 1205 (similar to pattern tags) are then used to mark the subsets. Figure 9 (The subset shown). The first subset 1205 of similar cases surrounds the first patient case 910, where the first patient case can be a case submitted by the user to the model selection and prediction system. In some instances, subsets 1210, 1215 are as shown regarding... Figure 11 The subset is generated as described, i.e., by using second and third threshold distances, both of which are less than the first threshold distance associated with subset 1205. It should be understood that the shape of the subset does not have to be a disk (or even approximate a disk); any geometry can be used. Alternatively, a subset of similar cases can be generated by random sampling based on a probability distribution, where the probability of being close to the first patient case received from the user (e.g., 910) is higher.

[0135] Figure 13 This is a third schematic diagram 900 of patient cases in a benchmark truth dataset 122 based on some aspects of this disclosure. Each label in Figure 900 represents a patient case in the benchmark truth dataset 122. As shown, the horizontal axis of the figure represents case parameter A (e.g., age), and the vertical axis represents case parameter B (e.g., weight). Figure 900 depicts three subsets 1305, 1310, and 1315 of similar cases.

[0136] In some instances, the three subsets of similar cases, 1305, 1310, and 1315, correspond to Figure 11 The subsets I-III are 1104, 1114, and 1124, of which subsets 1305, 1310, and 1315 are used. Figure 11 The alternative subset selection logic 1110 generates subsets 1305, 1310, and 1315, all centered on the first patient case 910, which can be a case submitted by the user to the model selection and prediction system. In some instances, subsets 1305, 1310, and 1315 are combined... Figure 11 The subset is generated as described herein, i.e., by using multiple different threshold distances measured for the first patient case 910. It should be understood that the shape of the subset does not have to be a disk (or approximate a disk); any geometry can be used. Alternatively, a subset of similar cases can be generated by random sampling based on a probability distribution, where the probability of being closer to the user's first patient case is higher.

[0137] Figure 14 This is a schematic block diagram of at least a portion of a model selection and prediction system 1400 according to aspects of this disclosure. The model selection and prediction system 1400 can select one or more models to generate predictions in response to clinical questions. The model selection and prediction system 1400 may include models 124, 128, 150, model selection logic 1432, a combination block 1434, and a display 1450.

[0138] Such as about Figure 10 As shown and described, Model A 124, Model B 128, and Model C 150 generate predictions (one or more) based on subsets of similar cases. Reliability values ​​of 1405, 1415, and 1425 and robustness values ​​of 1410, 1420, and 1430 can be determined for each of Models A, B, and C, respectively. In some instances, fewer models may be used; and in others, more models may be used.

[0139] The model selection logic 1432 can receive reliability values ​​1405, 1415, and 1425, and robustness values ​​1410, 1420, and 1430. Based on the reliability and robustness values, the model selection logic 1432 selects one or more models. (Illustrated explanation follows.) Figure 14 Two configurations are described. In the first configuration, model selection logic 1432 selects a single model, such as model A124. In the second configuration, model selection logic selects two models, namely model A124 and model B128.

[0140] In the first configuration, model A 124 receives clinical question 506 and generates output 1435, as described herein. The output 1435 of model A can be displayed to a user via display 1450.

[0141] In the second configuration, both Model A 124 and Model B 128 receive clinical question 506 and generate corresponding outputs 1435 and 1440. This second configuration (where multiple models are selected) can be referred to as the auxiliary configuration. Next, the outputs 1435 of Model A and 1440 of Model B can be received by the combination block 1434. The combination block 1434 generates an output by combining the outputs 1435 and 1440. In some instances, the combination block 1434 may include at least a portion of a neural network trained using ensemble learning. Therefore, the combination block 1434 can internally generate weights for combining the outputs 1435 and 1440. In other cases, the combination block 1434 can calculate the average of the outputs 1435 and 1440. For example, for an object detection task, the combination block can use linear interpolation to generate a single detection result based on each output detection result 1435 and 1440. In another example, if the task is disease stage classification, the outputs of the two models can be combined by a weighted average. If model A predicts disease stage 3 out of 5, and model B predicts disease stage 4 out of 5, then the combinatorial block can generate a disease stage classification of 3.5 out of 5, which is an equally weighted average. However, in some cases, assigning a higher weight to one model may be preferable. For example, model B could have a prediction weight of 0.8, and model A could have a prediction weight of 0.2. Using the previous disease stage classification, combinatorial block 1434 will generate a disease stage classification. .

[0142] The output of the combination block 1434 can be sent to the display 1450 for viewing by the user.

[0143] Figure 15 This is a schematic flowchart of a method 1500 for model selection and prediction based on aspects of this disclosure. It should be understood that, in other aspects, the steps of method 1500 may be consistent with... Figure 15The different sequences of execution shown may provide additional steps before, during, and after the steps, and / or may replace or eliminate some of the described steps. One or more steps of method 1500 may be performed by one or more devices and / or systems described herein, such as components and / or processor circuitry 250 of network system 100.

[0144] In step 1502, method 1500 includes receiving an incoming request for a first patient case (e.g., 504, including clinical questions and first patient case data). For example, as... Figure 5 As shown, an incoming request 504 is sent from the user computer 160 or the medical imaging console 170 to the model selector 120.

[0145] In step 1504, method 1500 includes accessing a baseline truth dataset (second patient cases, each case having second patient case data and baseline truth labels associated with a clinical problem). For example, model selector 120 accesses baseline truth dataset 122, such as... Figure 5 As shown in the image.

[0146] In step 1506, method 1500 includes selecting a subset of similar cases based on the first patient case data and the second patient case data. For example, the selection is based on a similarity metric 512 and a weight 514, such as... Figure 5 As depicted in the text.

[0147] In step 1508, method 1500 includes providing a subset to both the first and second prediction models. For example, sending a subset 520 of similar cases to... Figure 5 Models 124 and 128 in the example. Providing a subset to the model is not the same as training the model. In some instances, the model has already been trained.

[0148] In step 1510, method 1500 includes generating a first model prediction result from a subset of the first prediction model, and generating a second model prediction result from a subset of the second prediction model. For example, the first model prediction result and the second model prediction result include model output result 525, such as... Figure 5 As depicted in the text.

[0149] In step 1512, method 1500 includes determining the selected model(s) based on the baseline truth labels of a subset, the first model prediction result, and the second model prediction result. For example, based on... Figure 5 The model metric 530 shown is used to select the model.

[0150] In step 1514, method 1500 includes providing a clinical question to the selected model. For example, model selector 120 provides a clinical question to model A 124, such as... Figure 6BAs depicted in the text.

[0151] In step 1516, method 1500 includes generating a selected model prediction result for the first patient case from the selected model. For example, model A 124 generates a selected model output result 540, such as... Figure 5 As depicted in the text.

[0152] In step 1518, method 1500 includes outputting the selected model prediction result to a display. For example, the selected model output 540 is sent to a user computer 160 or a medical imaging console 170, such as... Figure 5 As depicted in the text.

[0153] In step 1520, method 1500 includes receiving user feedback associated with the selected model prediction result. For example, user feedback may be sent from user computer 160 or medical imaging console 170 to model selector 120, such as... Figure 5 As depicted in the text.

[0154] In step 1522, method 1500 includes updating the determination of the selected model based on user feedback. For example, by updating model metrics 530, such as... Figure 5 As described in the document. In some instances, updating the model selector's determination of the selected model will affect subsequent model selections, but will not affect the selection of the current incoming request 504.

[0155] Figure 16 This is a schematic diagram of patient case filtering and model pre-selection in a model selection and prediction system 1600 according to this disclosure. Given user input, the model selection and prediction system 1600 selects a model based on various conditions and metrics and then uses the selected model to make predictions. The model selection and prediction system 1600 can use information from the user input to pre-filter patient cases in a dataset and remove models from consideration that are not designed to perform the necessary task or are not expected to perform poorly on that task. User input may take the form of medical images and / or clinical tasks, and the output may be a response to the clinical task. The user may be any of several medical professionals. Alternatively, user input may not require user generation and submission; program instructions may automatically generate user input based on received images. The model selection and prediction system 1600 may include a model selector 1605, a benchmark ground truth dataset 122, model A 124, model B 128, and model C 150.

[0156] The model selector 1605 can receive an incoming request 504 from the user computer 160 or the medical imaging console 170. The incoming request 504 may include a clinical question 506 and first patient case data 508. In some instances, the clinical question may include medical images and / or clinical tasks, such as... Figure 7 Further description: Incoming request 504 can be generated based on input from the user to the user computer 160 or the medical imaging console 170.

[0157] Model selector 1605 may include a similar case selection module 1610 and a model metric module 1630. The similar case selection module 1610 identifies a set of cases similar to the received information contained in the incoming request 504 based on one or more metrics. The model metric module 1630 identifies one or more models for selection based on one or more metrics, user selection, model metadata, and / or other information.

[0158] The similar case selection module 1610 may include a similarity metric 1612 and weights 1614. The similarity metric 1612 measures how closely one patient case is similar to another. The similarity metric 1612 may depend on the patient's age, weight, height, and other characteristics or biomarkers. The weights 1614 may be parameters used in the similarity metric to generate numerical values ​​for the similarity metric 1612 based on two or more cases. For example, the similarity metric may be derived from a function... Defined, expressed by Formula 1.

[0159] The model metrics module 1630 may include an outcome distance metric 1632, a reliability metric 1634, a robustness metric 1636, and weights 1638. The outcome distance metric 1632 determines the distance between the baseline ground truth labels of cases in a subset of similar cases and the labels contained in the model outputs 1660 generated by models 124 and 128. The reliability metric 534 measures the reliability (i.e., 190) of each model's performance in generating outputs matching the baseline ground truth labels for a subset 520 of similar cases. The robustness metric 536 measures the robustness of the model's performance on different subsets of the baseline ground truth data. In other words, the robustness metric indicates the model's sensitivity to inputs that are close in the sense of similarity metrics described herein. These weights are used as parameters for the outcome distance metric 532, the reliability metric 534, and the robustness metric 536 to generate numerical values ​​for the metrics based on the model outputs and the baseline ground truth labels.

[0160] The similar case selection module 1610 can receive an incoming request 504 and pre-filter cases from the benchmark ground truth dataset 122 based on the incoming request. In some instances, the similar case selection module 1610 can filter or pre-filter cases in the benchmark ground truth dataset using task type, task query, image metadata, etc. Therefore, only the filtered portion 1640 of cases from the benchmark ground truth dataset 122 can be selected. For example, the incoming request may seek target detection of the uterus, so the similar case selection module 1610 can pre-filter out all cases in the patient case data of the benchmark ground truth dataset where the biometric sex identifier is male. Pre-filtering can be performed by other devices and systems described herein.

[0161] The similar case selection module 1610 can generate a subset 1645 of cases from the filtered portion 1640 that are similar to the patient cases described in the incoming request 504. For example, the similar case selection module 510 can select cases from all filtered case portions 1640 whose similarity metric values ​​are less than a maximum value. The maximum value can be pre-programmed into the module or determined directly or indirectly based on the incoming request 504.

[0162] In some instances, model selector 1605 can pre-select models that will receive a subset 1645 of similar cases. In some instances, model selector 1605 can pre-select models to be tested using task type, task query, image metadata, etc. For example, some models may not be designed for object detection, so generating output from such models will produce output irrelevant to the task. For illustrative purposes, Figure 16 A configuration is described in which model A 124 and model B 128 have been pre-selected by model selector 1605.

[0163] For example, in model pre-selection, there might be 15 models to choose from: 5 models trained using ultrasound images, 5 models trained using MRI images, and 5 models trained using CT images. If the incoming request includes MRI images, the 5 models trained using ultrasound images and the 5 models trained using CT images can be excluded. Therefore, the 5 models trained using MRI images can be pre-selected. Pre-selection allows models to be excluded from consideration without requiring significant computational resources.

[0164] Both Model A 124 and Model B 128 can receive cases from a subset 1645 of similar cases and generate their respective model outputs 1660. For example, model output 1660 can appear in the form of image segmentation, i.e., since the task is segmentation. Model output 1660 can take any form, depending on the clinical task the model is required to solve, such as diagnosis, segmentation, classification, etc. The model selection and prediction system 1600 can contain different numbers of models. For example, the selection can be between two models, between four and twenty models, and / or other larger or smaller values.

[0165] The model measurement module 1630 can receive model output 1660 and generate selected model output 1662, reliability value 1665, and / or robustness value 1670. The model measurement module 1630 can select a model, such as one or more of model A 124 and model B 128. The model selection and prediction system 1600 can use the selected model to generate the selected model output 1662 based on the incoming request. The model measurement module 1630 can generate a reliability value 1665 using a reliability metric 1634 and weights 1638. The model measurement module 1605 can generate a robustness value 1670 using a robustness metric 1636 and weights 1638.

[0166] User computer 160 or medical imaging console 170 can receive the selected model output 1662, reliability value 1665, and / or robustness value 1670. User computer 160 or medical imaging console 170 can display and / or post-process information from model selection and prediction system 1600. When a user (e.g., a physician) interacts with the display, they can evaluate the results generated by the model selection and prediction system according to incoming request 504. The user can provide user feedback 1680 to model selection and prediction system 1600, and in some cases, also to model selector 1605, which evaluates the system's performance. User feedback 1680 may also include parameters or other options to directly or indirectly change the weights 1614, 1638, or metrics 1612, 1632, 1634, or 1636 in model selector 1605. The user can iteratively use model selection and prediction system 1600 to obtain desired clinical outcomes. Furthermore, the model can be fine-tuned based on user feedback on performance.

[0167] Figure 17 This is a schematic flowchart of a model selection and prediction method 1700 with patient case filtering and model pre-selection according to this disclosure. It should be understood that, in other respects, the steps of method 1700 can be... Figure 17The different sequences of execution shown may provide additional steps before, during, and after the steps, and / or may replace or eliminate some of the described steps. One or more steps of method 1700 may be performed by one or more devices and / or systems described herein, such as components and / or processor circuitry 250 of network system 100.

[0168] In step 1702, method 1700 includes receiving an incoming request for a first patient case (clinical question and first patient case data). For example, as Figure 16 As shown, an incoming request 504 is sent from the user computer 160 or the medical imaging console 170 to the model selector 1605.

[0169] In step 1704, method 1700 includes accessing a baseline truth dataset (second patient cases, each case having second patient case data and baseline truth labels associated with a clinical problem). For example, model selector 1605 accesses baseline truth dataset 122, such as... Figure 16 As shown in the image.

[0170] In step 1706, method 1700 includes filtering the second patient case based on a comparison between the clinical question and a baseline truth label for the second patient case. In some instances, filtering may also be based on image metadata. For example, Figure 16 The filtered section of the case described in the text is 1640.

[0171] In step 1708, method 1700 includes selecting a subset of similar cases from a filtered portion of the second patient case data, based on the first patient case data and the second patient case data. For example, the selection is based on a similarity metric 1612 and a weight 1614, such as... Figure 16 As depicted in the text.

[0172] In step 1710, method 1700 includes pre-selecting a first prediction model and a second prediction model from a plurality of prediction models. For example, pre-selecting model 1650, such as Figure 16 As depicted in the text.

[0173] In step 1712, method 1700 includes providing a subset to the first prediction model and the second prediction model. For example, a subset 1645 of similar cases is sent to models 124 and 128, such as... Figure 16 As illustrated in the diagram, providing a subset to the model is not the same as training the model. In some instances, the model has already been trained.

[0174] In step 1714, method 1700 includes generating a first model prediction result from a subset of the first prediction model, and generating a second model prediction result from a subset of the second prediction model. For example, the first and second model prediction results of model output result 1660 are as follows: Figure 16 As depicted in the text.

[0175] In step 1716, method 1700 includes determining the selected model(s) based on a subset of baseline truth labels, a first model prediction result, and a second model prediction result. For example, based on... Figure 16 The model metric 1630 shown is used to select the model.

[0176] In step 1718, method 1700 includes providing a clinical question to the selected model. For example, model selector 120 provides a clinical question to model A 124, such as... Figure 6B As depicted in the text.

[0177] In step 1720, method 1700 includes generating a selected model prediction result for the first patient case from the selected model. For example, model A 124 generates a selected model output result 1662, such as... Figure 16 As depicted in the text.

[0178] In step 1722, method 1700 includes outputting the selected model prediction result to a display. For example, the selected model output 1662 is sent to a user computer 160 or a medical imaging console 170, such as... Figure 16 As depicted in the text.

[0179] Figure 18 This is a schematic diagram of a model selection and prediction system 1800 with request comparison according to aspects of this disclosure. The model selection and prediction system 1800 compares a user's new request with a previous request to determine if the new request is similar to the old request. If it is determined that the new request is sufficiently similar to the old request, the same model as the old request can be selected for the new request. Therefore, significant savings in computation time and resources can be achieved. The model selection and prediction system may include a user computer 160 or a medical imaging console 170, a model selector 1805, a previously received request dataset 1820, a baseline ground truth dataset 122, and models A 124, B 128, and C 150.

[0180] The model selector 1805 can receive an incoming request 504 from the user computer 160 or the medical imaging console 170. The incoming request 504 may include a clinical question 506 and first patient case data 508. In some instances, the clinical question may include medical images and / or clinical tasks, such as... Figure 7Further description: Incoming request 504 can be generated based on input from the user to the user computer 160 or the medical imaging console 170.

[0181] The model selector 1805 may include a previous request comparison module 1810, a similar case selection module 510, and a model measurement module 530. The previous request comparison determines whether the incoming request 504 is sufficiently similar to previous requests in the previous request dataset 1820, so that the previously selected model can be used to process the incoming request 504. The previous request dataset 1820 includes the previous clinical question 1822, the previous first patient case data 1824, and the previously selected model 1826.

[0182] The previous request comparison module 1810 can use any information provided in the incoming request 504 to determine whether the selected model can be chosen based on a previous request. In some instances, the comparison can be based on age, weight, and other biomarkers, as well as task type, task query, image metadata, etc. If the previous request comparison module determines that the incoming request is sufficiently close to a previous request, it selects the previously selected model as the model to evaluate the incoming request. In this case, it may not be necessary to use the similarity case selection module 510 or the model measurement module 530. For example, Figure 18 The diagram describes the selection of model A to generate the selected model output 1830 according to the incoming request 504. The selected model output 1830 can be provided to the user computer 160 or the medical imaging console 170 and displayed to the user.

[0183] As an example of comparing an incoming request with a previous request, the incoming request might ask for disease stage classification of cirrhosis in a 60-year-old patient. The selection and prediction system 1800 can search previous requests for patients of similar age whose clinical task is disease stage classification of cirrhosis. For example, the system might ignore previous requests for image segmentation of the patient's lungs. If a sufficiently close match is found, the model used for the identified previous request is applied to the current incoming request.

[0184] Figure 19 This is a schematic flowchart of a method 1900 for model selection and prediction with incoming request comparisons, according to various aspects of this disclosure. It should be understood that, in other aspects, the steps of method 1900 can be compared with... Figure 19 The different sequences of execution shown may provide additional steps before, during, and after the steps, and / or may replace or eliminate some of the described steps. One or more steps of method 1900 may be performed by one or more devices and / or systems described herein, such as components and / or processor circuitry 250 of network system 100.

[0185] In step 1902, method 1900 includes receiving an incoming request for a first patient case (clinical question and first patient case data). For example, as Figure 18 As shown, an incoming request 504 is sent from the user computer 160 or the medical imaging console 170 to the model selector 1805.

[0186] In step 1904, method 1900 includes performing a comparison between the incoming request and a previously received incoming request stored in memory. For example, as Figure 18 As depicted, the incoming request 504 is compared with the previous request in the previous request dataset 1820 by the previous request comparison 1810. For example, the comparison may include calculating a similarity metric (such as similarity metric 512) using the first patient case data 508 and the previous first patient case data 1824.

[0187] In step 1906, method 1900 includes a decision step to determine whether an incoming request is sufficiently similar to a previously received incoming request. For example, if the value of a similarity metric meets a threshold (e.g., greater than or less than one of the thresholds), the incoming request is sufficiently similar to at least one incoming request. If the value of a similarity metric does not meet a threshold (e.g., greater than or less than the other of the thresholds), the incoming request is not sufficiently similar to at least one incoming request. If sufficiently similar, the method proceeds to step 1907; if not sufficiently similar, the method proceeds to step 1908. In some instances, whether two incoming requests are sufficiently close can be determined via a threshold score (e.g., a percentage representing the equivalence of two requests). Alternatively, model selection and prediction systems may require task type matching and case parameters to be within predefined ranges relative to each other. For example, an incoming request for an 80-year-old patient may be similar to an incoming request for a 75-year-old patient, but not similar to an incoming request for a 55-year-old patient.

[0188] In step 1907, method 1900 includes identifying the previously determined selected model from a previously incoming request as the selected model. For example, the previously selected model in previously selected model 1826 is associated with a previous request, such as... Figure 18 As described in the text. For example... Figure 18 As depicted. Model A 124 is the previously selected model.

[0189] In step 1908, method 1900 includes accessing a baseline truth dataset (second patient cases, each case having second patient case data and baseline truth labels associated with a clinical problem). For example, model selector 120 accesses baseline truth dataset 122, such as... Figure 5 As shown in the image.

[0190] In step 1910, method 1900 includes selecting a subset of similar cases based on the first patient case data and the second patient case data. For example, the selection is based on a similarity metric 512 and a weight 514, such as... Figure 5 As depicted in the text.

[0191] In step 1912, method 1900 includes providing a subset to the first and second prediction models. For example, a subset 520 of similar cases is sent to models 124, 128, and 150, such as... Figure 5 As illustrated in the diagram, providing a subset to the model is not the same as training the model. In some instances, the model has already been trained.

[0192] In step 1914, method 1900 includes generating a first model prediction result from a subset of the first prediction model, and generating a second model prediction result from a subset of the second prediction model. For example, the first model prediction result and the second model prediction result include model output 525, such as... Figure 5 As depicted in the text.

[0193] In step 1916, method 1900 includes determining the selected model(s) based on a subset of baseline truth labels, a first model prediction result, and a second model prediction result. For example, based on... Figure 5 The model metric 530 shown is used to select the model.

[0194] In step 1918, method 1900 includes providing a clinical question to the selected model. For example, model selector 120 provides a clinical question to model A 124, such as... Figure 6B As depicted in the text.

[0195] In step 1920, method 1900 includes generating a selected model prediction result for the first patient case from the selected model. For example, model A 124 generates a selected model output result 540, such as... Figure 5 As depicted in the text.

[0196] In step 1922, method 1900 includes outputting the selected model prediction result to a display. For example, the selected model output 540 is sent to a user computer 160 or a medical imaging console 170, such as... Figure 5 As depicted in the text.

[0197] Figure 20This refers to aspects of the disclosure, including sample user interfaces for displaying the selected model and model results. Users can both input information (e.g., a first patient case) and receive information (e.g., model selection and prediction results) via the user interface on the display. Sample user interfaces may include: patient data 2005, clinical question interface 2010, case parameter interface 2015, feedback interface 2020, and results interface 2025.

[0198] Patient data 2005 may include patient identification information, such as name, date of birth, record identifier, patient identifier, and the medical facility the patient is affiliated with. Patient data 2005 may include hyperlinks to databases containing the patient's medical records.

[0199] The Clinical Problem Interface 2010 provides the ability to select or upload image files and input clinical tasks. Clinical task input can include task type, such as target detection, and inquiry, such as what target needs to be detected. Task type and / or inquiry...

[0200] The Case Parameters Interface 2015 describes various values ​​and ranges for case parameters used in model selection and prediction. For example, patient / subject ages can be labeled on a line, along with the maximum and minimum ages of patients used in a subset of similar cases described herein. In some instances, users can configure the labeling to select wider, narrower, or offset age ranges for use in the model selection and prediction system. By making new selections to the age ranges, the model selection and prediction system can be rerun to update the results based on user input. Similar functionality can be used for other parameters, such as weight, blood pressure, and other biomarkers.

[0201] The Feedback Interface 2020 allows users to evaluate the results generated by the model selection and prediction system. Feedback can be a simple yes / no question: "Did the selected AI model successfully answer the clinical question?" In some instances, feedback can solicit ratings, such as numerical inputs between 1 and 10. The Feedback Interface 2020 may include options for rerunning the model selection and prediction system to account for any changes to the inputs in Interfaces 2010, 2015, and 2020.

[0202] like Figure 20As described, Results Interface 2025 displays the detection results for kidneys in ultrasound images. Results Interface 2025 includes the ultrasound image and target detection boxes generated as the output of the model selection and prediction system. Results Interface 2025 may include various metrics, including the reliability and robustness metrics described herein, and model confidence of the results. Confidence from the model is a different metric from reliability and robustness metrics. In some instances, the name and / or source of the model used to generate the results may be listed. Furthermore, the results interface may include a list of models that also performed highly during the model selection process.

[0203] It should be understood that icons can be rearranged as needed, and different levels of technical information can be displayed on the user interface. For example, more advanced users may be shown very detailed information about results and case parameters, as well as a greater ability to provide feedback. Certain users may need to be granted permissions to provide feedback, thus ensuring that only users with the required level of expertise can contribute to any model training or updates based on feedback.

[0204] Furthermore, the techniques disclosed herein are also applicable to other medical imaging modalities obtained from medical imaging devices that provide 3D data, such as other ultrasound applications, camera-based video, X-ray video, and 3D volumetric images, such as computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, optical coherence tomography (OCT) scans, or intravenous ultrasound (IVUS) pull-back sequences. The techniques described herein can be used in a variety of settings, including emergency departments, intensive care units, inpatient wards, and outpatient settings.

[0205] Therefore, the logical operations constituting the various aspects of the technology described herein can be referred to as operations, steps, objects, layers, elements, components, algorithms, or modules, respectively. Furthermore, it should be understood that, unless otherwise expressly claimed or the language of the claims inherently requires a particular order, these can occur, be performed, or arranged in any order.

[0206] All directional references, such as up, down, inside, outside, upward, downward, left, right, side, front, back, top, bottom, above, below, vertical, horizontal, clockwise, counterclockwise, near, and far, are for identification purposes only to help the reader understand the claimed object and do not impose limitations, particularly regarding the location, orientation, or purpose of the said automatic measurement point detection system. Connection references, such as attached, coupled, connected, combined, or “communicating with,” should be interpreted broadly and may include intermediate elements between sets of elements and relative movement between elements, unless otherwise indicated. Therefore, a connection reference does not necessarily mean that two elements are directly connected and have a fixed relationship with each other. The term “or” should be interpreted as “and / or,” not “exclusive or.” The word “comprising” does not exclude other elements or steps, and the words “a” or “an” do not exclude multiple. Unless otherwise stated in the claims, the numerical values ​​given should be interpreted as exemplary and should not be considered restrictive.

[0207] The foregoing descriptions, examples, and data provide a complete description of the structure and use of exemplary aspects of the model selection and prediction system described herein. While various aspects of the claimed object have been described in a particular degree of detail above, or with reference to one or more aspects, those skilled in the art can make numerous modifications to the disclosed aspects without departing from the spirit or scope of the claimed object.

[0208] Other aspects have also been conceived. All content contained in the above description and drawings should be construed as illustrative of specific aspects only and is not intended to be limiting. Changes in detail or structure may be made without departing from the essential elements of the subject matter as defined in the following claims.

[0209] Specifically, an advantageous embodiment is a computer-implemented method comprising the steps of: receiving a clinical question and first patient case data for a first patient case. It includes the step of performing a comparison of the clinical question with a plurality of previous clinical questions. It includes the step of: using a similarity metric, calculating a similarity value for each of the plurality of previous clinical questions based on the comparison of the current clinical question and the plurality of previous clinical questions. It includes the step of determining a selected predictive model. It includes the step of providing the clinical question to the selected model. It includes the step of: generating a selected model prediction result for the first patient case from the selected model. It includes the step of outputting the selected model prediction result to a display, wherein determining the selected model includes:

[0210] Access multiple second patient cases (each of which contains second patient case data and baseline truth labels).

[0211] Based on the first patient case data and the second patient case data among the plurality of second patient cases, a subset of the plurality of second patient cases is selected;

[0212] The subset is provided to the first prediction model and the second prediction model;

[0213] Advantageous embodiments further include the steps of: generating multiple first model prediction results from the first prediction model for the subset; and generating multiple second model prediction results from the second prediction model for the subset. It includes the steps of: selecting at least one of the first prediction model or the second prediction model as the selected prediction model based on the baseline truth label of the subset, the multiple first model prediction results, and the multiple second model prediction results, wherein determining the selected model includes:

[0214] The previously identified predictive model for a specific clinical problem was selected as the chosen model.

Claims

1. A computer-implemented method, comprising: Clinical questions regarding the reception of first patient cases (506) and data from first patient cases (508); Access multiple second patient cases (122), wherein each second patient case (850) includes second patient case data (860) and baseline truth labels (865). Based on the first patient case data (508) and the second patient case data (860) of the plurality of second patient cases, a subset (520) of the plurality of second patient cases is selected. The subset (520) is provided to the first prediction model (124) and the second prediction model (128). The first prediction model (124) generates multiple first model prediction results (618) for the subset, and the second prediction model (128) generates multiple second model prediction results (620) for the subset. The selected model is determined based on the baseline truth labels of the subset, the multiple first model prediction results (618) and the multiple second model prediction results (620), wherein the selected model includes at least one of the following: the first prediction model (124) or the second prediction model (128). Provide the clinical question to the selected model (506); The selected model generates a prediction result for the first patient case; and The selected model prediction result (540) is output to the display (166).

2. The method according to claim 1, in, The clinical problem (506) includes a first medical image (710) and a clinical task (720) associated with the first medical image, and The selected model prediction results include a graphical representation (460) superimposed on the first medical image (450) and associated with the clinical task.

3. The method according to claim 2, wherein, The clinical task (720) includes at least one of the following: anatomical structure examination, anatomical structure segmentation, and disease stage classification.

4. The method according to any one of the preceding claims, wherein, Both the first prediction model (124) and the second prediction model (128) include a pre-trained convolutional neural network (CNN) (312).

5. The method according to any one of the preceding claims, in, Each second patient case (850) also includes a second medical image (855). The baseline truth label (865) is associated with the second medical image.

6. The method according to claim 5, wherein, Providing the subset to the first prediction model and the second prediction model includes: The second medical image (855) in the subset (122) is provided to the first prediction model (124) and the second prediction model (128).

7. The method according to any one of the preceding claims, wherein, Selecting the subset includes: Using a similarity metric (512), a similarity value is calculated for each second patient case based on a comparison between the first patient case data (508) and the second patient case data (860) of the plurality of second patient cases; and The subset (122) is identified based on the similarity value.

8. The method according to any one of the preceding claims, wherein, Determining the selected model includes: Using a first plurality of outcome distance metric (512), a first plurality of outcome distance value (1032) is calculated based on a comparison between the plurality of first model predictions (1022) and the baseline ground truth labels (1010) of the subset; and Using a second plurality of result distance metric, a second plurality of result distance value is calculated based on a comparison between the plurality of second model prediction results and the baseline ground truth labels of a subset.

9. The method according to claim 8, wherein, Determining the selected model includes: A first reliability value (1405) is calculated based on the first plurality of result distance values ​​using a first reliability metric for the first prediction model (124). Using a second reliability metric for the second prediction model (128), a second reliability value (1415) is calculated based on the second plurality of result distance values; and The selected model is identified based on the first reliability value and the second reliability value.

10. The method according to any one of the preceding claims, further comprising: Using a first robustness measure for the first prediction model, a first robustness value (1410) is calculated based on the first plurality of result distance values. The second robustness value (1420) is calculated based on the second plurality of result distance values ​​using a second robustness metric for the second prediction model. The selection of the model is further based on the first robustness value and the second robustness value.

11. The method according to claim 10, in, The calculation of the first robustness value (1410) for the first prediction model (124) is based on another subset (1210) of the plurality of second patient cases, and The calculation of the second robustness value (1420) for the second prediction model (128) is based on another subset (1215) of the plurality of second patient cases.

12. The method according to any one of claims 9-11, further comprising outputting at least one of the first reliability value (1405) or the second reliability value (1415) to the display (166).

13. The method according to any one of the preceding claims, wherein, The selected models include the first prediction model (124) and the second prediction model (128).

14. The method according to any one of the preceding claims, It also includes generating a combined output result (1434) based on the selected model prediction result (1435) made by the first prediction model (124) for the first patient case and the selected model prediction result (1440) made by the second model (128) for the first patient case. in, The selected model prediction results include the combined output results.

15. The method according to any one of the preceding claims, It also includes pre-selecting the first prediction model and the second prediction model from multiple prediction models (1650). in, The plurality of prediction models include the first prediction model (124), the second prediction model (128), and the third prediction model (150). Each of the plurality of prediction models includes model metadata (126) representing model training. Among them, the pre-selection includes: A comparison is performed between the model metadata of the plurality of predictive models and at least one of the following: the clinical question (506) or the first patient case data (508); and At least one of the following: Based on the comparison, the first prediction model (124) and the second prediction model (128) are identified; or Based on the comparison, the third prediction model (150) is excluded.

16. The method according to any one of the preceding claims, This also includes filtering the multiple second patient cases (122), in, The filtering includes: Perform a comparison between the following items: At least one of the first patient case data (508) or the clinical problem (506); and At least one of the second patient case data (860) or the baseline truth label (865) of the plurality of second patient cases; Based on the comparison, filtered portions (1640) of the plurality of second patient cases (122) are identified. The subset (1645) is selected from the filtered portion (1640) of the plurality of second patient cases (122).

17. The method according to any one of the preceding claims, Receive user feedback (560) indicating whether the selected model prediction results are responsive to the clinical question (506); and Based on the user feedback, the method for determining the selected model will be updated.

18. A system (100) comprising: The processor circuit (112) is configured as follows: Clinical questions regarding the reception of first patient cases (506) and data from first patient cases (508); Access multiple second patient cases (122), wherein each second patient case (850) includes second patient case data (860) and baseline truth labels (865). Based on the first patient case data (508) and the second patient case data (860) of the plurality of second patient cases, a subset (520) of the plurality of second patient cases is selected. The subset (520) is provided to the first prediction model (124) and the second prediction model (128). The first prediction model (124) generates multiple first model prediction results (618) for the subset, and the second prediction model (128) generates multiple second model prediction results (620) for the subset. The selected model is determined based on the baseline truth labels of the subset, the multiple first model prediction results (618) and the multiple second model prediction results (620), wherein the selected model includes at least one of the following: the first prediction model (124) or the second prediction model (128). Provide the clinical question to the selected model (506); The selected model generates a prediction result for the first patient case; and The selected model prediction result (540) is output to the display (166).

19. A non-transient machine-readable medium comprising a plurality of machine-executable instructions, said plurality of machine-executable instructions, when executed by one or more processors, being adapted to cause said one or more processors to perform operations including: Clinical questions regarding the reception of first patient cases (506) and data from first patient cases (508); Multiple second patient cases were visited (122), among which, Each second patient case (850) includes second patient case data (860) and baseline truth labels (865); Based on the first patient case data (508) and the second patient case data (860) of the plurality of second patient cases, a subset (520) of the plurality of second patient cases is selected. The subset (520) is provided to the first prediction model (124) and the second prediction model (128). The first prediction model generates multiple first model prediction results for the subset, and the second prediction model generates multiple second model prediction results for the subset; The selected model is determined based on the baseline truth labels of the subset, the multiple first model prediction results (618) and the multiple second model prediction results (620), wherein the selected model includes at least one of the following: the first prediction model (124) or the second prediction model (128). Provide the clinical question to the selected model (506); The selected model generates the selected model prediction result for the first patient case; as well as The selected model prediction result (540) is output to the display (166).

Images