PCCT image viewer based on deep learning

Abstract

The invention relates to a PCCT image viewer based on deep learning. Systems and methods for generating a guided review of one or more input medical images are provided. One or more input medical images of a patient and text-based patient data of the patient are received. One or more clinical tasks are identified based on the text-based patient data using a language model. One or more machine learning based models are selected based on the identified one or more clinical tasks. One or more medical imaging analysis tasks are performed based on the one or more input medical images using the selected one or more machine learning-based models. A guided review of the one or more input medical images is generated based on results of the one or more medical imaging analysis tasks. A guided review of the one or more input medical images is output.

Description

Technical Field

[0001] The present invention generally relates to an AI / ML (artificial intelligence / machine learning)-based medical image viewer, and more particularly to a DL (deep learning)-based PCCT (photon counting computed tomography) image viewer. Background Art

[0002] PCCT is an imaging technique that uses X-ray detectors to count the number of incident photons and measure their energy. Compared to standard CT (computed tomography), PCCT provides high-resolution images with improved image quality, a greater signal-to-noise ratio, and reduced radiation dose. However, in current clinical workflows, such high-resolution PCCT images generated by PCCT are typically viewed using conventional image viewers. Such conventional image viewers are unable to take advantage of the advantages provided by high-resolution PCCT images. Summary of the Invention

[0003] According to one or more embodiments, a system and method for generating a guided review of one or more input medical images is provided. One or more input medical images of a patient and text-based patient data of the patient are received. One or more clinical tasks are identified based on the text-based patient data using a language model. One or more machine learning-based models are selected based on the identified one or more clinical tasks. One or more medical imaging analysis tasks are performed based on the one or more input medical images using the selected one or more machine learning-based models. A guided review of the one or more input medical images is generated based on the results of the one or more medical imaging analysis tasks. The guided review of the one or more input medical images is output.

[0004] In one embodiment, a guided review is generated that includes a full image review of one or more input medical images, a zonal review of one or more anatomical cropped regions delineating the one or more input medical images, and a findings review of one or more pathological cropped regions delineating the one or more input medical images.

[0005] In one embodiment, a guided review of one or more input medical images is presented to a user via a display device. A cropped region depicting a first finding is presented, user input is received from the user confirming a decision regarding the first finding, and in response to receiving the user input, a cropped region depicting a second finding is automatically depicted. The guided review of the one or more input medical images may be presented according to parameters determined based on one or more identified clinical tasks and outputs of one or more selected machine learning-based models.

[0006] In one embodiment, the language model receives text-based data as input and generates one or more vectors corresponding to one or more clinical tasks as output. The language model may be an LLM (Large Language Model).

[0007] In one embodiment, one or more machine learning-based models are selected from a database of pre-trained machine learning-based models, including machine learning-based models for performing different medical imaging analysis tasks and machine learning-based models with different sensitivities.

[0008] In one embodiment, the one or more input medical images include one or more PCCT (Photon Counting Computed Tomography) images.

[0009] These and other advantages of the present invention will become apparent to those of ordinary skill in the art upon reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A workflow for generating a guided review of one or more PCCT images according to one or more embodiments is shown;

[0011] Figure 2 A method for generating a guided review of one or more input medical images according to one or more embodiments is shown;

[0012] Figure 3 An exemplary artificial neural network that can be used to implement one or more embodiments is shown;

[0013] Figure 4 shows a convolutional neural network that can be used to implement one or more embodiments;

[0014] Figure 5 shows a schematic structure of a recursive machine learning model that can be used to implement one or more embodiments; and

[0015] Figure 6 A high-level block diagram of a computer is shown that can be used to implement one or more embodiments. DETAILED DESCRIPTION

[0016] The present invention generally relates to methods and systems for deep learning-based PCCT image viewers. Embodiments of the present invention are described herein to provide a visual understanding of such methods and systems. A digital image typically consists of digital representations of one or more objects (or shapes). In this document, digital representations of objects are typically described in terms of identifying and manipulating each object. Such manipulations are virtual manipulations performed in the memory or other circuitry / hardware of a computer system. Therefore, it is understood that embodiments of the present invention can be performed within a computer system using data stored within the computer system. Further, references herein to image pixels may equally refer to image voxels, and vice versa.

[0017] Embodiments described herein provide a deep learning-based PCCT image viewer. The PCCT image viewer provides radiologists (or any other user) with automated or semi-automated guided review of PCCT images using various anatomical objects and pathological findings. At each step of the guided review, the PCCT image viewer displays appropriate images with or derived from the appropriately configured PCCT image to aid the radiologist in making a diagnosis and decision. Advantageously, the PCCT image viewer accelerates and automates the clinical workflow for PCCT image reading and interpretation by leveraging features provided by PCCT.

[0018] Figure 1 A workflow 100 is shown for generating a guided review of one or more PCCT images, according to one or more embodiments. Figure 2 A method 200 for generating a guided review of one or more input medical images is shown in accordance with one or more embodiments. Figure 2 The steps and sub-steps of method 200 may be performed using one or more suitable computing devices (such as, for example, Figure 6 computer 602) to execute. Figure 1 and Figure 2 They will be described together.

[0019] exist Figure 2 At step 202, one or more input medical images of a patient and text-based patient data of the patient are received. In one example, Figure 1 As shown in the workflow 100 , the one or more input medical images are PCCT images 104 , and the text-based patient data is text-based patient data 102 .

[0020] The one or more input medical images may depict one or more anatomical objects, such as, for example, organs, bones, blood vessels, tumors or other abnormalities, or any other anatomical objects of interest to the patient. In one embodiment, the one or more input medical images comprise a PCCT image of the patient. However, the one or more input medical images may be of any other suitable modality, such as, for example, CT (computed tomography), MRI (magnetic resonance imaging), US (ultrasound), x-ray, or any other medical imaging modality or combination of medical imaging modalities. The one or more input medical images may comprise 2D (two-dimensional) images and / or 3D (three-dimensional) volumes.

[0021] The text-based patient data may include any suitable text-based data about the patient. In one embodiment, the text-based patient data may include a text-based clinical or radiology report of the patient, current symptoms experienced by the patient, and current prescribed medications the patient is taking. However, the text-based patient data may include any other suitable text-based data about the patient, such as, for example, the patient's demographic information, vital signs, medical history, family history, laboratory results, measurements, and information extracted from medical images.

[0022] One or more input medical images and text-based patient data may be received, for example, by transmitting the medical image data from an image acquisition device (e.g., Figure 6 The image acquisition device 614) directly receives one or more input medical images by reading from the storage device or memory of the computer system (e.g., Figure 6 602 memory 610 or storage device 612) of the computer 602, or by loading one or more input medical images and / or text-based patient data from a remote computer system (e.g., Figure 6 The computer system 602 receives one or more input medical images and / or text-based patient data. Such a computer system or remote computer system may include one or more patient databases, such as, for example, an EHR (electronic health record), an EMR (electronic medical record), a PHR (personal health record), a HIS (health information system), a RIS (radiology information system), a PACS (picture archiving and communication system), a LIMS (laboratory information management system), or any other suitable database or system.

[0023] exist Figure 2 At step 204, a language model is used to identify one or more clinical tasks based on the text-based patient data. In one example, Figure 1 As shown in the workflow 100 of FIG. 1 , a clinical task selection 106 is performed based on text-based patient data 102 to identify a clinical task 108. Figure 1As illustratively shown in FIG, clinical tasks 108 include coronary artery plaque detection, lung nodule detection, and cervical spine fracture detection.

[0024] A language model is an AI / ML-based model that is trained to predict sequences in a language domain. In one embodiment, the language model is an LLM (Large Language Model). However, the language model can be any other suitable language model. For example, the language model can be a small language model, which uses a relatively small neural network, has fewer parameters, and is trained based on less training data than an LLM.

[0025] The LLM can be any suitable pre-trained deep learning-based LLM. For example, the LLM can be based on a transformer architecture that uses an attention mechanism to capture long-range dependencies in text. An example of a transformer-based architecture is GPT (Generative Pre-trained Transformer), which has a multi-layer transformer decoder architecture that can be pre-trained to optimize the next token prediction task and then fine-tuned for various downstream tasks using labeled data. GPT-based LLMs can be trained and / or fine-tuned using reinforcement learning with human feedback for performing various natural language processing tasks. Other exemplary transformer-based architectures include BLOOM (BigScience Large Open Science Open Access Multilingual Language Model) and BERT (Bidirectional Encoder Representations from Transformers). The LLM is fine-tuned for use in the field of medical imaging. In some embodiments, the LLM can be a multimodal LLM to receive, for example, imaging data (e.g., one or more input medical images) as well as text-based reports and multiple predefined templates. Pre-training and fine-tuning are performed during one or more previous offline or training phases. Once trained, the LLM is applied in the online or inference phase, e.g. to perform Figure 2 Step 204.

[0026] The language model receives one or more prompts comprising text-based patient data as input and generates one or more identified clinical tasks as output. The prompt refers to the input to the language model for generating a response. The prompt can be received from the computer system, for example, via one or more APIs (application programming interfaces), or from a user interacting with the computer system. The identified one or more clinical tasks can be output as one or more numerical vectors. The numerical vector has a predefined meaning associated with the identified one or more clinical tasks learned during pre-training and / or fine-tuning. However, the one or more clinical tasks can be output in any other suitable format, such as, for example, a natural language output.

[0027] The one or more clinical tasks may be any appropriate clinical tasks for the patient. In one example, for a patient who has severe headaches, has previously suffered a stroke, and is prescribed a PCCT scan of the head, the one or more clinical tasks may be a brain acute findings task. In another example, for a patient who has shortness of breath and chest pain and has previously suffered a head attack, the one or more clinical tasks may be a coronary artery plaque screening task.

[0028] In one embodiment, a language model may be used to identify multiple candidate clinical tasks and presented to a user (eg, a clinician) via a display device. The user may select one or more of the multiple candidate clinical tasks as the one or more clinical tasks.

[0029] exist Figure 2 At step 206, one or more machine learning based models are selected based on the identified one or more clinical tasks. In one example, Figure 1 As shown in the workflow 100 of FIG, the one or more machine learning based models selected are a selected set of DL models 110 selected for a specific clinical task 108. For example, for a neurotriage task in the emergency room, a DL model is initiated to detect acute findings (hemorrhage, infarction, mass effect, etc.) in the brain.

[0030] One or more machine learning-based models are selected from a database of pre-trained machine learning-based models. The one or more machine learning-based models are used to perform any suitable one or more medical imaging analysis tasks. In one example, the one or more machine learning-based models include classification, detection, segmentation, quantification, etc. for a specific pathology or for any other anatomical object of interest. In another example, the one or more machine learning-based models include a high sensitivity model to exclude the presence of any findings in a specific image sub-region. In one embodiment, the one or more machine learning-based models may also output uncertainty information associated with the model output.

[0031] In one example, where one or more clinical tasks include a task for acute cerebral findings, a machine learning-based image classification model can be selected to classify acute hemorrhage, infarction, and mass effect. If any of acute hemorrhage, infarction, or mass effect is classified as positive, a machine learning-based segmentation model can be selected to segment the positive findings. The machine learning-based model generates a classification result and a segmentation map, along with corresponding uncertainties, as outputs. A highly sensitive machine learning-based extracranial finding model can be selected, for example, to exclude the presence of findings below the foramen magnum.

[0032] In another example, when one or more clinical tasks include a coronary plaque detection task, a machine learning-based coronary plaque detection model can be selected to detect coronary plaque in PCCT images. The machine learning-based coronary plaque detection model generates a bounding box around the detected coronary plaque and a corresponding uncertainty as output.

[0033] The one or more machine learning based models are pre-trained during a previous offline or training phase. Once trained, the one or more machine learning based models are stored in a database of machine learning based models (e.g., Figure 6 610 or 612 of the computer 602) and is used during the online or inference phase, for example, to perform Figure 2 Step 208 (described in further detail below).

[0034] exist Figure 2 At step 208, one or more medical imaging analysis tasks are performed based on the one or more input medical images using the selected one or more machine learning based models. The one or more medical imaging analysis tasks may include any suitable medical imaging analysis tasks, such as, for example, detection, classification, segmentation, quantification of anatomical objects (e.g., pathology). In one example, Figure 1 As shown in the workflow 100 of FIG, one or more medical imaging analysis tasks include DL-based image classification, object detection, and segmentation tasks 112 performed based on PCCT images 104 to generate object detection and segmentation results 114 with uncertainty outputs. The selected one or more machine learning-based models receive one or more input medical images as input and generate results of the medical imaging analysis tasks as outputs. The results of the medical imaging analysis tasks may include or be used to generate anatomical findings and pathological findings of the one or more input medical images.

[0035] exist Figure 2 At step 210, a guided review of one or more input medical images is generated based on the results of the one or more medical imaging analysis tasks. Figure 1 As shown in the workflow 100 of FIG. 1 , a guided review 118 of the PCCT image 104 is generated by a photon counting CT viewer 116 based on an object detection and segmentation result 114 with an uncertainty output.

[0036] The guided review includes steps in a preferred order with automatic rendering. In one embodiment, the guided review stage includes a full image review of each of the one or more input medical images, a partitioned review of (one or more) anatomical cropped regions depicting the anatomical findings of the one or more input medical images, and a review of the findings of (one or more) pathology cropped regions depicting the pathological findings of the one or more input medical images. The (one or more) anatomical cropped regions and (one or more) pathology cropped regions are cropped based on the results of one or more medical imaging analysis tasks. For example, the (one or more) cropped regions can be a predefined sized region cropped around a segmentation result, a detection result, a bounding box, etc.

[0037] In one example, the guided review can begin with a full image review of one or more input medical images at a standard resolution, after which the one or more input medical images will be zoomed in to a cropped area depicting a first finding. After receiving user input from the user confirming receipt of a decision or diagnosis regarding the first finding, the one or more input medical images will be zoomed out and then zoomed in to a cropped area depicting a second finding. After receiving user input from the user confirming receipt of a decision or diagnosis regarding the second finding, the one or more input medical images will be zoomed out and then zoomed in to a cropped area depicting a third finding. The guided review will continue based on the remaining findings. At each stage, the viewer parameters (e.g., reconstruction resolution, zoom level, window / level, overlay selection) of the image viewer used to present the guided review will be determined by the clinical task and model output (e.g., finding proportion and uncertainty). Where appropriate, high-resolution image information from PCCT images can be used to balance the clinical benefit of the additional information with the time cost of reviewing the additional information. During this process, appropriate images from PCCT are shown based on the diagnostic task, for example, the original PCCT image or a material decomposition image for an iodine contrast-enhanced imaging task.

[0038] For finding review, if the uncertainty of the finding from the machine learning-based model is high, or if the finding itself is small in size (e.g., as compared to a threshold), the finding in the ultra-high-resolution PCCT image can be magnified at a high level to show the details of the finding. If the uncertainty of the finding from the machine learning-based model is low, or if the finding itself is large in size (e.g., as compared to a threshold), the finding in the PCCT image can be magnified at a standard level to show the full picture of the finding and accelerate diagnosis. For example, for small lung nodules with high detection uncertainty, the findings on the PCCT image will be magnified at a high level to show the details. In this way, the ultra-high resolution of PCCT images is utilized to optimize the workflow of radiologists.

[0039] In addition to using the ultra-high resolution of PCCT images, for different types of partitions or findings predicted by the machine learning-based model, corresponding types of images derived from the PCCT images can be automatically loaded and shown. For example, in the case of a finding of calcified coronary plaque, a calcium map derived from the PCCT image acquisition will be used. In the case of a finding of a coronary stent in CT angiography, an iodine map will be shown. In the case of a finding of knee osteoarthritis, a virtual monoenergetic image at 60keV (kiloelectronvolt) is calculated and shown because it provides sufficient soft tissue contrast to characterize surfaces, fractures, cartilage calcifications, bone hyperplasia, and bone cysts, and the volume and density of bone cysts can be quantified.

[0040] exist Figure 2 At step 212, the guidance review of one or more input medical images is output. In one embodiment, by displaying the medical image on a display device (e.g., Figure 6 In one example, the guidance review is outputted by presenting the guidance review to the user on the I / O 608 of the device. Figure 1 As shown in the workflow 100 of FIG. 1 , the guided review 118 of the PCCT image 104 is presented 120 to the user. However, the guided review may also be presented 120 to the user by, for example, storing the guided review in a memory or storage device (e.g., Figure 6 602 memory 610 or storage device 612), or by transmitting the guided review to a remote computer system (e.g., Figure 6 Computer 602) to output guidance review.

[0041] Advantageously, the image viewer for generating a guided review of one or more input medical images according to the embodiments described herein is oriented towards clinical workflow. The image viewer is not only used to view medical images, but also optimizes and accelerates the clinical workflow. Using a language model selected for the clinical task and a machine learning-based model for performing the medical imaging analysis task, the image viewer generates a guided review of one or more input medical images to guide the radiologist in identifying where to focus and what to look for, while guiding the radiologist in walking through each finding to facilitate clinical decision-making. Less information can be reviewed more quickly without the need for additional information. In this way, the clinical workflow of the radiology department can be optimized and accelerated by the image viewer.

[0042] Furthermore, the image viewer according to the embodiments described herein is empowered by machine learning-based models. Different sets of machine learning-based models are used for different clinical tasks. The results of the medical imaging analysis tasks and the corresponding uncertainty information output by the machine learning-based models are used to optimize clinical workflows and PCCT image visualization.

[0043] Furthermore, image views according to embodiments described herein utilize additional information provided by PCCT images. Different types of images from PCCT (e.g., ultra-high resolution PCCT images, material decomposition images, and virtual monoenergetic images of PCCT images) are displayed in the image viewer using different machine learning-based models in appropriate configurations (e.g., zoom level, material for material decomposition images, keV for virtual monoenergetic images, etc.) to best facilitate and accelerate the diagnostic workflow.

[0044] The embodiments described herein are described with respect to the claimed systems and claimed methods. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. In other words, claims and embodiments directed to systems may be improved upon by utilizing features described or claimed in the context of the corresponding methods. In this case, the functional features of the methods are implemented by the physical units of the system.

[0045] Further, certain embodiments described herein are described with respect to methods and systems for utilizing trained machine learning models, and with respect to methods and systems for providing trained machine learning models. Features, advantages, or alternative embodiments herein may be assigned to other claimed subject matter, and vice versa. In other words, claims and embodiments for providing trained machine learning models may be improved using features described or claimed in the context of utilizing trained machine learning models, and vice versa. In particular, the datasets used in the methods and systems for utilizing trained machine learning models may have the same properties and characteristics as the corresponding datasets used in the methods and systems for providing trained machine learning models, and the trained machine learning models provided by the corresponding methods and systems may be used in the methods and systems for utilizing trained machine learning models.

[0046] Generally speaking, a trained machine learning model mimics the cognitive functions that humans associate with other human minds. In particular, by training on training data, a machine learning model is able to adapt to new environments and detect and reason about patterns. Another term for a "trained machine learning model" is "trained function."

[0047] Generally speaking, the parameters of a machine learning model can be adapted using training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of a machine learning model can be iteratively adapted through several training steps. In particular, during training, a cost function can be minimized. In particular, in training neural networks, a backpropagation algorithm can be used.

[0048] In particular, machine learning models (such as Figure 1 The selected set of DL models 110 and Figure 2 The language model utilized at step 204 and the one or more machine learning-based models selected and applied at steps 206 and 208 may include, for example, a neural network, a support vector machine, a decision tree, and / or a Bayesian network, and / or the machine learning model may be based on, for example, k-means clustering, Q-learning, a genetic algorithm, and / or association rules. In particular, the neural network may be, for example, a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, the neural network may be, for example, an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0049] Figure 3 An embodiment of an artificial neural network 300 is shown that can be used to implement one or more machine learning models described herein. Alternative terms for "artificial neural network" are "neural network," "artificial neural net," or "neural net."

[0050] The artificial neural network 300 includes nodes 320, ..., 332 and edges 340, ..., 342, wherein each edge 340, ..., 342 is a directed connection from a first node 320, ..., 332 to a second node 320, ..., 332. Generally, the first node 320, ..., 332 and the second node 320, ..., 332 are different nodes 320, ..., 332. It is also possible that the first node 320, ..., 332 and the second node 320, ..., 332 are the same. For example, in Figure 3 , edge 340 is a directed connection from node 320 to node 323, and edge 342 is a directed connection from node 330 to node 332. The edges 340, ..., 342 from the first nodes 320, ..., 332 to the second nodes 320, ..., 332 are also labeled as “input edges” to the second nodes 320, ..., 332 and “output edges” to the first nodes 320, ..., 332.

[0051] In this embodiment, the nodes 320, ..., 332 of the artificial neural network 300 can be arranged in layers 310, ..., 313, where these layers can include an inherent order introduced by edges 340, ..., 342 between the nodes 320, ..., 332. In particular, edges 340, ..., 342 can exist only between adjacent node layers. In the illustrated embodiment, the input layer 310 includes only nodes 320, ..., 322 without input edges, the output layer 313 includes only nodes 331, 332 without output edges, and hidden layers 311, 312 are located between the input layer 310 and the output layer 313. In general, the number of hidden layers 311, 312 can be arbitrarily selected. The number of nodes 320, ..., 322 within the input layer 310 generally correlates with the number of input values ​​of the neural network, while the number of nodes 331, 332 within the output layer 313 generally correlates with the number of output values ​​of the neural network.

[0052] In particular, a (real) number may be assigned as a value to each node 320, ..., 332 of the neural network 300. Here, x (n) i Indicates the value of the i-th node 320, ..., 332 of the n-th layer 310, ..., 313. The values ​​of the nodes 320, ..., 322 of the input layer 310 are equivalent to the input values ​​of the neural network 300, and the values ​​of the nodes 331, 332 of the output layer 313 are equivalent to the output values ​​of the neural network 300. Furthermore, each edge 340, ..., 342 may include a weight as a real number, in particular, the weight is a real number in the interval [-1, 1] or in the interval [0, 1]. Here, w (m，n) i，j The weight of the edge between the i-th node 320, ..., 332 of the m-th layer 310, ..., 313 and the j-th node 320, ..., 332 of the n-th layer 310, ..., 313 is indicated. (n) i，j is defined as the weight w (n，n+1) i，j .

[0053] In particular, in order to calculate the output value of the neural network 300, the input value is propagated through the neural network. Specifically, the values ​​of the nodes 320, ..., 332 of the (n+1)th layer 310, ..., 313 can be calculated based on the values ​​of the nodes 320, ..., 332 of the nth layer 310, ..., 313 by the following formula:

[0054]

[0055] Herein, function f is a transfer function (another term is "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logistic function, generalized logistic function, hyperbolic tangent function, inverse tangent function, error function, smoothed step function), or rectifier functions. Transfer functions are primarily used for normalization purposes.

[0056] In particular, these values ​​are propagated layer by layer through the neural network, where the value of the input layer 310 is given by the input of the neural network 300, where the value of the first hidden layer 311 can be calculated based on the value of the input layer 310 of the neural network, where the value of the second hidden layer 312 can be calculated based on the value of the first hidden layer 311, and so on.

[0057] To set the value for the edge Training data must be used to train the neural network 300. In particular, the training data includes training input data and training output data (denoted as t i ). For the training step, the neural network 300 is applied to the training input data to generate the calculated output data. In particular, the training data and the calculated output data include a number of values, the number of which is equal to the number of nodes in the output layer.

[0058] In particular, the comparison between the calculated output data and the training data is used to recursively adapt the weights within the neural network 300 (back propagation algorithm). Specifically, the weights are changed according to

[0059]

[0060] where γ is the learning rate, and the number δ (n) j can be recursively calculated as

[0061]

[0062] Based on δ (n+1) j , if the (n+1)th layer is not the output layer, and

[0063]

[0064] If the (n+1)th layer is the output layer 313, where f' is the first derivative of the activation function, and t (n+1) j is the comparative training value of the j-th node of the output layer 313.

[0065] A convolutional neural network is a neural network that uses convolution operations instead of general matrix multiplication in at least one of its layers (the so-called "convolutional layer"). Specifically, a convolutional layer performs a dot product of one or more convolution kernels with the input data / image of the convolutional layer, where the entries of the one or more convolution kernels are parameters or weights adapted through training. In particular, Frobenius inner products and ReLU activation functions can be used. A convolutional neural network can include additional layers such as pooling layers, fully connected layers, and normalization layers.

[0066] By using convolutional neural networks, input images can be processed in a very efficient manner because convolution operations based on different kernels can extract various image features, so that by adapting the weights of the convolution kernels, relevant image features can be found during training. Furthermore, due to the sharing of weights in the convolution kernels, fewer parameters need to be trained, which prevents overfitting during the training phase and allows for faster training or more layers in the network, thereby improving the performance of the network.

[0067] Figure 4 An embodiment of a convolutional neural network 400 is shown that can be used to implement one or more machine learning models described herein. In the embodiment shown, the convolutional neural network 400 includes an input node layer 410, a convolutional layer 411, a pooling layer 413, a fully connected layer 414, and an output node layer 416, as well as hidden node layers 412, 414. Alternatively, the convolutional neural network 400 can include several convolutional layers 411, several pooling layers 413, and several fully connected layers 415, as well as other types of layers. The order of the layers can be chosen arbitrarily, and typically the fully connected layer 415 is used as the last layer before the output layer 416.

[0068] Specifically, within the convolutional neural network 400, the nodes 420, 422, 424 of the node layers 410, 412, 414 can be viewed as being arranged as a d-dimensional matrix or a d-dimensional image. Specifically, in the two-dimensional case, the values ​​of the nodes 420, 422, 424 indexed by i and j in the n-th node layer 410, 412, 414 can be denoted as x(n)[i, j]. However, the arrangement of the nodes 420, 422, 424 of a node layer 410, 412, 414 has no effect on the computations performed within the convolutional neural network 400, as these are given only by the structure and weights of the edges.

[0069] The convolution layer 411 is a connection layer between the previous node layer 410 (with node value x(n-1)) and the subsequent node layer 412 (with node value x(n)). In particular, the convolution layer 411 is characterized by the structure and weight of the input edges that form the convolution operation based on a certain number of kernels. Specifically, the structure and weight of the edges of the convolution layer 411 are selected so that the value x(n) of the node 422 of the subsequent node layer 412 is calculated as the convolution x(n) = K*x(n-1) based on the value x(n-1) of the node 420 of the previous node layer 410, where the convolution* is defined in the two-dimensional case as

[0070]

[0071] Here, the kernel K is a d-dimensional matrix (in this embodiment, a two-dimensional matrix), which is typically small compared to the number of nodes 420, 422 (e.g., a 3×3 matrix or a 5×5 matrix). In particular, this implies that the weights of the edges in the convolution layer 411 are not independent, but are chosen so that they produce the convolution equation. In particular, for a 3×3 kernel, there are only 9 independent weights (each entry in the kernel matrix corresponds to an independent weight), regardless of the number of nodes 420, 422 in the preceding node layer 410 and the following node layer 412.

[0072] In general, and particularly due to the use of multiple kernels in the convolutional layer 411, the convolutional neural network 400 uses node layers 410, 412, 414 with multiple channels. In these cases, the node layers can be thought of as (d+1)-dimensional matrices (the first dimension indexes the channels). The action of the convolutional layer 411 is then a two-dimensional example, defined as

[0073]

[0074] in corresponding to the ath channel of the front node layer 410, corresponds to the bth channel of the back node layer 412, and K a，b Corresponding to one of the kernels. If the convolutional layer 411 acts on the front node layer 410 with A channels and outputs the back node layer 412 with B channels, then there is an A·B independent d-dimensional kernel K a，b .

[0075] Generally speaking, an activation function is used in the convolutional neural network 400. In this embodiment, re ReLU (the acronym for "rectified linear unit") is used, where R(z) = max(0, z), so that the action of the convolution layer 411 in the two-dimensional example is

[0076]

[0077] Other activation functions may also be used, such as ELU (acronym for “Exponential Linear Unit”), LeakyReLU, Sigmoid, Tanh, or Softmax.

[0078] In the embodiment shown, the input layer 410 includes 36 nodes 420 arranged in a two-dimensional 6×6 matrix. The first hidden node layer 412 includes 72 nodes 422 arranged in two two-dimensional 6×6 matrices, each of which is the result of convolving the values ​​of the input layer with a 3×3 kernel within the convolution layer 411. Equivalently, the nodes 422 of the first hidden node layer 412 can be interpreted as being arranged in a three-dimensional 2×6×6 matrix, where the first dimension corresponds to the channel dimension.

[0079] The advantage of using convolutional layers 411 is that the spatial local correlation of the input data can be exploited by enforcing local connectivity patterns between nodes in adjacent layers, in particular by having each node connected to only a small region of nodes in the previous layer.

[0080] The pooling layer 413 is a connection layer between the front node layer 412 (with node value x(n-1)) and the back node layer 414 (with node value x(n)). Specifically, the pooling layer 413 can be characterized by the structure and weights of the edges and activation functions that form the pooling operation based on the nonlinear pooling function f. For example, in the two-dimensional case, the value x(n) of the node 424 of the back node layer 414 can be calculated based on the value x(n-1) of the node 422 of the front node layer 412 as

[0081]

[0082] In other words, by using pooling layer 413, the number of nodes 422, 424 can be reduced by replacing the number d1·d2 of neighboring nodes 422 in the preceding node layer 412 with a single node 422 calculated as a function of the values ​​of the number of neighboring nodes in the following node layer 414. In particular, the pooling function f can be a maximum function, an average value, or an L2 norm. In particular, for pooling layer 413, the weights of the input edges are fixed and are not modified during training.

[0083] The advantage of using the pooling layer 413 is that the number of nodes 422, 424 and the number of parameters are reduced. This leads to a reduction in the amount of computation in the network and controls overfitting.

[0084] In the embodiment shown, pooling layer 413 is a max pooling layer that replaces four adjacent nodes with only one node whose value is the maximum of the values ​​of the four adjacent nodes. Max pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, max pooling is applied to each of the two 2-dimensional matrices, reducing the number of nodes from 72 to 18.

[0085] Generally speaking, the last few layers of the convolutional neural network 400 are fully connected layers 415. The fully connected layer 415 is a connection layer between the previous node layer 414 and the next node layer 416. The fully connected layer 413 is characterized by the fact that most, especially all, edges exist between the nodes 414 of the previous node layer 414 and the nodes 416 of the next node layer, and the weight of each of these edges can be adjusted separately.

[0086] In this embodiment, the nodes 424 of the preceding node layer 414 of the fully connected layer 415 are each displayed as a two-dimensional matrix, and are additionally displayed as unrelated nodes (indicated as a row of nodes, where the number of nodes is reduced for better presentation). This operation is also called "flattening." In this embodiment, the number of nodes 426 in the subsequent node layer 416 of the fully connected layer 415 is less than the number of nodes 424 in the preceding node layer 414. Alternatively, the number of nodes 426 can be equal to or greater.

[0087] Furthermore, in this embodiment, a Softmax activation function is used within the fully connected layer 415. By applying the Softmax function, the sum of the values ​​of all nodes 426 of the output layer 416 is 1, and all values ​​of all nodes 426 of the output layer 416 are real numbers between 0 and 1. In particular, if the convolutional neural network 400 is used to classify input data, the value of the output layer 416 can be interpreted as the probability that the input data falls into one of the different categories.

[0088] In particular, the convolutional neural network 400 can be trained based on a back-propagation algorithm. To prevent overfitting, regularization methods can be used, such as dropping nodes 420, ..., 424, random pooling, using artificial data, weight decay based on L1 or L2 norms, or maximum norm constraints.

[0089] According to one aspect, the machine learning model may include one or more residual networks (ResNets). Specifically, ResNet is an artificial neural network comprising at least one jump or skip connection for skipping at least one layer of the artificial neural network. In particular, ResNet may be a convolutional neural network comprising one or more skip connections that skip one or more convolutional layers, respectively. According to some examples, ResNet may be represented as an m-layer ResNet, where m is the number of layers in the corresponding architecture, and according to some examples, may take values ​​of 34, 50, 101, or 152. According to some examples, such an m-layer ResNet may each include (m-2) / 2 skip connections.

[0090] A skip connection can be thought of as a bypass that feeds the output of a previous layer directly to the layer following the bypassed layer(s), via the bypassed layer(s). The bypassed layer(s) will then have to fit a residual map that "balances" the directly fed output, rather than having to fit the desired map directly.

[0091] Fitting a residual map is computationally easier to optimize than a directed map. More importantly, this alleviates the problem of vanishing / exploding gradients during optimization when training machine learning models: if a bypassed layer encounters such a problem, its contribution can be skipped by regularizing the directly fed output. Therefore, the advantage of using ResNet is that deeper networks can be trained.

[0092] In particular, a recursive machine learning model is a machine learning model whose output depends not only on the input values ​​and the parameters of the machine learning model adapted by the training process, but also on a hidden state vector, where the hidden state vector is based on previous inputs to the recursive machine learning model. Specifically, a recursive machine learning model can include additional memory states or additional structures that incorporate time delays or include feedback loops.

[0093] In particular, the underlying structure of the recursive machine learning model can be a neural network, which can be represented as a recursive neural network. Such a recursive neural network can be described as an artificial neural network in which the connections between nodes form a directed graph along a time series. In particular, the recursive neural network can be interpreted as a directed acyclic graph. In particular, the recursive neural network can be a finite-spiked recursive neural network or an infinite-spiked recursive neural network (wherein a finite-spiked network can be unfolded and replaced by a strictly feedforward neural network, and an infinite-spiked network cannot be unfolded and replaced by a strictly feedforward neural network).

[0094] In particular, training a recurrent neural network can be based on a BPTT algorithm (acronym for “Back Propagation Through Time”), an RTRL algorithm (acronym for “Real-Time Recursive Learning”) and / or a genetic algorithm.

[0095] By using a recursive machine learning model, input data comprising sequences of variable length can be used. In particular, this means that the method cannot be used only for a fixed number of input datasets (and would need to be trained differently for each additional number of input datasets used as input), but can be used for any number of input datasets. This means that, independent of the number of input datasets contained in the different sequences, the entire training dataset can be used in training, and the training data is not reduced to training data corresponding to a certain number of consecutive input datasets.

[0096] Figure 5The schematic structure of a recursive machine learning model F is shown in a recursive representation 502 and an expanded representation 504, which can be used to implement one or more machine learning models described herein. The recursive machine learning model takes several input data sets x, x1, ..., x N 506 as input and create a corresponding set of output data sets y, y1, ..., y N 508. Furthermore, the output depends on the so-called hidden vectors h, h1, ..., h N 510, which implicitly includes information about the input dataset that was previously used as input to the recursive machine learning model F 512. By using these latent vectors h, h1, ..., h N 510, the sequential nature of the input dataset can be exploited.

[0097] In a single step of processing, the recursive machine learning model F 512 transforms the hidden vector h created in the previous step n-1 and the input dataset x n As input. In this step, the recursive machine learning model F generates an updated hidden vector h n and the output dataset y n As output. In other words, a processing step computes (y n , h n )=F(x n , h n-1 ), or by dividing the recursive machine learning model F 512 into a part F(y) for calculating the output data and a part F(h) for calculating the hidden vector, one processing step calculates y n =F (y) (x n , h n-1 ) and h n =F (h) (x n , h n-1 ). For the first processing step, h0 can be randomly selected or filled with all zero entries. The parameters of the recursive machine learning model F512 previously trained based on the training dataset remain unchanged between different processing steps.

[0098] In particular, the output data and hidden vectors of a processing step depend on all previous input datasets used in previous steps. n =F (y) (x n , F (h) (x n-1 , h n-2 )) and h n =F (h) (x n , F (h) (xn-1 , h n-2 )).

[0099] The systems, apparatus, and methods described herein can be implemented using digital circuitry or using one or more computers utilizing known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or be coupled to one or more mass storage devices, such as one or more magnetic disks, internal hard disks and removable disks, magneto-optical disks, optical disks, and the like.

[0100] The systems, devices, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is remote from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0101] The systems, devices, and methods described herein can be implemented in a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor connected to the network communicates with one or more client computers via the network. For example, a client computer can communicate with a server via a web browser application resident and running on the client computer. The client computer can store data on the server and access the data via the network. The client computer can send data requests or online service requests to the server via the network. The server can perform the requested service and provide the data to the (one or more) client computers. The server can also transmit data that is adapted to cause the client computer to perform a specific function, such as performing a calculation, displaying specific data on a screen, etc. For example, the server can send a request that is adapted to cause the client computer to perform one or more steps or functions of the methods and workflows described herein, including Figure 1 or one or more steps or functions of 2. Certain steps or functions of the methods and workflows described herein include Figure 1 or Figure 2 One or more steps or functions of the method and workflow described herein may be performed by a server or another processor in a network-based cloud computing system. Figure 1 or Figure 2 One or more steps of the method and workflow described herein may be performed by a client computer in a network-based cloud computing system. Figure 1 or Figure 2 One or more steps may be performed by a server and / or a client computer in a network-based cloud computing system in any combination.

[0102] The systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier, such as a non-transitory machine-readable storage device, for execution by a programmable processor; and the method and workflow steps described herein, including Figure 1 or Figure 2 One or more steps or functions of a program may be implemented using one or more computer programs executable by such a processor. A computer program is a set of computer program instructions that can be used, directly or indirectly, in a computer to perform a specific activity or produce a specific result. A computer program may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0103] Figure 6 A high-level block diagram of an example computer 602 that can be used to implement the systems, apparatus, and methods described herein is depicted in FIG. The computer 602 includes a processor 604 operatively coupled to a data storage device 612 and a memory 610. The processor 604 controls the operation of the computer 602 by executing computer program instructions that define the overall operation. The computer program instructions may be stored in the data storage device 612 or other computer-readable medium and loaded into the memory 610 when execution of the computer program instructions is desired. Thus, Figure 1 or Figure 2 The method and workflow steps or functions may be defined by computer program instructions stored in the memory 610 and / or the data storage device 612 and controlled by the processor 604 executing the computer program instructions. For example, the computer program instructions may be implemented as computer executable code programmed by a person skilled in the art to perform Figure 1 or Figure 2 Thus, by executing the computer program instructions, the processor 604 performs Figure 1 or Figure 2 The computer 602 may also include one or more network interfaces 606 for communicating with other devices via a network. The computer 602 may also include one or more input / output devices 608 that enable a user to interact with the computer 602 (e.g., a display, keyboard, mouse, speakers, buttons, etc.).

[0104] The processor 604 may include both general-purpose and special-purpose microprocessors and may be the sole processor or one of multiple processors of the computer 602. For example, the processor 604 may include one or more central processing units (CPUs). The processor 604, the data storage device 612, and / or the memory 610 may include, be supplemented by, or incorporate one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs).

[0105] The data storage device 612 and the memory 610 each include a tangible, non-transitory computer-readable storage medium. The data storage device 612 and the memory 610 may each include a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), digital versatile disk read-only memory (DVD-ROM), or other non-volatile solid-state storage devices.

[0106] The input / output devices 608 may include peripheral devices such as printers, scanners, display screens, etc. For example, the input / output devices 608 may include a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor for displaying information to a user, a keyboard, and a pointing device such as a mouse or trackball through which a user can provide input to the computer 602.

[0107] An image acquisition device 614 may be connected to the computer 602 to input image data (e.g., medical images) into the computer 602. The image acquisition device 614 and the computer 602 may be implemented as a single device. The image acquisition device 614 and the computer 602 may also communicate wirelessly over a network. In a possible embodiment, the computer 602 may be remotely located relative to the image acquisition device 614.

[0108] Any or all of the systems, apparatus, and methods discussed herein may be implemented using one or more computers, such as computer 602 .

[0109] Those skilled in the art will recognize that actual computer or computer system implementations may have other structures and may include other components, and Figure 6 is a high-level representation of some of the components of such a computer for illustrative purposes.

[0110] Independent of the use of the grammatical term, individuals with either male or female gender identities are included in the term.

[0111] The foregoing detailed description should be understood to be illustrative and exemplary in every respect and not restrictive, and the scope of the invention disclosed herein is not determined by the detailed description, but by the claims interpreted to the maximum extent permitted by patent law. It should be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and that those skilled in the art may implement various modifications without departing from the scope and spirit of the invention. Various other feature combinations may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

[0112] The following is a list of non-limiting illustrative embodiments disclosed herein:

[0113] Illustrative Embodiment 1. A computer-implemented method includes: receiving one or more input medical images of a patient and text-based patient data of the patient; identifying one or more clinical tasks based on the text-based patient data using a language model; selecting one or more machine learning-based models based on the identified one or more clinical tasks; performing one or more medical imaging analysis tasks based on the one or more input medical images using the selected one or more machine learning-based models; generating a guided review of the one or more input medical images based on results of the one or more medical imaging analysis tasks; and outputting the guided review of the one or more input medical images.

[0114] Illustrative embodiment 2. The computer-implemented method of illustrative embodiment 1, wherein generating a guided review of the one or more input medical images based on the results of the one or more medical imaging analysis tasks comprises: generating a guided review comprising a full image review of the one or more input medical images, a partitioned review delineating one or more anatomical cropped regions of the one or more input medical images, and a finding review delineating one or more pathological cropped regions of the one or more input medical images.

[0115] Illustrative embodiment 3. The computer-implemented method of any of illustrative embodiments 1-2, wherein outputting the guided review of the one or more input medical images comprises presenting the guided review of the one or more input medical images to a user via a display device.

[0116] Illustrative Embodiment 4. The computer-implemented method of any of Illustrative Embodiments 1-3, wherein presenting the guided review of one or more input medical images to a user via a display device comprises: delineating a cropped region depicting a first finding; receiving user input from the user confirming a decision regarding the first finding; and automatically delineating a cropped region depicting a second finding in response to receiving the user input.

[0117] Illustrative Embodiment 5. The computer-implemented method of any of Illustrative Embodiments 1-4, wherein presenting the guided review of the one or more input medical images to the user via a display device comprises presenting the guided review of the one or more input medical images according to parameters determined based on the identified one or more clinical tasks and the output of the selected one or more machine learning-based models.

[0118] Illustrative Embodiment 6. The computer-implemented method of any of Illustrative Embodiments 1-5, wherein the language model receives text-based data as input and generates one or more vectors corresponding to one or more clinical tasks as output.

[0119] Illustrative Example 7. The computer-implemented method of any of Illustrative Examples 1-6, wherein selecting one or more machine-learning-based models based on the identified one or more clinical tasks comprises: selecting one or more machine-learning-based models from a database of pre-trained machine-learning-based models, the pre-trained machine-learning-based models including machine-learning-based models for performing different medical imaging analysis tasks and machine-learning-based models with different sensitivities.

[0120] Illustrative embodiment 8. The computer-implemented method of any of illustrative embodiments 1-7, wherein the one or more input medical images include one or more PCCT (photon counting computed tomography) images.

[0121] Illustrative embodiment 9. The computer-implemented method of any of illustrative embodiments 1-8, wherein the language model is an LLM (Large Language Model).

[0122] Illustrative embodiment 10. An apparatus comprising: means for receiving one or more input medical images of a patient and text-based patient data of the patient; means for identifying one or more clinical tasks based on the text-based patient data using a language model; means for selecting one or more machine learning-based models based on the identified one or more clinical tasks; means for performing one or more medical imaging analysis tasks based on the one or more input medical images using the selected one or more machine learning-based models; means for generating a guided review of the one or more input medical images based on results of the one or more medical imaging analysis tasks; and means for outputting the guided review of the one or more input medical images.

[0123] Illustrative embodiment 11. The apparatus of illustrative embodiment 10, wherein the means for generating a guided review of the one or more input medical images based on the results of the one or more medical imaging analysis tasks comprises means for generating the guided review, the guided review comprising a full image review of the one or more input medical images, a zonal review delineating one or more anatomical cropped regions of the one or more input medical images, and a finding review delineating one or more pathological cropped regions of the one or more input medical images.

[0124] Illustrative Embodiment 12. The apparatus of any of Illustrative Embodiments 10-11, wherein the means for outputting the guided review of the one or more input medical images comprises: means for presenting the guided review of the one or more input medical images to a user via a display device.

[0125] Illustrative Embodiment 13. The apparatus of any of Illustrative Embodiments 10-12, wherein the means for presenting a guided review of one or more input medical images to a user via a display device comprises: means for delineating a cropped region depicting a first finding; means for receiving user input from a user confirming a decision regarding the first finding; and means for automatically delineating a cropped region depicting a second finding in response to receiving the user input.

[0126] Illustrative Example 14. The apparatus of any of Illustrative Examples 10-13, wherein the means for presenting a guided review of one or more input medical images to a user via a display device comprises means for presenting the guided review of the one or more input medical images according to parameters determined based on the identified one or more clinical tasks and the output of the selected one or more machine learning-based models.

[0127] Illustrative Embodiment 15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising: receiving one or more input medical images of a patient and text-based patient data of the patient; identifying one or more clinical tasks based on the text-based patient data using a language model; selecting one or more machine learning-based models based on the identified one or more clinical tasks; performing one or more medical imaging analysis tasks based on the one or more input medical images using the selected one or more machine learning-based models; generating a guided review of the one or more input medical images based on results of the one or more medical imaging analysis tasks; and outputting the guided review of the one or more input medical images.

[0128] Illustrative embodiment 16. The non-transitory computer-readable storage medium of illustrative embodiment 15, wherein generating a guided review of the one or more input medical images based on the results of the one or more medical imaging analysis tasks comprises: generating a guided review comprising a full image review of the one or more input medical images, a partitioned review delineating one or more anatomical cropped regions of the one or more input medical images, and a finding review delineating one or more pathological cropped regions of the one or more input medical images.

[0129] Illustrative Embodiment 17. The non-transitory computer-readable storage medium of any one of Illustrative Embodiments 15-16, wherein the language model receives text-based data as input and generates one or more vectors corresponding to one or more clinical tasks as output.

[0130] Illustrative Example 18. The non-transitory computer-readable storage medium of any of Illustrative Example 15-Illustrative Example 17, wherein selecting one or more machine learning-based models based on the identified one or more clinical tasks comprises: selecting one or more machine learning-based models from a database of pre-trained machine learning-based models, the pre-trained machine learning-based models including machine learning-based models for performing different medical imaging analysis tasks and machine learning-based models with different sensitivities.

[0131] Illustrative Embodiment 19. The non-transitory computer-readable storage medium of any one of Illustrative Embodiments 15-18, wherein the one or more input medical images include one or more PCCT (Photon Counting Computed Tomography) images.

[0132] Illustrative Embodiment 20. The non-transitory computer-readable storage medium of any one of Illustrative Embodiments 15-19, wherein the language model is an LLM (Large Language Model).

Claims

1. A computer-implemented method comprising: receiving one or more input medical images of a patient and text-based patient data of the patient; identifying one or more clinical tasks based on the text-based patient data using a language model; selecting one or more machine learning-based models based on the identified one or more clinical tasks; performing one or more medical imaging analysis tasks based on the one or more input medical images using the selected one or more machine learning-based models; generating a guided review of the one or more input medical images based on results of the one or more medical imaging analysis tasks; as well as The guided review of the one or more input medical images is output.

2. The computer-implemented method of claim 1 , wherein generating a guided review of the one or more input medical images based on the results of the one or more medical imaging analysis tasks comprises: The guided review is generated, the guided review comprising a full image review of the one or more input medical images, a zonal review delineating one or more anatomical cropped regions of the one or more input medical images, and a finding review delineating one or more pathological cropped regions of the one or more input medical images.

3. The computer-implemented method of claim 1 , wherein outputting the guided review of the one or more input medical images comprises: The guided review of the one or more input medical images is presented to a user via a display device.

4. The computer-implemented method of claim 3 , wherein presenting the guided review of the one or more input medical images to a user via a display device comprises: Describe the cropped region depicting the first finding; receiving user input from the user confirming a decision regarding the first finding; as well as In response to receiving the user input, a crop region is automatically delineated that delineates a second finding.

5. The computer-implemented method of claim 3 , wherein presenting the guided review of the one or more input medical images to a user via a display device comprises: The guided review of the one or more input medical images is presented according to parameters determined based on the identified one or more clinical tasks and outputs of the selected one or more machine learning-based models.

6. The computer-implemented method of claim 1, wherein the language model receives text-based data as input and generates one or more vectors corresponding to the one or more clinical tasks as output.

7. The computer-implemented method of claim 1 , wherein selecting one or more machine learning-based models based on the identified one or more clinical tasks comprises: The one or more machine learning-based models are selected from a database of pre-trained machine learning-based models, wherein the pre-trained machine learning-based models include machine learning-based models for performing different medical imaging analysis tasks and machine learning-based models with different sensitivities.

8. The computer-implemented method of claim 1, wherein the one or more input medical images comprise one or more PCCT (Photon Counting Computed Tomography) images.

9. The computer-implemented method of claim 1, wherein the language model is an LLM (Large Language Model).

10. A device comprising: means for receiving one or more input medical images of a patient and text-based patient data of the patient; means for identifying one or more clinical tasks based on the text-based patient data using a language model; means for selecting one or more machine learning-based models based on the identified one or more clinical tasks; means for performing one or more medical imaging analysis tasks based on the one or more input medical images using the selected one or more machine learning-based models; means for generating a guided review of the one or more input medical images based on results of the one or more medical imaging analysis tasks; as well as Means for outputting the guided review of the one or more input medical images.

11. The apparatus of claim 10, wherein the means for generating a guided review of the one or more input medical images based on the results of the one or more medical imaging analysis tasks comprises: Means for generating the guided review, the guided review comprising a full image review of the one or more input medical images, a zonal review delineating one or more anatomical cropped regions of the one or more input medical images, and a finding review delineating one or more pathological cropped regions of the one or more input medical images.

12. The apparatus of claim 10, wherein the means for outputting the guided review of the one or more input medical images comprises: Means for presenting the guided review of the one or more input medical images to a user via a display device.

13. The apparatus of claim 12, wherein the means for presenting the guided review of the one or more input medical images to a user via a display device comprises: A component for depicting a clipping region depicting a first finding; means for receiving user input from the user confirming a decision regarding the first finding; as well as Means for automatically delineating a cropping region delineating a second finding in response to receiving the user input.

14. The apparatus of claim 12, wherein the means for presenting the guided review of the one or more input medical images to a user via a display device comprises: Means for presenting the guided review of the one or more input medical images according to parameters determined based on the identified one or more clinical tasks and outputs of the selected one or more machine learning-based models.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising: receiving one or more input medical images of a patient and text-based patient data of the patient; identifying one or more clinical tasks based on the text-based patient data using a language model; selecting one or more machine learning-based models based on the identified one or more clinical tasks; performing one or more medical imaging analysis tasks based on the one or more input medical images using the selected one or more machine learning-based models; generating a guided review of the one or more input medical images based on results of the one or more medical imaging analysis tasks; as well as The guided review of the one or more input medical images is output.

16. The non-transitory computer-readable storage medium of claim 15, wherein generating a guided review of the one or more input medical images based on the results of the one or more medical imaging analysis tasks comprises: The guided review is generated, the guided review comprising a full image review of the one or more input medical images, a zonal review delineating one or more anatomical cropped regions of the one or more input medical images, and a finding review delineating one or more pathological cropped regions of the one or more input medical images. 17 . The non-transitory computer-readable storage medium of claim 15 , wherein the language model receives text-based data as input and generates one or more vectors corresponding to the one or more clinical tasks as output.

18. The non-transitory computer-readable storage medium of claim 15, wherein selecting one or more machine learning-based models based on the identified one or more clinical tasks comprises: The one or more machine learning-based models are selected from a database of pre-trained machine learning-based models, wherein the pre-trained machine learning-based models include machine learning-based models for performing different medical imaging analysis tasks and machine learning-based models with different sensitivities.

19. The non-transitory computer-readable storage medium of claim 15, wherein the one or more input medical images include one or more PCCT (Photon Counting Computed Tomography) images.

20. The non-transitory computer-readable storage medium of claim 15, wherein the language model is an LLM (Large Language Model).

Images