Algorithmic orchestration of workflows to aid healthcare imaging diagnostics

Abstract

Techniques for orchestrating execution of algorithms for medical images according to predefined workflows, and techniques for managing workflow and model execution are provided. In an embodiment, a system includes: a memory storing computer executable components; and a processor executing the computer executable components stored in the memory. The computer executable components include an algorithm catalog, the catalog including algorithm information identifying algorithms that can be used to process medical images, the algorithm information including algorithm execution instructions for executing the algorithm as a web service; and an orchestration component that adds the algorithm information to the algorithm catalog in response to receiving the algorithm information via a loading user interface of an algorithm management application, wherein based on including the algorithm information in the algorithm catalog, the algorithm is enabled to be incorporated into a workflow for executing the algorithm on the medical image.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Application Serial No. 62 / 939,910, filed on November 25, 2019, entitled “ALGORITHM ORCHESTRATION OF WORKFLOWS TO FACILITATE HEALTHCARE IMAGING DIAGNOSTICS,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to algorithm orchestration in the medical imaging domain, including techniques for orchestrating algorithms for executing on medical images according to a predefined workflow and techniques for managing workflow and model execution. Background Art

[0004] The healthcare industry has countless opportunities to leverage artificial intelligence (AI), machine learning (ML), and other analytical models to enable more accurate, proactive, and comprehensive patient care. From reducing administrative burdens to supporting precision medicine, these analytical tools show promise in clinical, financial, and operational areas. For example, learning algorithms can become more precise and accurate as they interact with training data, allowing humans to gain unprecedented insights into diagnoses, care processes, treatment variability, and patient outcomes. However, even organizations with industry-leading analytical capabilities today face complex challenges when it comes to applying a variety of analytics to clinical care. Summary of the invention

[0005] The following presents a summary of the invention to provide a basic understanding of one or more examples of the invention. The present summary is not intended to identify key or important elements or to delimit any scope of different examples or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a preface to a more detailed description presented later. In one or more examples, described herein are systems, computer-implemented methods, devices, and / or computer program products that facilitate the integration of AI informatics into healthcare systems using a centralized orchestration component that orchestrates the execution of AI algorithms according to predefined workflows. The disclosed subject matter also provides systems, computer-implemented methods, devices, and / or computer program products that facilitate the management of model execution of workflows and algorithm orchestration components.

[0006] According to an example, a system is provided, including a memory storing computer executable components. The system includes a processor executing the computer executable components stored in the memory. The computer executable components include an algorithm catalog, the algorithm catalog including a plurality of diagnostic algorithms for analyzing at least one medical image of a patient and metadata describing attributes of the at least one medical image. An algorithm orchestration component executes workflow instructions describing a computer executable diagnostic routine to be applied to the at least one medical image and the metadata. The algorithm orchestration component selects at least one diagnostic algorithm from the algorithm catalog to execute the diagnostic routine based on the at least one medical image and the metadata in response to the workflow instructions. A workflow execution engine executes the selected at least one diagnostic algorithm from the algorithm catalog in response to a command from the algorithm orchestration component.

[0007] The system may also include a notification component that provides a notification output from the algorithm orchestration component in response to executing the diagnostic routine. The notification output may include at least one of a status output indicating the progress of the diagnostic routine in response to the workflow instructions and a metadata analysis of the metadata.

[0008] In some embodiments, the algorithm orchestration component performs metadata analysis in response to the workflow instruction, the metadata analysis including at least one test to determine the status of the patient, a test to determine the status of the medical image, a test to determine the available algorithms in the algorithm catalog, and a test to determine the status of the diagnostic threshold of the selected algorithm applied to the medical image from the algorithm catalog. The test results may also be included in the notification output. In various embodiments, the analysis data is determined by at least one AI algorithm selected from the algorithm catalog and applied to at least one medical image to diagnose the patient in response to the workflow instruction.

[0009] The system may also include an algorithm management interface including a threshold setting adjustment to enable setting of a diagnostic threshold that defines whether a given medical condition is applicable to a patient based on execution of at least one artificial intelligence algorithm. The system may also include an active dashboard interface that displays at least one of: an overview of potential medical conditions determined by the artificial intelligence algorithm, multiple studies used to analyze the potential medical conditions, and multiple patients previously analyzed. In some implementations, the active dashboard displays at least one of an artificial intelligence prediction of a medical diagnostic condition and a selected radiologist diagnosis of the medical diagnostic condition, at least one other artificial intelligence prediction of the medical diagnostic condition and another selected radiologist diagnosis of the medical diagnostic condition, and a confidence output displaying a false positive diagnosis rate and a true positive diagnosis rate.

[0010] The system may also include a workflow configuration tool to enable at least one of selection, configuration, and editing of workflow instructions. In some embodiments, the workflow configuration tool also includes a simulator tool to enable testing of workflow instructions based on a set of test medical images and test metadata associated with the test medical images, and then downloading the workflow instructions to the algorithm orchestration component. The workflow configuration tool includes at least one template for executing initial workflow instructions for a given workflow, the at least one template including a start workflow instruction and an end workflow instruction that define the start and end of a given workflow. The workflow instructions may include a fork instruction that defines a parallel processing path for a given workflow and a join instruction that combines at least two parallel processing paths of a given workflow. The workflow instructions may also include at least one of the following: a wait instruction that causes a given workflow to wait for an asynchronous task to complete, a decision branch that generates a workflow decision based on a logical expression, and a hypertext transfer (HTTP) task instruction that calls an HTTP service to be initiated by a given workflow. The workflow instructions may also include a sub-workflow instruction that causes another workflow to be executed within a given set of workflow instructions.

[0011] The system may also include at least one security component that defines and executes security protocols to be employed by the algorithm orchestration component when exchanging files with other components. In various implementations, the algorithm orchestration component executes workflow instructions to process a plurality of medical images employed in a three-dimensional reconstruction of a patient from which the images were captured.

[0012] In another example embodiment, a system is provided, comprising a memory storing computer executable components. The system comprises a processor executing the computer executable components stored in the memory. The computer executable components comprise: an algorithm orchestration component that receives a request to process a medical image using at least one medical image inference model; and an orchestration director component that identifies a workflow comprising a medical image inference model applicable to a medical image, wherein the medical image inference model is stored at a network accessible source. The computer executable components may also comprise a workflow execution engine that executes a workflow using the medical image, including accessing the medical image inference model at a network accessible source, and applying the medical image inference model to the medical image, thereby generating a workflow result. The algorithm orchestration component may also provide result information about the workflow result to an entity associated with the request.

[0013] In another example embodiment, a system is provided, comprising a memory storing computer executable components. The system comprises a processor executing the computer executable components stored in the memory. The computer executable components comprise: an algorithm catalog comprising algorithm information identifying algorithms that can be used to process medical images, the algorithm information comprising algorithm execution instructions for executing the algorithms as a web service; and a loading component, the loading component adding the algorithm information to the algorithm catalog in response to receiving the algorithm information via a loading user interface of an algorithm management application, wherein based on including the algorithm information in the algorithm catalog, the algorithm is enabled to be incorporated into a workflow for executing the algorithm on a medical image. In various implementations, the algorithm comprises a medical image inference model. The algorithm may also comprise internal algorithms and third-party algorithms stored at different network accessible file locations.

[0014] The computer executable components may also include: an algorithm orchestration component that receives an image processing request from a medical imaging provider via a network; and a workflow execution component that executes a workflow on a medical image in association with receiving the image processing request and returns a workflow result to the medical imaging provider, the workflow result including an output of one or more of the algorithms. The image processing request is received from a device associated with the medical imaging provider, and the computer executable components may also include a security component that limits acceptance of imaging processing requests to registered devices included in the device registry information.

[0015] The computer executable component may also include an algorithm execution engine that calls the algorithm at the algorithm's network accessible file location to receive output associated with executing the workflow. The computer executable component may also include a conversion component that converts the algorithm format from a first format that is incompatible with the algorithm execution engine to a second format that is compatible with the algorithm execution engine in association with loading. For example, the second format may be compatible with an application program interface employed by the algorithm execution engine. In some implementations, the algorithm management application also provides for adjusting one or more parameters and thresholds of the algorithm in association with receiving the algorithm information.

[0016] The computer executable components may also include a workflow creation component that provides a workflow creation application for creating a workflow using an algorithm based on the algorithm information included in the algorithm catalog. A simulator component may also be provided that runs the workflow in a test mode on one or more test images and identifies execution errors.

[0017] The computer executable components may also include a workflow management component that provides access to workflow information via an algorithm management application, the workflow information defining the workflow and workflow management functions associated with the workflow. In some implementations, the workflow management functions include an activation / deactivation function that provides for selective activation and deactivation of one or more of the workflows. The workflow management functions may also provide for editing a workflow, deleting a workflow, adding tags to a workflow, and exporting and importing a workflow.

[0018] In another example embodiment, a system is provided, comprising a memory storing computer executable components. The system comprises a processor executing the computer executable components stored in the memory. The computer executable components comprise: an algorithm orchestration component that receives an image processing request from a medical imaging provider via a network and identifies a workflow applicable to a medical image associated with the medical image processing request, the workflows respectively comprising one or more medical image inference models; and a workflow execution component that executes the workflow on the medical image in association with receiving the image processing request.

[0019] In various implementations, the computer executable component also includes an activity logging component that tracks activity information about image processing requests and workflows executed for image processing requests. For example, the activity information may include identifying a specific workflow executed for each request and the workflow status (e.g., completed or failed), one or more medical image inference models executed for the image processing request, the device from which the request was received, attributes of the medical image (e.g., study type, study modality, etc.), errors detected in the workflow, etc. The management component may also provide access to the activity information via a management application.

[0020] In some implementations, the computer executable component also includes: an activity analysis component that processes the activity information and generates performance evaluation report information based on the activity information; and a dashboard component that presents the performance evaluation report information via a dashboard user interface of the management application. The workflow execution engine may also generate workflow results and return the workflow results to the medical imaging provider, the workflow results including the output of one or more medical image inference models, and the computer executable component may also include a feedback component that receives feedback on the accuracy of the output of the one or more inference models, wherein the activity analysis component also generates the performance evaluation report information based on the feedback.

[0021] The activity logging component may also generate an activity log of the image processing request, the activity log identifying the workflow executed for each image processing request and the execution status of the workflow. The computer executable component may also include an audit component that provides access to the activity log via the management application and provides an audit function via the activity log for performing a root cause analysis of workflow execution errors encountered with respect to one or more of the workflows.

[0022] In some examples, elements described in connection with the disclosed systems may be embodied in various forms, such as a computer-implemented method, a computer program product, or another form. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A block diagram of an example algorithm orchestration system is shown in accordance with one or more implementations of the disclosed subject matter.

[0024] FIG. 2A to FIG. 2C An example workflow diagram is presented in accordance with one or more implementations of the disclosed subject matter.

[0025] Figure 3 A block diagram of another example algorithm orchestration system is shown in accordance with one or more embodiments of the disclosed subject matter.

[0026] Figure 4 A high-level flow chart of an example computer-implemented method for executing workflow instructions for analyzing medical images and associated metadata in accordance with one or more embodiments of the disclosed subject matter is presented.

[0027] FIG. 5A to FIG. 5B A block diagram is provided of example algorithm orchestration components in accordance with one or more implementations of the disclosed subject matter.

[0028] Figure 6 An example administrative login page for accessing an algorithmic orchestration management application is presented in accordance with one or more embodiments of the disclosed subject matter.

[0029] Figure 7 An example algorithm management user interface (UI) is presented in accordance with one or more implementations of the disclosed subject matter.

[0030] Figure 8 An example algorithm management editor UI is presented in accordance with one or more implementations of the disclosed subject matter.

[0031] FIG. 9A to FIG. 9B An example algorithm loading UI is presented in accordance with one or more implementations of the disclosed subject matter.

[0032] FIG. 10A to FIG. 10B An example workflow management UI is presented in accordance with one or more implementations of the disclosed subject matter.

[0033] FIG. 11A to FIG. 11X An example UI of a workflow creation application associated with creating a workflow using the workflow creation application in accordance with one or more embodiments of the disclosed subject matter is presented.

[0034] FIG. 12A to FIG. 12B A workflow simulation function of an algorithm orchestration management application according to one or more embodiments of the disclosed subject matter is illustrated.

[0035] FIG. 12C to FIG. 12D Some additional example workflow diagrams are presented in accordance with one or more implementations of the disclosed subject matter.

[0036] FIG. 13A to FIG. 13D An example task registry UI is presented in accordance with one or more implementations of the disclosed subject matter.

[0037] FIG. 14A to FIG. 14B An example device management UI is presented in accordance with one or more implementations of the disclosed subject matter.

[0038] FIG. 15A to FIG. 15B An example activity log UI is presented in accordance with one or more implementations of the disclosed subject matter.

[0039] FIG. 16A to FIG. 16B An example activity dashboard UI is presented in accordance with one or more embodiments of the disclosed subject matter.

[0040] Fig.17 An example algorithm orchestration system architecture is presented in accordance with one or more implementations of the disclosed subject matter.

[0041] FIG. 18A to FIG. 18B

[0013] Another example algorithm orchestration system is presented in accordance with one or more embodiments of the disclosed subject matter.

[0042] Fig.19 A block diagram illustrating workflow procedures and computational stages for analyzing medical images and metadata according to one or more embodiments of the disclosed subject matter.

[0043] Fig. 20 is an example of a workflow program as executed by an algorithmic orchestration component and the interaction of the algorithmic orchestration component with other components of a workflow according to one or more embodiments of the disclosed subject matter.

[0044] Fig.21 is an example of an algorithm management program as executed by an algorithm management component and the interaction of the algorithm management component with other components of the program in accordance with one or more embodiments of the disclosed subject matter.

[0045] Fig. 22An example of a workflow implementation for analyzing a chest X-ray for a pneumothorax (PTX) condition in accordance with one or more embodiments of the disclosed subject matter is shown.

[0046] Fig.23 A high-level flow chart of another example computer-implemented method for executing workflow instructions to analyze medical images and associated metadata in accordance with one or more embodiments of the disclosed subject matter is presented.

[0047] Fig.24 A high-level flow chart of an example computer-implemented method for workflow algorithm management in accordance with one or more embodiments of the disclosed subject matter is presented.

[0048] Fig.25 A high-level flow chart of an example computer-implemented method for orchestrating an algorithm for executing paired medical images is presented in accordance with one or more embodiments of the disclosed subject matter.

[0049] Fig.26 A high-level flow chart of an example computer-implemented method for orchestrating medical image execution algorithms according to a predefined workflow and managing workflow and model execution in accordance with one or more embodiments of the disclosed subject matter is presented.

[0050] Fig. 27 Block diagram illustrating an example non-limiting operating environment in which one or more examples described herein may be facilitated.

[0051] Fig.28 Block diagram illustrating another example non-limiting operating environment in which one or more examples described herein may be facilitated.

[0052] Fig.29 Block diagram illustrating another example non-limiting operating environment in which one or more examples described herein may be facilitated.

[0053] Fig.30 Block diagram illustrating another example non-limiting operating environment in which one or more examples described herein may be facilitated.

[0054] Fig.31 Block diagram illustrating another example non-limiting operating environment in which one or more examples described herein may be facilitated.

[0055] Fig.32 Block diagram illustrating another example non-limiting operating environment in which one or more examples described herein may be facilitated. DETAILED DESCRIPTION

[0056] The following detailed description is merely exemplary and is not intended to limit the examples shown and described herein and / or the application or use of these examples. In addition, it is not intended to be bound by any explicit or implicit information presented in the aforementioned "Summary of the Invention" section or "Detailed Description of the Invention" section.

[0057] The disclosed subject matter relates to algorithm orchestration in the field of medical imaging, including techniques for orchestrating algorithms for executing medical images according to predefined workflows and techniques for managing workflow and model execution. The disclosed algorithm orchestration technology employs a centralized network-accessible algorithm orchestration component that acts as an intermediate layer that connects medical image providers with AI model processing web services for executing AI models on their medical images and returning consumable results (outcome / result). In this regard, the algorithm orchestration component may receive a medical image (image) (or images) from a medical imaging provider in association with a request to process the medical image using one or more image processing algorithms. Given a medical image, the algorithm orchestration component may identify one or more workflows and execute one or more workflows on the medical image, the one or more workflows involving processing the medical image using one or more medical image processing algorithms compatible with the image. The algorithm orchestration component also provides workflow results to the requesting entity.

[0058] In this context, a workflow refers to a set of workflow instructions that define which computer-implemented processing steps are to be applied to a medical image (or images). A workflow consists of an orchestrated and repeatable pattern of service calls to process a medical image, execute algorithms, and produce results to be consumed by other systems. In various embodiments, algorithms may include, but are not limited to, image restoration algorithms (e.g., to improve the quality of an image), image analysis algorithms (e.g., classification / diagnosis models, organ segmentation models, etc.), image synthesis algorithms (e.g., to construct a three-dimensional image based on multiple two-dimensional images), image enhancement algorithms (e.g., to improve an image by using filters or adding information that will assist in visualization), and image compression algorithms (e.g., to reduce the size of an image to enhance the required transmission time and storage).

[0059] In various embodiments, the workflow can be configured to apply the algorithm in different clinical contexts and filter the images / studies based on metadata associated with the images / studies (e.g., modality, series, study description, etc.). The workflow can also be configured to call a Representational State Transfer (REST) ​​service to retrieve information from other systems. The workflow can also be configured to execute the algorithm, including executing the algorithm and other tasks in parallel. The workflow can also be configured to execute asynchronous tasks and wait for the task to complete. The workflow can also be configured to evaluate the output of a task and use the output as the input of another task.

[0060] In some embodiments, the workflow may include at least one algorithm that can be used to assist radiologists in diagnosing and treating diseases. For example, the workflow may call one or more algorithms including AI algorithms applicable to medical images to determine the probability associated with a given medical condition of a patient from whom the image was obtained. However, the concept of an algorithm as used herein is not limited to a machine learning (ML) model or a deep learning (DL) model. For example, in addition to a medical image inference model, the workflow described herein may integrate various algorithms for filtering input images based on associated metadata (e.g., based on patient-related demographics, based on image attributes, and other factors), heuristic algorithms, algorithms for controlling the order and timing of service calls throughout the workflow, and other types of algorithms.

[0061] In various embodiments, the algorithm orchestration component may employ an algorithm catalog that identifies algorithms included in predefined workflows that can be used to analyze a given medical. The catalog of algorithms may include various internal and third-party algorithms / models that can be integrated into the workflow. In some embodiments, the algorithm / model can be integrated into the workflow as an HTTP task and retrieved / called as a web service call through one or more networks (e.g., the Internet, etc.). In this regard, the algorithm orchestration component can connect the healthcare provider system with algorithms created by various AI model providers / developers, wherein the algorithm orchestration component can use a defined application program interface (API) for the algorithm to orchestrate access and apply the algorithm to medical images as a web service. Additionally or alternatively, the algorithm / model can be integrated into the workflow as a "job". Specifically, the algorithm / model and other tasks defined by computer executable instructions can be wrapped in a KubernetesJob function, and the algorithm orchestration component can asynchronously execute the function inside the cluster on behalf of the user. This feature opens up a variety of possibilities that are particularly relevant to legacy systems, where client services are not under HTTP and are simply command lines and / or executables.

[0062] In some embodiments, an image provider may identify or indicate a particular algorithm and / or workflow to be applied to a medical image in association with providing an image processing request. Additionally or alternatively, an algorithm orchestration component may determine which workflows and associated algorithms to apply to a given image based on metadata associated with the image and information included in an algorithm catalog. Using these embodiments, the algorithm orchestration component may automatically select at least one workflow to be performed on the medical image.

[0063] The algorithm orchestration component also directs the workflow execution engine to execute the workflow and apply one or more algorithms included in the workflow to the image. In various embodiments, the workflow execution procedure involves employing the algorithm execution engine to execute one or more AI algorithms included in the workflow by calling / retrieving the algorithm at its network accessible file source (e.g., using its corresponding API call as defined by the workflow code / logic). Based on various determinations made in response to the execution of the workflow, the algorithm orchestration component can generate notifications (e.g., text, email, report) and send the notifications to the image provider, radiologist, or another related healthcare system to assist them in their reporting of patient results and diagnoses.

[0064] According to various exemplary embodiments, the algorithm orchestration component may be configured to interface with a medical image storage system employed by a medical image service provider, such as a picture archiving and communication system (PACS) and / or a vendor neutral archive (VNA). According to these exemplary embodiments, when a new medical image / image study is received at the medical image storage system, the medical image provider / storage system may be configured to send a Hypertext Transfer Protocol (HTTP) request to an API exposed by an API gateway for the algorithm orchestration component. For example, the request may be identified as a "study process notification" and may contain study metadata and a payload. The gateway may be configured to forward the request to an algorithm orchestration director, which validates the request payload and responds with a request execution identifier (ID) and status. The algorithm orchestration component may then direct the workflow execution engine to call all (or one or more) applicable workflows for the image / imaging study. In implementations where two or more workflows are applicable to the image / image study, the workflow execution engine may execute each workflow as a separate thread.

[0065] A typical workflow will begin by validating the medical image metadata to determine if the image meets the defined workflow requirements (e.g., regarding modality, view position, study description, etc.). According to these workflows, if the workflow execution engine determines that the image / image study meets the workflow requirements, then the algorithm orchestration component and / or the workflow execution engine transfers the image / study data from the image source (e.g., PACS, VNA, etc.) to the local file storage device employed by the algorithm orchestration component. When the transfer is complete, the workflow execution engine executes all algorithms defined in the workflow, thereby employing the algorithm execution engine to execute any algorithm of the workflow defined in the algorithm catalog. In this regard, for each algorithm included in the workflow, the workflow execution engine calls / invokes the algorithm in the algorithm catalog and waits for the algorithm execution engine to return a response. Once the responses of all algorithms included in the workflow have been received, the algorithm orchestration component transfers the output files generated by the algorithm back to the image / image study source system (e.g., PACS, VNA, etc.) and sends a notification message to the imaging provider, thereby notifying the provider that the processing of the image / study is complete. This notification message may also include information identifying the specific algorithm executed and the results of each algorithm.

[0066] In this regard, the algorithm orchestration component may act as a bridge between client input metadata and an AI model suitable for analyzing an image to provide a request message and / or a customized message in a request format (e.g., email, text message). The algorithm orchestration component may be connected to an AI model for diagnosing a disease condition (e.g., PTX, stroke, MR) and receive a confidence score from the AI ​​model indicating a probability that can be delivered on an end-user solution (e.g., via an active dashboard interface). The algorithm orchestration component may be integrated with substantially any user platform that requires an AI model solution to facilitate its worklist priorities. For example, in the case of an emergency medical condition, the algorithm orchestration component responsive to workflow instructions facilitates the validity of magnetic resonance / computed tomography (MR / CT) images and the review of the images by an AI algorithm, and generates a priority list to expedite reporting to the corresponding healthcare provider. The algorithm orchestration component may be integrated as an intermediate layer between a medical image provider and an AI model provider to populate a radiologist / healthcare provider terminal system with an overview message, findings, and / or importance in addition to providing a secondary capture supported by annotated overlays of the corresponding medical image. Thus, in one example, algorithmic orchestration enables rapid generation of workflow instructions that, when executed, allow a healthcare user to customize diagnostic reporting, information flow, and notification of downstream reporting to other members of the healthcare team.

[0067] Various application services are supported by the systems and methods described herein. In one example application service example, workflow processing is provided by an algorithm orchestration component that executes a given workflow with workflow instructions that define the type of analysis / diagnosis processing to be applied to a given medical image and associated metadata. The algorithm orchestration component operates with an algorithm catalog that defines the available algorithms for a given image and an algorithm execution engine to execute the selected algorithm of the algorithm orchestration component in response to the workflow instructions.

[0068] In another application service example, an algorithm management interface and tools may be provided. This includes procedures for viewing existing algorithms, loading algorithms, editing algorithms, searching algorithms, and deleting algorithms. These algorithms are represented by workflows and are selectively associated with a given type of image (e.g., chest, brain, limbs, etc.). An algorithm management interface may be provided that enables setting of diagnostic thresholds that define whether a given medical condition (e.g., a pneumothorax condition (PTX)) is associated with a probability determined by a selected model algorithm during workflow processing.

[0069] In another example application service example, a workflow configuration tool and user interface (UI) may be provided to facilitate workflow development and creation. This tool may be implemented using features that an administrator seeking to quickly satisfy a radiologist's request for a certain type of analysis / diagnosis output from the system will be interested in, without having to redesign existing software or commission new software for the requested task. The tool allows creation of workflows consisting of workflow instructions that define the associations and relationships that the administrator (the one setting up the workflow) wants from when an image is received to when / how notifications / reports to the radiologist occur (various checks of image data and metadata).

[0070] In yet another application service example, an active dashboard interface provides the results from the AI ​​model analysis relative to the actual radiology analysis of the same or similar images. This includes displaying the number of available studies used to generate the AI ​​model analysis and the false positive rate and true positive rate analysis. The active dashboard provides a measure of the radiologist's confidence in the potential automated analysis of the AI ​​model. Compliance with the desired diagnostic criteria can be monitored in the active dashboard, where the actual diagnosis of one or more radiologists is compared to the AI ​​model output.

[0071] In yet another application service example, simulation and testing tools may be provided for workflow instructions. A simulation tool operated using a workflow configuration tool may be provided, where a workflow may be run in an offline setting using test images or newly acquired images prior to implementation in a live system operated by an algorithmic orchestration component. In yet another application service, security (e.g., network or file security) may be managed between components that interact with the algorithmic orchestration component. This includes how the components of the system communicate and what type of security credentials are passed for the respective communications to be made.

[0072] Unless the context warrants a specific distinction between the terms, the terms "algorithm" and "model" are used interchangeably herein. The term "image inference model" is used herein to refer to an AI / ML algorithm configured to perform image processing or analysis tasks on an image. The image processing or analysis tasks may vary. In various embodiments, the image processing tasks may include (but are not limited to): image restoration tasks (e.g., to improve the quality of an image), image synthesis tasks (e.g., to construct a three-dimensional image based on multiple two-dimensional images), image enhancement tasks (e.g., to improve an image by using a filter or adding information that will assist visualization), and image compression tasks (e.g., to reduce the size of an image to enhance the required transmission time and storage). Some example image analysis tasks may include, but are not limited to, classification tasks (e.g., diagnosis), segmentation tasks (e.g., organ segmentation, region of interest segmentation, etc.), object recognition tasks, motion detection tasks, video tracking tasks, optical flow tasks, etc. The image inference model described herein may include a 2D image processing model as well as a 3D image processing model. The image processing model may employ various types of AI / ML algorithms, including (but not limited to): deep learning models, neural network models, deep neural network models (DNN), convolutional neural network models (CNN), generative adversarial neural network models (GAN), etc. However, the concept of an algorithm or model as described herein is not limited to machine learning or deep learning models. Unless the context warrants a specific distinction between the terms, the terms "image inference algorithm", "image inference model", "image processing algorithm", "image processing model", "image analysis model", etc. may be used interchangeably herein.

[0073] The term "image-based inference output" is used herein to refer to a determination or prediction that an image processing model is configured to produce. For example, an image-based inference output may include a segmentation mask, a reconstructed image, an adapted image, an annotated image, a classification, a value, etc. The image-based inference output will vary based on the type of model and the specific task that the model is configured to perform. The image-based inference output may include a data object that can be presented (e.g., a visual data object), stored, used as an input to another processing task, etc. Unless the context warrants a specific distinction between the terms, the terms "image-based inference output", "inference output", "inference result", "inference", "output", "result", "prediction", etc. are used interchangeably herein.

[0074] As used herein, "medical imaging inference model" refers to an image inference model that is customized to perform image processing / analysis tasks on one or more medical images. For example, medical imaging processing / analysis tasks may include (but are not limited to): disease / condition classification, disease region segmentation, organ segmentation, disease quantification, disease / condition staging, risk prediction, temporal analysis, abnormality detection, anatomical feature characterization, medical image reconstruction, etc. Unless the context warrants a specific distinction between the terms, the terms "medical image inference algorithm", "medical image inference model", "medical image processing algorithm", "medical image processing model", "medical image analysis model", etc. are used interchangeably herein. The output, result, etc. of the medical image inference model is an artifact produced by an algorithm executed using one or more medical images as input. The output may be in different formats, such as: Digital Imaging and Communication in Medicine (DICOM) Structured Report (SR), DICOM Secondary Capture, DICOM Parameter Mapping, Image, Text, and / or JavaScript Object Specification (JSON).

[0075] The types of medical images processed / analyzed by the medical image inference model described herein may include images captured using various types of image capture modalities. For example, medical images may include (but are not limited to): radiation therapy (RT) images, X-ray (XR) images, digital radiography (DX) X-ray images, X-ray angiography (XA) images, panoramic X-ray (PX) images, computed tomography (CT) images, mammography (MG) images (including tomosynthesis equipment), magnetic resonance imaging (MRI) images, ultrasound (US) images, color flow Doppler (CD) images, positron emission tomography (PET) images, single photon emission computed tomography (SPECT) images, nuclear medicine (NM) images, etc. Medical images may also include synthesized versions of native medical images, such as synthesized X-ray (SXR) images, modified or enhanced versions of native medical images, extended versions of native medical images, and similar versions generated using one or more image processing techniques. The medical imaging processing model disclosed herein may also be configured to process 3D images.

[0076] As used herein, "capture modality" refers to a specific technical mode of capturing images or image data using one or more machines or devices. In this regard, as applied to medical imaging, different capture modalities may include, but are not limited to: 2D capture modality, 3D capture modality, RT capture modality, XR capture modality, DX capture modality, XA capture modality, PX capture modality, CT, MG capture modality, MRI capture modality, US capture modality, CD capture modality, PET capture modality, SPECT capture modality, NM capture modality, etc.

[0077] As used herein, the term "web platform" refers to any platform that enables the delivery of content and services over a network (i.e., web pages / the Internet) using a network transfer protocol such as HTTP, sFTP, or another network transfer protocol. By way of example, a web platform may include, but is not limited to, web applications (i.e., interactive websites), mobile websites, mobile applications, and the like. The terms "web-based platform," "network platform," "platform," and the like may be used interchangeably herein unless the context warrants a particular distinction between the terms.

[0078] One or more embodiments are now described with reference to the accompanying drawings, wherein the same reference numerals are used to represent the same elements throughout. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a more thorough understanding of one or more embodiments. However, it is apparent that in various cases, one or more embodiments may be practiced without these specific details.

[0079] Figure 1A block diagram of an example algorithm orchestration system 100 is shown in accordance with one or more embodiments of the disclosed subject matter. Embodiments of the system described herein may include one or more machine-executable components embodied within one or more machines (e.g., embodied in one or more computer-readable storage media associated with one or more machines). Such components, when executed by one or more machines (e.g., processors, computers, computing devices, virtual machines, etc.), may cause one or more machines to perform the operations described.

[0080] The system 100 includes an algorithm orchestration component 110 that facilitates the orchestration of various services for medical image providers and administrators related to processing medical images using various image processing algorithms as web services and / or package jobs. A medical image provider may include substantially any entity that provides medical images for processing by the algorithm orchestration component 110. For example, a medical image provider may include a healthcare system, a hospital system, a medical imaging system, an individual clinician / radiologist (e.g., a sole practitioner), etc. In the system 100, a medical image provider is represented by an image provider system / device 102. In this regard, a medical image provider accesses and employs services orchestrated by the algorithm or orchestration component 110 via one or more systems or devices represented in the system as image provider systems / devices 102.

[0081] The image provider system / device 102 may include substantially any system or device that provides medical images to the algorithm orchestration component 110. For example, in various embodiments, the image provider system / device 102 may be, include, or be communicatively coupled to one or more data sources that store medical images and / or information related to medical images, such as metadata describing various attributes of the medical images, radiology reports associated with the medical images, related patient information, and the like. For example, attributes may refer to patient data associated with the medical images, such as name, age, previous medical history data, and / or other data related to the patient and the medical image. Attributes may also describe physical examination series and image level metadata to identify and / or perform related workflows. For example, metadata associated with medical images may include information about (but not limited to) image acquisition protocols, image quality, signal noise, image index size, matrix size, resolution, orientation, capture modality, type and size of anatomical features depicted in the image data, image rating, image quality, image storage location (or locations), image capture time, and other features. Metadata tags may also include, but are not limited to, information describing the image type, the anatomical part depicted in the image data, the medical condition / pathology reflected in the image data, patient demographics, and relevant patient medical history factors.

[0082] These image provider systems / devices 102 may include different types of data sources, provided by the same entity / organization that owns / operates the algorithm orchestration component 110, as well as provided by various third parties or external entities / organizations. For example, the image provider system / device 102 may include or be communicatively coupled to one or more internal medical image databases, one or more external medical image databases, one or more internal and external workstations, one or more internal and external PACS and consoles, etc. The image provider systems / devices 102 may be located on the same network and across multiple networks and regions of the world. The number of image provider systems / devices 102 is not limited.

[0083] In this regard, the term "internal" as applied to the image provider system / device 102 is used herein to refer to proprietary data sources / systems associated with a single enterprise that owns, provides, manages, and / or controls the features and functions of the algorithm orchestration component 110. For example, in some implementations, the single enterprise may be a health information technology (HIT) company (such as General Electric (GE) Healthcare Corporation) that provides a range of products and services, including medical imaging and information technology, electronic medical records, medical diagnostics, patient monitoring systems, drug discovery, biopharmaceutical manufacturing technology, etc. According to this example, the HIT company may also provide, manage, and / or control the algorithm orchestration component 110 and employ the algorithm orchestration component 110 to process its own medical images using various image processing algorithms. In another implementation, the single enterprise may include a hospital organization that provides medical services to patients via one or more hospitals, outpatient medical facilities, etc. Regardless of the nature, operations, and / or services provided by the enterprise, the image provider system / device 102 may include internal data sources / systems owned and / or operated by the enterprise that owns / manages the algorithm orchestration component 110.

[0084] The term “external” as applied to a medical image provider is used herein to refer to systems, devices, and / or sources of medical image data that are owned and / or operated by a third party entity that does not provide, manage, and / or control the algorithm orchestration component 110 .

[0085] The medical images provided by the image provider system / device 102 may include images captured using substantially any medical imaging modality. For example, the medical images may include, but are not limited to: conventional X-ray (XR) images, digital radiography (DX) X-ray images, X-ray angiography (XA) equipment, panoramic X-ray (PX) images, computed tomography (CT) images, mammography (MG) images (including tomosynthesis equipment), magnetic resonance imaging (MRI) images, ultrasound (US) images, color flow Doppler (CD) images, positron emission tomography (PET) images, single photon emission computed tomography (SPECT) images, nuclear medicine (NM) images, and other types of medical images. The medical images may also include or be associated with (e.g., as metadata): raw pixel data, waveform data, three-dimensional or depth data, point cloud data, etc.

[0086] In various embodiments, the image provider system / device 102 may provide medical images and associated metadata formatted according to the DICOM standard. DICOM is a global standard for processing, storing, printing, and transmitting medical imaging information. It includes file format definitions and network communication protocols. DICOM is used worldwide to store, exchange, and transmit medical images. For example, DICOM combines standards for various imaging modalities, such as radiography, ultrasonography, computed tomography, magnetic resonance, mammography, etc. DICOM includes protocols for image exchange (e.g., via portable media such as DVD), image compression, 3D visualization, image presentation, and result reporting. Due to the widespread adoption of DICOM, it may be easier to use imaging data sets of the size and complexity required to develop complex AI imaging diagnostic models than to use other types of data sets. DICOM is a medical imaging standard that has been in existence for more than two decades. For imaging, DICOM plays the role of connecting all modalities in a hospital. Printers, displays, MRI machines, and other acquisition devices all communicate using the DICOM standard.

[0087] In some implementations, the image provider system / device 102 may be or include one or more PACS. PACS provides economical storage and convenient access to images from multiple modalities (source machine types). Electronic images and reports can be transmitted digitally via PACS without the need for manual archiving, retrieval, or transportation of film folders, i.e., folders used to store and protect X-ray films. The common format for PACS image storage and transmission is DICOM. Once encapsulated in DICOM, non-image data such as scanned documents can be merged using consumer industry standard formats such as PDF (Portable Document Format).

[0088] Additionally or alternatively, the image provider system / device 102 may be or include one or more vendor neutral archive (VNA) systems. VNA is a medical imaging technology in which images and documents (and possibly any clinically relevant files) are stored (archived) in a standard format with a standard interface so that other systems can access them in a vendor neutral manner. In this regard, VNA can collect and store images from multiple modalities and departments (e.g., radiology, cardiology, orthopedics, etc.) and aggregate all content into one large archive. In addition to DICOM images, VNA can also store data objects that are not directly related to images or DICOM data, such as manually generated requests and reports, health level 23 clinical document architecture (HL23-CDA) documents, etc. VNA can also use non-DICOM access protocols, such as cross-enterprise document sharing (XDS and XDS-I). VNA can also provide cross-domain identity and code resolution (patient ID, accession number, procedure code), dynamic DICOM tag deformation, and other adaptive features. The image provider system / device 102 may also be or include a workstation (e.g., deployed at one or more computing devices) where medical images are viewed and / or processed for various purposes (e.g., an AI model development workstation, a radiology viewer workstation, an annotation workstation, etc.).

[0089] The algorithm orchestration component 110 may receive an image processing request 106 from a medical image provider (e.g., via its corresponding image provider system / device 102), the image processing request corresponding to a request to apply one or more image processing algorithms to a medical image or a group of medical images (such as a group of medical images included in a particular imaging study). These image processing algorithms may include various medical image inference algorithms or models (e.g., AI models). For example, image processing algorithms may include, but are not limited to, image restoration algorithms (e.g., to improve the quality of an image), image analysis algorithms (e.g., classification / diagnosis models, organ segmentation models, etc.), image synthesis algorithms (e.g., to construct a three-dimensional image based on multiple two-dimensional images), image enhancement algorithms (e.g., to improve an image by using filters, or adding information that will assist in visualization), and image compression algorithms (e.g., to reduce the size of an image to enhance the required transmission time and storage).

[0090] In the illustrated embodiment, the image processing algorithm is stored at one or more network accessible locations, which are accessed and applied to medical images according to a predefined workflow (e.g., by a workflow execution component and / or an algorithm execution engine 116 and / or an algorithm execution engine 118). In various embodiments, the image processing algorithm may be integrated into the predefined workflow as an HTTP task and accessed and applied to the medical image as a web service according to the predefined workflow. Additionally or alternatively, the algorithm / model may be integrated into the workflow as a "job". Specifically, the algorithm / model and other tasks defined by computer executable instructions may be wrapped in a Kubernetes Job function, and the algorithm orchestration component may execute the function asynchronously within the cluster on behalf of the user. This feature opens up a variety of possibilities that are particularly relevant to legacy systems, where client services are not under HTTP and are simply command lines and / or executables.

[0091] In the illustrated embodiment, the system 100 includes one or more file sharing data sources 132, which correspond to network-accessible data sources storing image processing algorithms. In addition or alternatively, one or more of the algorithms may be stored and accessed in one or more AO databases 120. The image processing algorithms may include internal algorithms 134 and third-party algorithms 136. In this context, the internal algorithms 134 may include various image / medical image processing algorithms / models provided by the same entity that owns / manages the algorithm orchestration component 110, and the third-party algorithms 136 may include various image / medical image processing algorithms / models provided by various third-party AI developers / providers. In this regard, the algorithm orchestration system 100 may enable a variety of different medical image providers to access and adopt the algorithm orchestration component 110 to process their medical images using various medical image inference algorithms or models provided by various AI model developers / providers.

[0092] The algorithm orchestration component 110 may include an orchestration director component 112 to orchestrate the fulfillment of the image processing request 106 and return request results and status notifications 108 to the medical image provider. Specifically, the orchestration director component 112 may coordinate the interaction between the image provider system / device 102 and various components of the algorithm orchestration system 100 (and other systems described herein) that retrieve and store medical image data to be processed using algorithms, execute workflows and call algorithms associated with them, and return request results and status notifications 108. As described in more detail below, these components may include (but are not limited to) a workflow execution engine 116, an algorithm execution engine 118, one or more file sharing data sources 134, one or more algorithm orchestration databases 120, and various additional software and hardware components.

[0093] In this regard, the algorithm orchestration component 110 acts as an intermediary layer connecting medical image providers with medical image processing services for executing AI algorithms on their medical images and returning consumable results / results. From the perspective of the imaging provider, the interaction with the algorithm orchestration component 110 may involve sending imaging processing requests 106 when new medical studies are available for processing, including pushing medical image studies to the algorithm orchestration component 110 for processing. The imaging provider may also receive notifications from the algorithm orchestration component 110 about workflows and / or algorithms applicable to processing imaging studies, notifications about the completion and results of study processing. In some implementations, when one or more workflows are matched, the image provider system / device 102 may initiate the transfer of imaging studies to the algorithm orchestration component 110. Other interactions between the imaging provider and the algorithm orchestration component 110 may include (but are not limited to) initiating the transfer of algorithm results (if available) once all algorithms have been executed, canceling previously submitted image processing requests, and sending requests with different priority levels.

[0094] In various embodiments, the algorithm orchestration component 110 provides the image provider system / device 102 with access to internal algorithms 134 and third-party algorithms 136 as web services and / or jobs that can be executed simultaneously according to predefined workflows for multiple providers and multiple images. In this regard, the algorithm orchestration component 110 supports different workflows based on the same set of services arranged in different combinations.

[0095] In this context, a workflow refers to a set of computer executable workflow instructions that define which computer-executed processing steps will be applied to a medical image (or images). A workflow consists of an orchestrated and repeatable service call pattern that processes a medical image, executes an algorithm, and produces results to be consumed by other systems. In various embodiments, a workflow consists of nodes corresponding to different computer executable instructions. These nodes may include, but are not limited to, start and end nodes, decisions, task nodes, sub-workflow nodes, wait nodes, fork nodes, combination nodes, and function nodes. The start and end nodes define where the workflow begins and ends. The decision node corresponds to a decision-making task and is used to evaluate an expression that will define the next step to be performed (similar to a switch case instruction in a programming language). For example, a decision node can be used to filter input images based on one or more defined criteria to ensure that the workflow is performed on an authorized image. For example, if the input image does not meet one or more defined criteria (e.g., based on various parameters), many workflows can stop executing the decision node of the workflow on the input image.

[0096] Task nodes represent tasks to be performed by the workflow. These tasks may include HTTP tasks, lambda tasks, Kubernetes jobs (e.g., commonly referred to as k8s jobs), etc. In various embodiments, task nodes correspond to algorithms / models to be called and executed (such as calls to internal algorithms 134 and / or third-party algorithms 136). Task nodes can also be used to call other HTTP services, lambda tasks, and / or jobs, such as services including: retrieving images or other data for processing (e.g., from PACS, etc.), moving data from one source to another, deleting data, etc. In various embodiments, the workflow instructions of the task nodes corresponding to the algorithms / models to be called / executed may include (e.g., as embedded therein) algorithm execution instructions for calling and executing the corresponding algorithms, such as network accessible algorithm file locations, APIs for accessing and running algorithms and receiving results, etc. In this regard, the task nodes corresponding to the algorithms / models may substantially include links to the actual algorithms / models, wherein executing the task nodes involves using the links to access and run the models on the input images, and receiving the results.

[0097] A sub-workflow node corresponds to a sub-workflow that may be related to any of the nodes for a workflow. A wait node adds a delay in a workflow based on one or more defined conditions. For example, a wait node may be used to limit the initiation of a task until another task or sub-workflow is completed. Therefore, a wait node may be used to track the execution of asynchronous tasks as part of an orchestration, and is typically used for time-consuming operations such as moving an imaging study from a PACS to an algorithm orchestration component (or a local data storage device used by the algorithm orchestration component 110), pushing algorithm results to a PACS, or executing a deep learning model. A fork node may be used to fork a workflow execution into two or more parallel programs. A combine node may be used to combine two or more parallel programs initiated by a fork node, and may wait for two or more parallel programs to complete. A function node may be used to evaluate an expression and provide a response. For example, a function node may be used to perform a data transformation, run a calculation, or another function that may be expressed as a script that may be run by the workflow execution engine 116. A workflow may aggregate the results of different algorithms executed, and notify other systems about the status of the orchestration.

[0098] FIG. 2A to FIG. 2C An example workflow diagram is presented in accordance with one or more implementations of the disclosed subject matter. FIG. 2A to FIG. 2C The diagram shown in exemplifies some of the different nodes described above as being configured to execute defined workflow instructions. FIG. 2A to FIG. 2C The workflow diagram presented in corresponds to an example workflow that may be included in the workflow and task registry data 124 and executed by the workflow execution engine 116. Each of the different nodes is represented by a different symbol shown in the node legend. Figure 2AAn example workflow diagram 201 for executing a single algorithm is shown, which in this example is a pneumothorax (PTX) classification model. The workflow begins at a start node 202, and three decision nodes are used to evaluate the input image before calling / applying the PTX model. In this regard, at 204, the first decision node is used to evaluate whether the modality of the input image is CR, DX. If the modality is not CR, DX, the workflow ends at 214. At 208, a second decision node is used to evaluate whether the viewing position is AP, PA. If the viewing position is not AP, PA, the workflow ends. Subsequently, at 208, a third decision node is used to evaluate whether the study description is classified as a chest study. If so, the workflow continues, and at 212, the PTX model is called and applied to the input image. If not, the workflow ends at 214.

[0099] Figure 2B An example workflow diagram 211 involving parallel algorithm execution is shown. The workflow diagram 211 starts at 216 and proceeds through two decision nodes (node ​​218 and node 220). If the input image satisfies both decision node criteria, then the workflow continues. At 222, a fork node is used to split the workflow into two parallel programs. At 224, the PTX model is called and applied to the input image. At the same time, at 226, the endotracheal (ET) tube model is called and applied to the input image. At 228, a combine node is used to wait for and combine the results of the two models, and then the workflow ends at 230.

[0100] Figure 2C An example workflow diagram 221 involving sequential algorithm execution is shown. According to the workflow diagram 221, the workflow starts at 232, followed by a modality evaluation decision node 234. If the input image satisfies the modality criteria, then at 236, the position model is called and applied to the input image. After receiving the result, at 238, the PTX model is called and applied to the input image. Subsequently, the workflow ends at 240. In this example, the PTX model is executed after the position model has been executed, where the position model is connected in series with the PTX model.

[0101] Reference again Figure 1In various embodiments, predefined workflows that can be used to process medical images may be provided in one or more AO databases accessible to the workflow execution engine 116. For example, in the illustrated embodiment, one or more AO databases may include workflow and registry task data 122. In some implementations, the workflow and task registry data 124 may include / store a separate data file for each defined workflow, wherein the data file includes computer executable workflow instructions (e.g., code / logic) for executing the workflow. Additionally or alternatively, the workflow files may be stored at one or more other accessible databases (e.g., one or more of the AO databases 120), and the workflow and task registry data 124 may be or correspond to an index that includes information identifying and / or describing the workflow and provides a file location of the corresponding workflow, where the workflow can be accessed by the workflow execution engine 116.

[0102] Regardless of the location where the workflow files are stored, each workflow can be identified by a unique title, name, or workflow identifier (ID). The corresponding workflow identified and / or included in the workflow and task registry data 124 may also include, or be associated with, information that describes the workflow (e.g., an overview of the workflow) and provides information that identifies or indicates relevant information about the medical images to which the workflow is applicable. For example, in various embodiments, a workflow may be associated with metadata tags that identify the properties of the medical images that can be processed by the workflow. In some embodiments, each workflow identified or included in the workflow and task registry data 124 may also include or be associated with information that identifies the creator (or creators) of the workflow, one or more tasks / algorithms included therein, the timing / date when the workflow was created and / or last updated, and various other relevant information about the workflow. As described below with reference to Figure 5A and FIG. 11A to FIG. 11X Describing in more detail, the algorithm orchestration component 110 may provide a workflow creation tool that provides for creating and editing workflows included in the workflow and task registry data 122 .

[0103] The one or more AO databases 120 may also include algorithm catalog data 122, which identifies various internal algorithms 134 and third-party algorithms 136 included in the defined workflow and / or algorithms that are otherwise applicable to medical images. In this regard, in some embodiments, the algorithm catalog data 124 includes or corresponds to an algorithm index, which includes information identifying algorithms stored at one or more file sharing data sources 132, which are loaded into the system 100 and included in the defined workflow and / or are otherwise applicable to medical images. For example, in various embodiments, each of the algorithms may include a unique identifier, a title, a name, etc. The algorithm catalog data 124 may also include algorithm execution instructions, which define how to access and execute the corresponding algorithm and receive results / outputs. For example, in various embodiments, the algorithm may be executed as a web service / .HTTP task, and the algorithm execution instructions may define code / logic for calling and running the algorithm as a web service, such as a network accessible algorithm file location, an API for accessing and running the algorithm and receiving results, etc. Additionally or alternatively, the algorithms / models may be integrated into workflows as a lambda task, Kubernetes job, or similar type of HTTP alternative execution method. This feature opens up a variety of possibilities that are particularly relevant to legacy systems where the client service is not under HTTP and is simply a simple command line and / or executable. The algorithm catalog data 124 may also include additional details for each algorithm, including, but not limited to: provider (e.g., internal or third party), version, description or overview, creator, creation / publication date / time, attributes of the input image to which the algorithm is applicable, input data format, and output data format. In some embodiments, the corresponding algorithms included in the algorithm catalog data 124 may also be associated with information describing one or more parameters of the algorithm. For example, parameters may include threshold parameters (e.g., criticality, diagnostic threshold, probability threshold, etc.), clinical relevance parameters / settings, input image parameter priority settings, etc. As described below with reference Figure 5A and Figure 8 As described in more detail in FIGS. A through 9 , the algorithm orchestration component 110 may provide a management tool that allows an administrator to define or adjust one or more parameters of an algorithm that is loaded into the system and may be used to process medical images.

[0104] The one or more AO databases 120 may also include system / device registry data 126 that identifies the image provider system / device 102 that the image provider system / device authorization algorithm orchestration component 110 communicates with 102 in association with receiving and responding to the image processing request 106. The one or more AO databases 120 may also include an orchestration scheme 128 and a director scheme 130.

[0105] In various embodiments, the orchestration director component 110 may respond to the received image processing request 106 by an initial verification request. In this context, verification refers to ensuring that the request is received from an authorized device / system. In this regard, the algorithmic orchestration comment 110 may be configured to only process image requests received from an authorized image provider system / device 102. Information identifying the authorized image provider system / device 102 from which the request can be received may be provided in one or more AO databases 120 (e.g., system / device registry data 126). In some implementations, after verifying the request, the orchestration director component 112 may generate a request identifier for the request and add the request to a request processing queue (not shown). Information about all received requests may also be stored in one or more AO databases 120 and tracked by the system 100.

[0106] After the request has been verified, the orchestration director component 112 may then direct the workflow execution engine 116 to execute one or more workflows identified / included in the workflow and task registry data 122 for the medical image or images associated with the request. The workflow execution procedure involves executing the instructions defined by the workflow, including executing the steps / actions of the corresponding nodes as configured. In this regard, the workflow execution engine 116 executes the workflow by executing the code / logic defined by the workflow as provided in the workflow and task registry data 120. As described above, this may involve executing tasks (e.g., task nodes) corresponding to algorithms stored at one or more file sharing data sources 132, including internal algorithms 134 and third-party algorithms 136.

[0107] In various embodiments, the workflow execution engine 116 accesses these algorithms as web services by calling / calling the algorithms at their network accessible file sharing data source locations (e.g., using their corresponding API calls as defined by the workflow code / logic). For example, associated with processing an input image through a workflow and encountering a task node corresponding to an internal algorithm 134 and / or a third-party algorithm 136, the workflow execution 116 engine can access the algorithm using the algorithm execution instructions defined for the task node, have the algorithm applied to the image, and receive the result / result. Additionally or alternatively, the algorithm / model can be integrated into the workflow as a "job". Specifically, the algorithm / model and other tasks defined by computer executable instructions can be wrapped in a Kubernetes Job function, and the algorithm orchestration component can asynchronously execute the function within the cluster on behalf of the user. In some embodiments where the algorithm is an internal algorithm, the workflow execution engine 116 can access and run the model / directly apply the model to the image. Using these embodiments, the workflow execution engine can be configured to apply the algorithm to the input image. In some embodiments where the algorithm is a third-party algorithm, the workflow execution engine 116 can send the input image to the third-party system with a request that the third-party system apply the algorithm to the input image and return the result. Using these embodiments, the third-party system can apply / execute the algorithm and provide the result back to the workflow execution engine 116.

[0108] Additionally or alternatively, the workflow execution engine 116 may employ an algorithm execution engine 118 to facilitate execution of the algorithms included in the workflow. Using these embodiments, the algorithm execution engine 118 may access the algorithm and apply the algorithm to the input image regardless of its source (e.g., internal or third party) using its corresponding API, and provide the results back to the workflow execution engine 116. For example, in some embodiments, the workflow execution engine may direct the algorithm execution engine 118 to manipulate the following operations: calling the algorithm / model defined by the task node at its network accessible location (e.g., at one or more file sharing data sources 132), providing it with medical images for the algorithm / model to process, running the algorithm / model, and receiving the algorithm / model output / results. In this context, the algorithm execution engine 118 may be or correspond to an inference engine that executes the algorithm / model included in the workflow and provides the results to the workflow execution engine 116.

[0109] Once the workflow execution engine has completed executing the workflow and compiled the results / outputs of the algorithm runs, the orchestration director component 122 can provide the results back to the requesting image provider system / device 102. For example, the results 108 may include information identifying the applied workflow, the applied algorithm, and the algorithm results. In some embodiments, the algorithm orchestration component 110 may also provide status notifications about the status of the image processing request to the requesting entity. For example, the status notification may indicate whether the workflow has been initiated and when the workflow has been completed. In this regard, based on various determinations made in response to the execution of the workflow, the algorithm orchestration component may generate notifications (e.g., text, email, reports) and send the notifications to the image provider, radiologist, or another related healthcare system to assist them in reporting patient results and diagnoses.

[0110] As noted above, the image processing request 106 corresponds to a request to process a medical image or study (e.g., where the study includes two or more medical images) using one or more image / medical image processing algorithms. In some embodiments, the image provider may identify or indicate a specific algorithm and / or workflow to be applied to the medical image or study in association with providing the image processing request 106. With these embodiments, the image processing request 106 may include information identifying or indicating a specific algorithm and / or workflow to be applied.

[0111] In other embodiments, the orchestration director component 112 may be configured to direct the workflow execution engine 116 to apply all available workflows included in the workflow and task registry data 124 to the medical image / study. With these embodiments, it is assumed that the decision nodes included in the workflow control the application of the appropriate workflow to the medical image. In this regard, as FIG. 2A to FIG. 2C , if the medical image fails to satisfy one or more criteria controlled by the decision node, the workflow will end and the status of the workflow may be considered "not applicable". In some implementations of these embodiments, the algorithm orchestration component 110 may notify the requesting entity as to which workflows are applicable and / or not applicable, the number of applicable / not applicable workflows, etc.

[0112] Additionally or alternatively, the orchestration director component 112 may be configured to evaluate the medical images / studies associated with the image processing request 106 based on the workflow and task registry data 122 and / or the algorithm catalog data 124 to determine which workflows are to be applied to the medical images / studies. In this regard, the orchestration director component 112 may evaluate information associated with the respective workflows and / or algorithms / models included therein that identify or indicate the medical images to which the workflows are applicable, and select only those workflows for application to the medical images / studies associated with the request. For example, the workflow and task registry data 124 may include information for each (or in some implementations, one or more) workflows defined therein that identify or indicate one or more defined criteria for an input image that may be processed by the workflow. For example, the one or more defined criteria may be based on the image / study type, the capture modality, the anatomical region / body part depicted, the medical condition reflected, and various other factors. Using these embodiments, orchestration director component 112 may analyze the images / studies and / or their associated metadata to determine and select any applicable workflows defined in workflow and task registry data 122 .

[0113] The manner in which the algorithm orchestration component 110 receives / retrieves the actual image data associated with the image processing request may also vary. In some embodiments, the image processing request 106 may include the actual image or study to be processed by the algorithm orchestration component 110. In other embodiments, the image processing request 106 may include metadata describing the relevant attributes / parameters of the image or study, and the algorithm orchestration component 110 may be configured to initially determine whether any workflow is applicable to the image or study in response to the request. With these embodiments, if at least one applicable workflow is found, the orchestration director 112 may receive / retrieve the actual image or study from its source (e.g., PACS, VNA, etc.). For example, in some embodiments, the orchestration director component 112 may direct the image provider from which the request is received to provide the image or study in response to determining that at least one workflow is applicable. In still other embodiments, the image processing request 106 may include an identifier of the image or study associated with the request, which may be used by the orchestration director component 112 to access the image or study at its storage location. For example, image processing request 106 may identify a new study that has been added to a particular provider's PACS and include a request to process the new study. Using these embodiments, orchestration director component 112 and / or workflow execution engine 116 may be configured to access and retrieve studies from the PACS in association with executing one or more workflows thereon.

[0114] The manner in which the image processing request 106 is received by the algorithm orchestration component 110 may also vary. In some embodiments, the image processing request 106 may be sent from the image provider system / device 102 in response to a user input having an application interfaced with the algorithm orchestration component 110. For example, in some implementations, the application may include a medical image viewer application that provides access to and viewing of medical images. The viewer application may also be configured to interface with the algorithm orchestration component 110 and provide for submitting the image processing request 106 in an associated selection of a particular image or imaging study. In some implementations of these embodiments, the image processing request may also include user-defined instructions that indicate or identify which workflow or workflows are to be applied to the medical image or imaging study. For example, the request may include information identifying one or more medical image processing algorithms required to be applied to the medical image / imaging study.

[0115] In other embodiments, the image provider system / device 102 may be configured to push the image processing request 106 to the algorithm orchestration component 110 based on the occurrence of one or more events and / or conditions. For example, in some embodiments, the image provider system / device 102 may include or be operably coupled to one or more medical image databases (e.g., PACS) and is configured to notify the algorithm orchestration component 110 in response to receiving a new image / study in one or more medical image databases. In some implementations of these embodiments, the orchestration director component 112 may be configured to respond to the notification by processing the newly added image / study using an applicable workflow. In this regard, the orchestration director component 112 may automatically initiate the processing of the new image / study as the new image / study is added to the one or more medical image databases of the image provider. The orchestration director component 112 may also automatically provide the result and status notification 108 back to the image provider system / device 102.

[0116] Figure 3 A block diagram of another example algorithm orchestration system 300 according to one or more embodiments of the disclosed subject matter is shown. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0117] The system 300 presents an example embodiment that integrates a medical image viewer application 304 and a medical image storage system 306, which may be or correspond to a PACS, a VNA, etc. According to the system 300, a clinician may employ a clinician device 302 to access / execute the image viewer application 304 to view medical images stored in the medical image storage system 306. In this regard, the clinician device 302 may include any suitable computing device capable of accessing the viewer application 304 and viewing medical images stored in the medical image storage system 306.

[0118] In various embodiments, the image viewer application 304 may provide image tools for viewing and interacting with medical image data stored at the medical image storage system 306. For example, in various embodiments, the image viewer application 304 may be or correspond to a medical image viewing application / program accessible using a web-based platform (e.g., a web application, a client application, etc.). The image viewer application 304 may include software and / or hardware components that facilitate viewing of medical images stored at the medical image storage system 306, and may optionally provide various interactive features for interacting with the image (e.g., changing viewing angles, editing images, applying annotations, etc.). For example, in some implementations, the image viewer application may provide annotation tools for annotating medical images, tools for applying and / or editing DICOM tags, and tools for reviewing / rating medical images. The image viewer application 304 may also be configured to present request results and status notifications 108 to clinicians. In some embodiments, the image viewer application 304 may also provide for receiving user input (e.g., from a clinician, such as a radiologist, etc.) for initiating / submitting an image processing request 106.

[0119] According to the system 300, the algorithm orchestration component 110 may interface with the medical image storage system 306 to receive the image processing request 106, retrieve the requested medical image data, and provide the request result and status notification 108. The image processing request 106 may be initiated from the image viewer application 304 and / or the medical image storage system 306. The request result and status notification 108 may also be provided back to the medical image storage system 306 and / or the image viewer application 304.

[0120] For example, in one or more embodiments, when a new medical image / image study is received at the medical image storage system 306, the medical image storage system 306 may be configured to send an image processing request 106 as an HTTP request to an API exposed by an API gateway (not shown) of the algorithm orchestration component 110. For example, the image processing request 106 may be identified as a "study process notification" and may contain study metadata and a payload. The gateway may be configured to forward the request to the algorithm orchestration director component 112, which validates the request payload and assigns a unique request ID to the request. In some implementations, the orchestration director component 112 may also notify the medical image storage system (and / or the image viewer application 304) that the request has been received and validated. The notification may include the request ID and indicate the current status of the request (e.g., status = "received and validated request"). The algorithm orchestration component may then direct the workflow execution engine 116 to invoke all (or one or more) applicable workflows for the image / imaging study. In implementations where two or more workflows are applicable to an image / image study, the workflow execution engine may execute each workflow as a separate thread.

[0121] As reference FIG. 2A to FIG. 2C As illustrated, a typical workflow would begin by validating the medical image metadata using one or more decision nodes to determine whether the image meets the defined workflow requirements (e.g., regarding modality, viewing location, study description, etc.). According to these workflows, if the workflow execution engine 118 determines that the image / image study meets the workflow requirements, the algorithm orchestration component 110 and / or the workflow execution engine 116 transfers the image / study data from the medical image storage system 306 to a local file storage device (e.g., stored in one or more AO databases 120) employed by the algorithm orchestration component 110. When the transfer is complete, the workflow execution engine 116 executes all algorithms defined in the workflow. In this regard, for each algorithm included in the workflow, the workflow execution engine 116 calls / invokes the algorithm as defined and waits for a response. Once responses have been received for all algorithms included in the workflow, the algorithm orchestration component transfers the request results 108 back to the medical image storage system 306 and sends a notification message to the image viewer application 304, which notifies the clinician that the processing of the image / study is complete. This notification message may also include information identifying the specific algorithms executed and the results of each algorithm. The request results 108 may include output files produced by the algorithms that may be viewed / reviewed by the clinician using the image viewer application 304 .

[0122] refer to Figure 1 and Figure 3In addition to orchestrating the application of AI algorithms to medical images of image providers, the algorithm orchestration component 110 also provides various management tools 114 for administrators. These management tools may include, but are not limited to: tools for loading algorithms to be included in workflows, tools for creating and configuring workflows, tools for configuring algorithm parameters used in workflows, tools for defining which workflows will be applied to input image processing requests, and tools for tracking, monitoring, and auditing requests received and processed. In the embodiments shown in system 100 and system 300, an administrator may access and employ these management tools using an administrator device 104. The administrator device 104 may be or correspond to any suitable computing device capable of accessing the algorithm orchestration component 110 and employing the management tools 114 provided thereby. Reference is made below to FIG. 5A to FIG. 16B The features and functionality of management tool 114 are described in more detail.

[0123] Deployment Architecture Algorithm Orchestration System 100, Algorithm Orchestration System 300, and other systems described herein may vary. In some embodiments, one or more components, devices, engines, databases, systems, or other elements of system 100, system 300, and other systems described herein may be deployed in a cloud architecture, a virtualized enterprise architecture, or an enterprise architecture, where one or more features and functions of algorithm orchestration component 110 are accessed by client devices / systems (e.g., image provider system / device 102, administrator device 104, and / or clinician device 202) via one or more networks using a server / client relationship. Fig.17 , FIG. 18A to FIG. 18B and Figure 29 to Figure 32 Various example deployment architectures for algorithm orchestration system 100 and algorithm orchestration system 300 are described in further detail.

[0124] Figure 4A high-level flow chart of an example computer-implemented method 400 for executing workflow instructions to analyze medical images and associated metadata in accordance with one or more embodiments of the disclosed subject matter is presented. According to the method 400, at 402, a system (e.g., system 100, system 300, and other systems described herein) operably coupled to a processor may receive a request (e.g., image processing request 106) to process a medical image using at least one medical image inference model (e.g., one or more of internal algorithms 134 and / or external algorithms 136). At 402, the system identifies a workflow (e.g., using an orchestration director component 112) that includes a medical image inference model applicable to a medical image, wherein the medical image inference model is stored at a network accessible source (e.g., one or more file sharing data sources 132). For example, the orchestration director component 112 may select a workflow applicable to a medical image included in the workflow and task registry data 124 based on one or more requirements of an input image of the workflow, such as image type, modality, depicted body part, etc. At 406, the system can use the medical image (e.g., using the workflow execution engine 116) to execute the workflow, including accessing the medical image inference model at a network-accessible source (e.g., as a web service and / or job), and applying the medical image inference model to the medical image, thereby generating a workflow result (e.g., including the result of the inference model). At 408, the system can provide result information about the workflow result to an entity associated with the request (e.g., an image provider, a radiologist authorized to access and view the image, etc.).

[0125] FIG. 5A to FIG. 5B A block diagram of an example algorithm orchestration component 110 according to one or more embodiments of the disclosed subject matter is provided. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0126] refer to Figure 5A , the algorithm orchestration component 110 may include several computer executable components that respectively provide the various computer executable functions described therefor. Figure 5A In the embodiment shown in , these computer executable components include the orchestration director component 112, the notification component 502, the access component 504, the security component 508, the user interface component 510, the feedback component 515, the algorithm management comment 516, the task registry component 524 and the workflow management component 526. The algorithm management comment 516 and the workflow management component 526 also include several subcomponents that can be or correspond to the computer executable components respectively. Figure 5B In the embodiment shown in FIG. 1 , the management tool 114 may also include a system / device management component 554 and an activity management component 562. It should be understood that the subcomponents of the algorithm management component 516 and the workflow management component 526 are described in detail in the following embodiments. Figure 5B It was removed only due to lack of display space.

[0127] The algorithm orchestration component 110 may also include or be operatively coupled to at least one memory (not shown) storing computer executable components, and at least one processor (not shown) executing the computer executable components stored in the memory. Fig. 27 Examples of the memory and processors described herein, and other suitable computer or computing-based components that may be incorporated into the implementation Figure 6 Or used in combination with one or more of the systems or components shown and described in other figures disclosed herein.

[0128] It should be appreciated that various implementations of the algorithm orchestration component 110 may include Figure 5A and Figure 5B In addition, one or more components of the algorithm orchestration component 110 may be deployed / executed by different devices / machines in a distributed and / or parallel computing environment.

[0129] refer to Figure 1 , Figure 3 , Figure 5A and Figure 5B In various embodiments, notification component 502 may provide for generating and sending / providing notifications to image provider systems / devices 102 and / or other system devices regarding the activity of algorithm orchestration component 110. For example, in some embodiments, notification component 502 may be configured to provide status notifications regarding the status of image processing requests 106. For example, notification component 502 may provide status notifications indicating whether and when an image processing request is received and validated, status notifications regarding whether any workflows and / or algorithms / models are being executed (e.g., ongoing workflow / model executions), and status notifications regarding workflow completions.

[0130] Notification component 502 may also be configured to provide a notification regarding whether an applicable workflow has been identified for an image / study associated with the image processing request. For example, in some embodiments, in response to receiving an image processing request 106 to process a medical image, orchestration director component 112 may use metadata describing properties of the medical image to identify any matching workflows applicable to the medical image based on information included in workflow and task registry data 122. Notification component 502 may also notify an entity associated with the request regarding the identified applicable workflow. In some embodiments, information describing the applicable workflow may also be included in the notification, such as information describing the type of algorithm / model included in the workflow.

[0131] The access component 504 and the security component 506 may control access to the algorithm orchestration component 110 by system, device, and user. For example, as noted above, in some embodiments, the algorithm orchestration component 110 may limit the acceptance of image processing requests 106 to only those systems / devices identified in the system / device task registry data 126. Using these embodiments, the security component 508 may control the authentication of the image processing request 106 using the system / device task registry data 126 to ensure that only requests from authorized systems / devices are authenticated. The security component 508 may also define and enforce security protocols to be employed by the algorithm orchestration component 110 when exchanging files with other components. This includes how components of a system (e.g., system 100, system 300, etc.) communicate and what type of security credentials are passed for the respective communications to be conducted.

[0132] The access component 504 and / or the security component 508 may also control access and use of the management tool 114 by authorized users. Users employing the management tool are generally referred to herein as "administrators." In this regard, the management tool 114 is designed to be used by administrators of a system (e.g., system 100, system 300, etc.) to manage, control, and track the use of algorithms and workflows by imaging providers. The management tool may also be used by workflow developers to create and edit workflows and algorithms.

[0133] Feedback component 515 can help receive feedback about workflow results from imaging providers, clinicians (e.g., radiologists, specialists, etc.). For example, in various embodiments, workflow orchestration component 110 can present workflow results to clinicians via an image viewer application (e.g., image viewer application 304, etc.), which can also include a mechanism for providing feedback about inference model results from one or more entities (reviewers). For example, in some implementations, the viewer application can provide a review prompt with one or more defined review questions about model outputs, which can be answered using predefined optional response options and / or open text. The review questions can vary depending on the corresponding inference model. In one or more embodiments, the review questions can include at least one question about the accuracy of the model output. For example, the review question can require the user to provide binary feedback stating whether the inference result is correct or incorrect. In another example, the review question can require the user to rate the accuracy of the model result using a defined rating / scoring scale. In some embodiments, the viewer application can also provide a mechanism for review to provide feedback on identified errors and / or corrections to errors in the inference results. Any feedback collected from the auditors can be provided back to the feedback component 515 and used to evaluate and monitor model performance (eg, via the activity management component 562).

[0134] In various embodiments, the algorithmic orchestration component 110 may provide the management tool 114 via a network-accessible platform, such as a web application, a mobile application, etc. In other implementations, the algorithmic orchestration component 110 may provide the management tool 114 as a management local application. Regardless of the deployment platform for the management tool 114, the access component 504 may control authorized users' access to some or all of the management tool 114. With these embodiments, the access component 504 may include an account component 506, which may store information identifying the authorized users and their account access information (e.g., username and password, etc.), which is required for the respective authorized users to access the management tool.

[0135] For example, Figure 6 An example management login page for accessing an algorithm orchestration management application according to one or more embodiments of the disclosed subject matter is presented. The algorithm orchestration management application is referred to herein as the "AI orchestrator." In the illustrated embodiment, in order to access the AI ​​orchestrator, a user must sign in with their correct account information (e.g., username and password).

[0136] In some embodiments, access component 504 may apply usage restrictions regarding the use of certain features and functions of the system based on individual permissions granted to certain user accounts based on their identity, position, account status, authorization, etc. For example, in some embodiments, access component 504 may manage and / or control the use of certain management tools and features, such as algorithm parameter adjustment capabilities, workflow creation capabilities, etc., based on the authorizations / restrictions associated with the corresponding user accounts.

[0137] The algorithm orchestration component 110 may include a user interface component 110 to facilitate generating and providing various UIs of the algorithm management application, whether an application of a web-based platform (eg, a web application, a mobile application, etc.) or a native application. Figures 7 to 16B Several example UIs that may be generated and provided by user interface component 510 are presented in FIG. Figure 7 A to Fig. 16B The UI presented in exemplifies various features and functions of the management tool 114, including the features and functions of the algorithm management component 516, the task registry component 524, the system / device management component 554, and the activity management component 562. Details regarding these UIs are described with reference to their corresponding components.

[0138] In this regard, the algorithm management component 516 may provide various management functions related to the algorithms used in the workflow, including viewing existing algorithms, loading algorithms, editing algorithms, searching algorithms, and deleting algorithms. In various embodiments, the algorithm management component 516 may provide access to the algorithm catalog data 124 for authorized administrators (e.g., after successful login). The algorithm management component 516 may also provide tools for performing the following operations: searching and filtering the algorithms included in the algorithm catalog data 122 by various criteria associated therewith, such as included in the catalog data. For example, in various embodiments, the corresponding algorithms included in the algorithm catalog data may include information identifying or describing the type of algorithm, the function of the algorithm, the input image criteria (e.g., patient parameters, modality, image attributes regarding quality, orientation, etc.), provider, version, last updated time, creator, etc. Any of these factors may be used to search and filter the algorithms included in the catalog. The algorithm management component 516 may also provide tools for adding algorithms and removing, transferring (e.g., from one file location to another file location), configuring, and editing algorithms. To facilitate this, the algorithm management component 516 may include a loading component 518 , a conversion component 520 , and a configuration component 522 .

[0139] In one or more embodiments, the loading component 518 provides the system with a loading function for loading algorithms included in the algorithm catalog data 124. In this context, loading refers to loading or adding information execution information required to access and execute an algorithm (e.g., an AI model) into the algorithm catalog. In various implementations, only algorithms that have been loaded and added to the algorithm catalog data 124 can be integrated into the workflow and applied to the medical image through the algorithm orchestration component (e.g., using the workflow execution engine 116 and / or the algorithm execution engine 118). The loading component 518 provides for loading both internal algorithms 134 and third-party algorithms 136.

[0140] The conversion component 520 provides a conversion function for converting the algorithm format in association with the algorithm loading so that the algorithm format is compatible with the format used by the algorithm orchestration component 510. This is usually used for third-party algorithms. Specifically, if the algorithm to be loaded is written (e.g., encoded), stored, or accessed / run in a format different from the format used by the algorithm orchestration component 510, then the workflow execution engine 116 and / or the algorithm execution engine 118 will not be able to properly access and run the algorithm. The conversion component 520 can perform a mapping between the format used by the algorithm orchestration component 510 and the format used by the third-party algorithm (or any algorithm written in a different format), and store this mapping with algorithm information in the algorithm catalog data 122. For example, the conversion component 520 can create a mapping between the API used by the algorithm execution engine 118 and the API used by the third-party provider. In some embodiments, when an algorithm is loaded based on a determination that its native format is not a format adopted by the algorithm orchestration component 510, the conversion component 520 can automatically create a mapping of the algorithm from its third-party format to the format adopted by the algorithm orchestration component 110. The conversion component 520 may also automatically store conversion / mapping information with the loaded algorithm in the algorithm catalog data 122 for use by the algorithm execution engine 118 .

[0141] Configuration component 522 provides configuration of one or more parameters and / or parameter values ​​of the algorithm included in the algorithm catalog data. For example, configuration component 522 can be used to set / adjust threshold parameters, including thresholds of criticality or priority, clinical relevance, diagnostic thresholds, probability thresholds, confidence score thresholds, etc. For example, with respect to a diagnostic model, configuration comment 522 can allow an administrator to set / adjust the confidence score threshold of the model for classifying a positive or negative diagnosis. Configuration component 522 can also provide setting / adjusting model input image parameters, which control the requirements of the input image to which the model can be applied.

[0142] In some embodiments, the configuration component 522 may also provide rules / instructions for setting / adjusting the criticality or priority of the model. In this regard, the algorithm / model may provide different AI solutions for different situations. Some models may be used for trauma situations (e.g., detecting brain leads), which have a higher urgency relative to route following or long-term analysis (e.g., using a knee segmentation model to assess knee injuries). Using these embodiments, the criticality or priority level of the model may control the order in which the model is applied to receive images in an image processing request, wherein the workflow execution engine 116 and / or the algorithm execution engine 118 may be configured to execute a higher priority model (and / or a workflow including a higher priority model), followed by a lower priority model (and / or a workflow including a lower priority model). For example, in some embodiments, when a new image processing request 106 is received (e.g., as pushed from a PACS, at the request of a clinician, etc.), the orchestration guide component 112 may identify all applicable workflows for medical images / research associated with the request. The orchestration director component 112 may also direct the workflow execution engine 116 to execute workflows in order based on the priority / criticality of the algorithms included in the workflows. In this regard, the orchestration director component 112 may direct the workflow execution engine to execute workflows that include high priority algorithms (e.g., relative to a defined threshold) before workflows that do not include high priority algorithms.

[0143] In various embodiments, the configuration component 522 may store any changes to the algorithm parameters with the algorithm information in the workflow and task registry data 122, where the information identifies the date / time when the update to the model was made and the administrator who applied the update. In some embodiments, the configuration component 522 may generate a new version of the model or algorithm in response to a change / adjustment of the model parameters. With these embodiments, any update / change in the model parameters may result in the creation of a new version of the model with the new parameters, and the previous version may be maintained in the algorithm catalog data 124.

[0144] Figure 7An example algorithm management user interface UI 701 according to one or more embodiments of the disclosed subject matter is presented. In some embodiments, the algorithm management UI 701 may be presented in response to selecting an algorithm tab from the upper toolbar. In this example implementation, the algorithm management user interface UI 700 may provide a searchable / filterable list of algorithms included in the algorithm catalog data 124. The algorithm list may provide certain high-level information for each algorithm, including a display name (i.e., a name / identifier for the algorithm), a date after the last update, a provider, and a version. In this example and other examples presented herein, the provider is "internal" or "third party". It should be understood that a third-party provider may also be specified to identify the specific third party mentioned. In addition, an internal provider may be identified more specifically (e.g., by name, location, etc.). In various embodiments, an algorithm included in the list may be selected to view more detailed information of the algorithm, and to configure / edit the parameters of the algorithm. The algorithm management user interface UI 701 also provides access to the algorithm loading function (e.g., via clicking / selecting the loading algorithm icon 702) and access to the algorithm editing function (e.g., via clicking / selecting the algorithm editing icon 704).

[0145] Figure 8 An example algorithm management editor UI 801 is presented in accordance with one or more embodiments of the disclosed subject matter. In some embodiments, the algorithm editor UI 801 can be displayed in response to selecting a particular algorithm from the list view of the algorithm management UI 701 in conjunction with a request to edit the algorithm (e.g., via selecting the algorithm editor icon 704).

[0146] The algorithm management editor UI 801 provides an example interface to operate the configuration component 522 to configure or adjust one or more parameters of the algorithm. In the embodiment shown, five algorithms are displayed, and an editor window 804 is opened for the selected algorithm from the list, and this example is the first algorithm 802 in the list. The remaining four algorithms are listed below the editor window, as indicated by reference arrow 806. Each algorithm in the five algorithms presented includes information identifying the last update date, display name, algorithm type (e.g., PTX, stroke, chest front, patient position, and simulation), provider, version, and algorithm criticality (or priority). An algorithm criticality slider is provided for the corresponding algorithm in the list to adjust the criticality given to the selected algorithm. Actions can be directly initiated from the algorithm editor UI 801 for each algorithm, including deleting the algorithm and removing the algorithm from the algorithm catalog data 122.

[0147] The editor window 804 provides some additional details of the selected algorithm, including a description of the model, creator, release date, and supported modalities. Threshold settings applicable to a given model are also presented, which can be adjusted via the UI. For example, thresholds may include probability thresholds for detecting a given condition (e.g., stroke, patient position) related to the model analysis of a given image and / or image metadata. The editor window also provides for setting clinical relevance thresholds related to the model analysis of a given image and / or image metadata. It should be understood that the parameters and thresholds available for editing shown in the algorithm management editor UI 801 are merely exemplary, and various other parameters and thresholds that may vary between models may be adjusted.

[0148] FIG. 9A to FIG. 9B An example algorithm loading UI 901 according to one or more embodiments of the disclosed subject matter is presented. In various embodiments, the algorithm loading UI 901 can be presented in response to selecting the algorithm loading icon 702 from the algorithm management UI 701. The algorithm loading UI 901 provides an example interface for operating the algorithm loading component 518 and the conversion component 520. According to this example implementation, the loading component 901 can provide for loading internal algorithms and third-party algorithms by selecting a corresponding provider via the provider button 902. Fig.9A presents loading of internal algorithms (e.g., algorithms provided by the same entity that owns / manages algorithm orchestration component 110), and Fig. 9B Shows loading of third-party algorithms.

[0149] refer to Fig.9A , the algorithm loading UI 901 provides several data fields for receiving administrator input in association with loading an internal algorithm, including receiving input identifying the algorithm name, version, display name, input format, and output format. In one or more embodiments, in response to selecting "internal" as the provider, the loading component 520 interfaces with an internal index of algorithms accessible to the algorithm orchestration component 110. For example, this index of algorithms may be different from the algorithm catalog data 122 and include a larger set of algorithms developed internally from which internal algorithms can be selected for loading into the algorithm catalog data 122. With these embodiments, the algorithm name and version can be selected from a drop-down list that provides all (or a filtered subset) of the algorithms included in the index. Upon selecting the desired algorithm name, the algorithm loading component 518 can extract and import the algorithm details (e.g., from the algorithm index) into the remaining data fields. For example, the loading component 518 can extract and import details about the description, creator, creation date, last updated date, input XSL fields, and output XSL fields. The algorithm loading UI 901 also provides an option to add a unique name for the algorithm display name as preferred by the administrator.

[0150] The algorithm loading UI 901 may also provide some configuration functions for configuring one or more parameters and thresholds of the algorithm. For example, in the illustrated embodiment, the algorithm loading UI 901 provides for receiving inputs that set the criticality of the algorithm to be loaded. Other algorithm parameters and / or thresholds may also be added to the UI for configuration at the time of loading. After the algorithm information is entered into the corresponding data fields, selecting the "Save" icon may cause the algorithm to be loaded into the algorithm catalog data 122 (e.g., via the loading component 518).

[0151] refer to Figure 5B , the third-party algorithm loader may differ slightly from the internal algorithm loader. Specifically, the third-party algorithm loader needs to receive an input identifying a model route, which corresponds to an accessible location / route for accessing the third-party model. In some embodiments, the third-party providers from which models can be loaded can be predefined, and the route of their models / algorithms can be selected from a drop-down menu in the route field. With these embodiments, one or more of the algorithm information data fields can be automatically populated in response to selecting an algorithm route and / or name from the drop-down menu.

[0152] The algorithm loading UI 901 also provides for editing the input and output formats of the algorithm. For example, in various embodiments, the administrator may provide inputs that change the input and / or output data formats required by the third-party algorithm in the input XSL and output XSL data fields. In response to the inputs that change the formats here, the conversion component 520 may create a mapping from the third party described in the API to the API supported by the algorithm orchestration component 110 in association with the saving and loading algorithms. In this regard, via the algorithm loading UI 901, the administrator may edit the input format (e.g., via the input XSL field) to define an input transformation to transform the available metadata in the expected input format of the algorithm (e.g., via the conversion component 520). The administrator may also edit the output format (e.g., via the output XSL field) to define an output transformation to convert the corresponding transformation from the algorithm (e.g., via the conversion component 520) to the standard format used by the algorithm orchestration component 110. In this regard, based on the defined input and output formats provided for the algorithm, the conversion component 520 may convert the algorithm format from a first format that is incompatible with the algorithm execution engine to a second format that is compatible with the algorithm execution engine (e.g., wherein the second format is compatible with the API adopted by the algorithm execution engine).

[0153] Reference again Figure 5A , the workflow management component 526 provides various functions related to managing, creating and editing workflows. To facilitate this, the workflow management component 526 may include a workflow creation component 528, an activation control component 546, an import / export component 548, a parameter tuning component 550 and an application control component 522.

[0154] In various embodiments, the workflow management component 526 can provide access to the workflow and task registry data 122 for authorized administrators (e.g., after successful login). For example, the workflow management component 526 can provide viewing of existing workflows, including detailed information about each workflow, such as information about the algorithms included therein, the criteria of the medical images to which it is applicable, the creator, the creation date / time, etc. The activation control component 546 can provide workflow activation and deactivation functions in association with accessing and viewing existing workflows. Specifically, the activation control component can allow a user to provide input to activate and deactivate a workflow. The activation control component 546 can also activate or deactivate a workflow accordingly. In this regard, when a workflow has been deactivated (in response to receiving a user input to deactivate the workflow), the activation control component 546 can change its state to deactivate in the workflow and task registry data 122. When a workflow has been deactivated, the workflow will become unavailable for execution by the workflow execution engine 116. Likewise, when a workflow is activated (in response to receiving user input to activate the workflow), the activation control component 546 may change its state to active in the workflow and task registry data 122. Only activated workflows will be available for execution by the workflow execution engine 116.

[0155] The import / export component 548 can provide for importing and exporting workflows. In this regard, the import / export component 548 can receive a workflow file stored at another location and add the workflow file to the workflow and task registry data 122. Likewise, the import / export component 548 can export or transfer the workflow file to a selected destination location for execution by another system.

[0156] The parameter operation component 550 can provide the same or similar features and functions as the configuration comment 522. In this regard, the parameter tuning component 550 can provide for adjusting the parameters and / or thresholds of the algorithms included in the workflow. Additionally or alternatively, the parameter tuning component 550 can provide for editing and / or adjusting the parameters of other nodes included in the workflow. For example, the parameter tuning component 550 can provide for editing / adjusting the parameters of decision nodes and function nodes.

[0157] The application control component 552 may provide for receiving user input that defines one or more conditions for controlling the execution / application of the workflow by the workflow execution component 116. Information defining the workflow execution conditions may also be associated with the workflow in the workflow and task registry data 122 and used by the orchestration director component 112 and / or the workflow execution engine 116 to control the execution / application of the workflow accordingly. For example, in some embodiments, the application control component 552 may allow a user to define input image criteria regarding the input images to which the workflow applies. The workflow execution conditions may relate to the imaging provider from which the images or studies were received (e.g., only apply this workflow for imaging provider 1 and imaging provider 3), timing (e.g., do not apply this workflow during peak hours), system load (e.g., do not apply this workflow when the system load exceeds X%), other workflows running, workflow priority, etc.

[0158] The workflow creation component 528 may provide a workflow creation application, through which developers may create and edit workflows. The user interface component 510 may also provide a workflow creation UI 512, through which users may access and employ the features and functions of the workflow creation component 528. The workflow creation component 528 includes several components that respectively provide different features and functions for creating and configuring workflows, including a decision configuration component 530, a fork configuration component 532, a sub-workflow configuration component 534, a wait configuration component 536, a task configuration component 538, a function configuration component 540, a simulator component 542, and a label component 544. FIG. 11A to FIG. 12D The workflow creation component 528 and its subcomponents are discussed in greater detail.

[0159] FIG. 10A to FIG. 10B An example workflow management UI 1001 is presented in accordance with one or more embodiments of the disclosed subject matter. In various embodiments, the workflow management UI 1001 can be generated and presented in response to selecting the "Workflow" tab from the upper toolbar. Fig. 10A , the workflow management UI 1001 may provide a searchable and filterable list of available workflows and sub-workflows (e.g., where a sub-workflow only runs when its parent workflow is running). The list view may identify the workflow / sub-workflow by name, the time the workflow / sub-workflow was last updated, the type (e.g., workflow or sub-workflow), the creator, and the status (e.g., activated / active or deactivated / paused). In the illustrated embodiment, three workflows are displayed, two workflows are disabled, and one workflow is active. The workflow management UI 1001 provides status adjustment buttons for workflows, which may be used to activate and reactivate (or pause) a workflow directly from the UI.

[0160] In various embodiments, workflows and sub-workflows may be selected from the list to view additional details about the workflow / sub-workflow, to edit the workflow, to delete the workflow, and / or to perform other actions related to the workflow. Fig. 10B As shown, in some implementations, right-clicking a workflow or sub-workflow from the list can cause a pop-up window 1002 to be generated with various actions / functions that can be selected and performed in relation to the workflow, including adding or removing tags, changing parameters, exporting the workflow, or running a workflow simulator (as described below). In this example, the selection of the add or remove tag function can cause a tag pop-up window 1004 to be generated, which has a selectable list of tags that can be added or removed from the workflow. The workflow management UI 1001 also provides a workflow creation button ("Create New workflow") that can be selected to open a workflow creation application / feature provided by the workflow creation component 528.

[0161] FIG. 11A to FIG. 11X An example UI of a workflow creation application provided by workflow creation component 528 in association with creating a workflow using the workflow creation application is presented in accordance with one or more embodiments of the disclosed subject matter.

[0162] Fig.11A An example workflow creation UI 1101 is presented that provides workflow templates for developing workflows according to one or more examples of the disclosed subject matter. In some embodiments, the workflow creation UI 1101 can be generated and presented in response to selecting a workflow creation button from the workflow management UI 1001. The workflow templates include a blank workflow template and various preformatted templates that can be used to create a new workflow. In this example, the preformatted templates include a PTX Series Aggregation Template, a Diffusion Weighted Imaging (DWI) Series Aggregation Template, an Xray Study Parent Workflow Template, a Study Stroke Parent Workflow Template, and a PACS Chest Frontal Child Workflow Template.

[0163] Fig. 11B An example workflow creation UI 1111 providing interactive tools for creating workflows according to various embodiments of the disclosed subject matter is presented. In the illustrated embodiment, the workflow creation UI 1111 is blank. The workflow creation UI 1111 provides selectable and configurable nodes 1102 that can be dragged and dropped onto a creation area 1104 to construct a workflow. In the illustrated embodiment, these nodes include decision nodes, sub-workflow nodes, task nodes, fork nodes, wait nodes, and function nodes. These nodes can be previously referenced. FIG. 2A to FIG. 2CThe corresponding nodes described provide the same functions. For the sake of brevity, repeated descriptions of similar elements are omitted.

[0164] Fig. 11C Drag and drop a decision node onto the creation area 1104 in association with building a new workflow. Workflow decision nodes can be used to filter input images at the start of a workflow to ensure that the input images meet one or more criteria of the workflow. Fig.11D As shown, after a node has been dropped onto the creation area, the node becomes selectable and configurable, and the workflow creation component 528 automatically adds connectors and endpoints.

[0165] FIG. 11E to FIG. 11F Display configuration decision node 1106. Fig.11D , the selection of decision node 1106 may cause a pop-up window 1108 to be generated, which provides fields for configuring decision inputs and outputs. In this example, the fields include situation description and initial decision. Fig.11F Presents a selection / entry situation statement. The situation statement provides for adding input filter parameters (e.g., by patient, source, modality, etc.). In this example, a drop-down list of predefined situation statements may be provided to select a decision situation statement. Additionally or alternatively, the administrator may type in a new situation statement. After input providing decision input and output fields has been received, the decision configuration component 530 generates the corresponding decision code / logic for the decision function and saves the configured decision.

[0166] Fig.11G Showing additional decision nodes for a workflow. In the illustrated embodiment, a second decision node 1112 has been added and configured, and a third decision node 1114 has been added.

[0167] FIG. 11H to FIG. 11I Shows adding a task node to the workflow. FIG. 11H to FIG. 11I As can be seen in FIG. 5 , whenever a new node is added to the workflow, the workflow creation component 528 automatically adds connectors and endpoints between the nodes.

[0168] Figures 11J to 11L Display configuration task node 1116. Fig.11J, selection of a task node may result in a pop-up window 1118 providing fields for configuring the task. As previously described, task nodes may be used to integrate HTTP service calls into a workflow. Task nodes may also be used to integrate models / algorithms and other functions into a workflow to be executed as lambda tasks, Kubernetes jobs, etc. This feature opens up a variety of possibilities that are particularly relevant to legacy systems where client services are not under HTTP and are simply command lines and / or executables. For example, a task node may be used to integrate an algorithm (e.g., an AI model) included in the algorithm catalog data 124 into a workflow as an HTTP request, a lambda task, or a Kubernetes job.

[0169] In the embodiment shown, a task can be configured by entering a request method, a request uniform resource method (URL) for the task, and a task title, or by selecting a predefined custom task from a drop-down menu. In various embodiments, a task registry component 524 can provide for generating and storing custom tasks (e.g., in workflow and task registry data 122), which can be exposed and selected in a custom task drop-down menu. Figure 11K Displays the request method selected for the task. In this embodiment, the task configuration provides configuration of four different types of HTTP tasks, including "GET", "POST", "PUT" and "DELETE". Fig.11L , in this example, a GET request method is selected and the URL for the desired AI model to be run (e.g., "xyzmodel") is entered. After the data fields have been completed and input has been received providing the request method, request URL and request title or custom task has been selected, the task configuration component 532 generates the corresponding task code / logic for the task and saves the configured task.

[0170] Figures 11M to 11N Shows adding a child workflow node 1124 to the workflow. Fig.11M As shown in FIG. 1 , a sub-workflow node can be dragged and dropped onto the creation area 1104. Thereafter, as shown in FIG. Fig.11N As shown in , a child workflow node is automatically connected to the previous node and it becomes selectable and configurable.

[0171] Figure 11O to Figure 11P The configuration sub-workflow node 1124 is displayed. Fig.11O , selection of a sub-workflow node 1124 may result in a configuration window 1126 providing fields for configuring the sub-workflow. According to this embodiment, the sub-workflow was previously a creation workflow saved as a sub-workflow type. In this regard, saving the creation workflow as a "sub-workflow" makes the sub-workflow selectable in the drop-down menu. Figure 11PAn updated sub-workflow configuration window 1128 is presented, with additional data fields for entering sub-workflow parameters that may be generated in response to selecting a particular sub-workflow, which in this example is identified as a patient location sub-workflow. These additional data fields may vary based on the selected sub-workflow. In this regard, after a sub-workflow has been selected and added and input parameters have been provided, the sub-workflow configuration component 534 generates the corresponding task code / logic for the sub-workflow and saves the configured sub-workflow.

[0172] Figures 11Q to 11R Shows adding a fork node 11130 to the workflow. Figure 11Q In the example, the forked node 1130 is dragged and dropped, and in Figure 11R In FIG. 1 , a fork node has been successfully attached after the sub-workflow 1124. As previously described, a fork node can be used to split a workflow into two or more parallel programs or strings. Figure 11R In response to adding the fork node 1130, the fork configuration component 532 automatically creates two branches. The number of branches as needed can be increased by repeatedly selecting the fork node. For example, Figure 11S The presentation adds six separate branches that may be generated in response to continued selection of the fork node 1130. In this example, the second task node 1132 is added to the first branch.

[0173] Figure 11T The process of the workflow after adding the third task node 1134 to the second branch, adding the fourth task node 1136 to the third branch, adding the second sub-workflow node 1138 to the fifth branch, and adding the function node 1140 to the fifth branch is presented. It should be understood that the additional task nodes and sub-workflow nodes can be configured as previously described. For the sake of brevity, repeated descriptions are omitted.

[0174] Figure 11U A configure function node 1140 is presented. Similar to other nodes, after a function node has been added to a workflow, it may become selectable and configurable, wherein selection of the function node causes generation of a configure function pop-up window 1142. The configure function pop-up window 1142 may be used to enter an expression (e.g., a Javascript code expression, etc.), such as a data conversion function, a calculation function, etc. that may be executed by the workflow execution engine 116. After the expression code has been entered, the function configuration component 540 generates the corresponding function code / logic for the function and maintains the configured function.

[0175] Figure 11V The display adds a wait node 1144 to the sixth branch. A wait node can be used to combine or synchronize two or more tasks. Figures 11W to 11X The display configures the waiting node in association with the selected drop node. Figure 11W As shown in , a wait node may be configured via a configure wait pop-up window 1146 by selecting a task and optionally adding input parameters. Figure 11X The task of selecting waiting is displayed. In this example, four tasks are included in the workflow, resulting in four task options that will be selected for the waiting node. In this regard, the selection of task 3, for example, will cause the workflow execution engine 116 to wait for any node to be executed after the waiting node until task 3 has been completed and the result has been received.

[0176] FIG. 12A to FIG. 12B The workflow simulation functionality of an algorithmic orchestration management application according to one or more embodiments of the disclosed subject matter is illustrated. Figure 5A and FIG. 12A to FIG. 12B After a workflow has been created, the simulator component 542 may provide a simulation function that provides for running the workflow in a test mode within the workflow creation UI 1111. For example, in Fig. 12A In the embodiment shown in , a "Run Test" icon 1204 has been added to the workflow creation UI 1111 that can be selected to initiate a simulation. In response to a request to run a workflow simulation, the simulator component 542 can direct the workflow execution engine 116 to run the workflow 1202 against one or more test images to ensure that the nodes are properly configured and to identify any problems with the workflow. The test images may include two images that meet and fail to meet any decision mode criteria.

[0177] For example, Fig. 12A Depicted is a workflow 1202 executing in simulation mode in response to selecting a run test icon 1204. In response to selecting the "Run Test" icon 1204, a progress bar 1206 may be displayed showing the progress of the test, which in this example is currently "Run Test Data." Fig. 12B The simulation results are presented. Fig. 12B , simulator component 524 can identify any problem or error of workflow, such as the problem that causes node to fail to be executed or properly executed. In this example, an error is detected at node 1208. Simulation component 542 can also produce and provide error message about the information of detected error. For example, error message can identify the node and the cause of error when error occurs. Some example errors can include but are not limited to: "no content" (indicating that there are no available resources in the response); "bad request" (indicating that the request is invalid); "unauthorized (indicating that the user / system is not authorized to access resources or perform actions); "not found" (indicating that resources do not exist); "conflict" (indicating that resources cannot be created or modified when conflicting with existing resources); "internal server error" (indicating that something unexpected has occurred); and "bad gateway" (indicating that dependency fails).

[0178] The simulation tool can also be used to test run any previously generated workflow. For example, refer again to Fig. 10B , in the illustrated embodiment, the simulator may be accessed and applied to a workflow directly from the workflow management UI 1001 via right clicking on the desired workflow and selecting the "Run Simulator" option from the pop-up window 1002.

[0179] Reference again Figure 5A and Fig. 12B , the label component 544 can also provide a workflow tagging function that can be accessed via the workflow creation UI 1111. For example, Fig. 12A B, a "workflow tag" icon 1210 is provided that can be selected to apply a tag (e.g., a metadata tag) to a workflow. The metadata tag may include any attribute that a developer or administrator wants to be associated with a workflow. For example, the metadata tags may include tags about attributes of the input image applicable to processing by the workflow (e.g., modality, anatomical part, patient attributes, image orientation, etc.), tags about the algorithms included in the workflow, tags about the workflow creation program and creator (e.g., timing, creator identity, collaborators, etc.), tags about the quality of the workflow, tags about the average execution duration of the workflow, etc. Tags added to the workflow can be used to facilitate searching and filtering workflows based on desired criteria. Tags added to the workflow can also be used by the orchestration director component 112 and / or the workflow execution engine 116 to identify applicable workflows for input images in association with image processing requests. The workflow import and export functions provided by the import / export component 548 can also be accessed in association with selecting the corresponding icon 1212 via the workflow creation UI 1111.

[0180] FIG. 12C to FIG. 12D Some additional example workflows that can be created using the workflow creation tools and functionality provided by workflow creation component 528 are presented in accordance with one or more embodiments of the disclosed subject matter.

[0181] Fig. 12CAn example of a workflow 1201 with parallel branches and sub-workflows according to one or more examples of the disclosed subject matter is shown. According to workflow 1211, a start instruction starts workflow 1211, and at 1222, a first task instruction is defined for workflow execution. At 1224, a first sub-workflow is defined for execution. As previously mentioned, a sub-workflow is a separate workflow that executes instructions (e.g., models, parallel / serial branches, tasks, etc.). At 1226, a first decision branch is executed, followed by a fork of a true path 1213 and a false path 1215 that create a first decision 1226. If the decision at 1226 is evaluated to true, then a wait instruction 1228 (e.g., a time delay or wait for some other task to complete) is executed. If the decision at 1226 is evaluated to false, branch 1215 is executed. As shown, branch 1215 also contains a first function 1230, a second function 1232, a second decision 1234, and a second sub-workflow 1236. At 1217, a combine instruction is executed to combine the processing paths from 1213 and 1215. At 1238, yet another sub-workflow is executed, and then the workflow ends.

[0182] Fig.12D FIG. 10 illustrates adding an additional workflow processing branch 1219 according to one or more examples of the disclosed subject matter. Fig. 12C 1201. Due to the similarity of workflow 1211 to workflow 1201, for the sake of brevity, not every processing instruction of workflow 1211 is described. At 1219, an additional branch is added. In place of the two parallel branches provided by the fork, a third branch 1219 is added to the existing branches of workflow 1211. The third branch 1219 includes a third decision 1240, which has a true path at 1221 and a false path at 1223. The true path 1221 includes a fourth sub-workflow at 1242, followed by a second task 1244. The false path 1223 includes a wait instruction 1246. Path 1221 and path 1223 are satisfied at sub-workflow 1228, where the program ends thereafter.

[0183] Reference again Figure 5A The task registry component 524 can provide for the creation of custom tasks that can be added to the workflow and task registry data 122 and used in the workflow. The task registry component 524 adds the ability to define custom tasks in the system using custom task templates and assign names to the custom tasks. Custom tasks can be directly selected via the task configuration pop-up window (see Fig.11J and pop-up window 1118), thereby minimizing the amount of coded knowledge (e.g., being able to determine the correct request method, such as entering a request URL and title) and user input required during workflow creation.

[0184] FIG. 13A to FIG. 13D An example task registry UI is presented that illustrates features and functionality of task registry component 524 in accordance with one or more embodiments of the disclosed subject matter.

[0185] Fig.13A An example task registry UI 1301 is presented that may be presented in response to selecting a task registry tab from the upper menu. The task registry UI 1301 presents a searchable and filterable list view of previously created custom tasks. The task registry UI 1301 provides information identifying the task name, type, last updated, creator, task description, and tags added to the task. The task registry UI 1301 also provides a "Create Task Template" button that can be selected to create a new custom task.

[0186] Fig. 13B An example custom task creation UI 1311 is presented that can be presented in response to selecting the "Create Task Template" button from the task registry UI 1301. The custom task creation UI 1311 provides several data fields that can receive user input to define a custom task. These data fields provide for input / receiving user input that defines the task name, description, selecting the color of the task icon, and selecting the task type. In this example, HTTP tasks, asynchronous tasks, and K8s-Job or Kubernetes jobs are included. In addition or alternatively, the task may include a lambda task. The remaining data input fields may vary based on the type of task selected. In this example, the selected (and default) task type is an HTTP task type. For this task type, the remaining data input fields include a request URL field (the request URL of the task is entered via the request URL field), a request method field, a request body field, and an input variable field 1302, a request title field, and a tag input field.

[0187] The input variables field 1302 may be selected to customize the input variables in the implementation of the HTTP request where the task includes executing an algorithm / model (in this case, the PTX detection model).

[0188] Fig. 13C An example drop-down menu 1304 is provided that may be generated in response to selecting the "Add" button of the input variable field 1302. Fig. 13C , a drop-down menu 1304 provides for setting the custom input variable requirements of the algorithm / model. After the required data fields have been filled, selecting the "Save" option from the custom task creation UI 1311 can cause the custom task to be added to the task registry (e.g., workflow and task registry data 122) and return the user to the task registry UI 1301, as shown. Fig.13D As shown in Fig.13D, a notification indicating that a task has been successfully created / added may be presented. In this example, the newly added PTX detection task 1306 is now displayed as the first entry in the task registration table.

[0189] Reference again Figure 5B , the system / device management component 554 can provide management functions with respect to managing systems and devices (e.g., image provider systems / devices 102, clinician devices 302, medical image storage systems 306, etc.) that can access the algorithm orchestration component 110 and the services (e.g., workflows and algorithms) it is authorized to employ. In this regard, the algorithm orchestration component can receive image processing requests 106 from multiple systems / devices (e.g., multiple instances of a PAC and / or healthcare system can send image processing requests to the algorithm orchestration component 110). As previously noted, the security component 508 and / or access component 504 can limit the receipt and acceptance of requests to only authorized and authenticated devices / systems that have been added to the system / device registry data 126. In this regard, the disclosed system (e.g., system 100, system 300, etc.) can provide a large (e.g., global) interconnected network for receiving input data and sending output data of a workflow, thereby enabling the execution of a fully integrated workflow in a clinical environment.

[0190] In the illustrated embodiment, the system / device management component 554 may include a device add / remove component 556 that may provide for adding and removing systems and devices from the system / device registry data 126. The device rule component 558 may also provide for defining customized rules and restrictions for specific systems and devices regarding what types of requests (e.g., regarding image types and image study sizes) the algorithm orchestration component will accept, as well as rules and restrictions for when and how the algorithm orchestration component 110 will fulfill the request. For example, the device rule component 558 may be used to add rules regarding device / system priority for fulfilling requests, rules regarding workflows and algorithms accessible to different systems / devices, rules regarding the timing of providing request results, etc. In this regard, the device rule component 508 may be used to customize the provision of services to different imaging providers. The device query component 560 may provide a search query tool to facilitate the discovery of registered systems and devices for reviewing their service activities and customizing services.

[0191] FIG. 14A to FIG. 14BAn example device management UI is presented in accordance with one or more embodiments of the disclosed subject matter. FIG. A presents an example device management UI 1401 that may be presented in response to selecting the Devices tab from the upper menu bar. The device management UI 1401 provides a searchable and filterable list of registered devices / systems from the algorithm management component 110 that may receive image processing requests. The list view identifies the device by name, indicates when its information was last updated, indicates the imaging type, host ID, and port ID. The device management UI 1401 also provides for adding and removing devices.

[0192] Fig. 14B An example device management UI 1411 is presented for adding a new device that may be generated in response to selecting an "add device" button from the device management UI 1401. In the illustrated embodiment, the only device type accepted is a DICOM device (e.g., a DICOM device that provides DICOM formatted image data). In order to add a device, the device name, AE title, host ID, and port ID are required. The device management UI 1411 also provides options for customizing device permissions associated with adding a device.

[0193] Reference again Figure 5B , activity management component 562 can provide management functions related to the management of the activities of the algorithm orchestration system (e.g., system 100, system 300, etc.), including activity information about received image processing requests and workflows executed thereon. In the illustrated embodiment, activity management component 562 can include activity logging component 564, audit component 566, activity analysis component 568, and dashboard reporting component 570.

[0194] The activity record component 564 can track and record the activities of the algorithm orchestration system (e.g., the algorithm orchestration system 100, the system 300, etc.). Specifically, the activity record component 564 can track the activity information about all imaging processing requests 106 received and processed, including the system / device receiving the request, the time / date when the request was received, the image data associated with the request (e.g., the attributes of the image or the image to be processed), the executed workflow, the executed algorithm (e.g., the algorithm included in the corresponding workflow that was successfully executed), the workflow results provided (e.g., the output of one or more algorithms), and the timing of providing the results. The activity record component 564 can also track and mark detailed activity information about each executed (or attempted to be executed but failed) workflow. In this regard, the activity record component 564 can track information about the timing of the initiation and completion of the workflow and detailed information about the execution of each workflow node. For example, the activity logging component 564 can track the execution timing and status of each workflow task, sub-workflow, and function, including information about whether the task, sub-workflow, and / or function was completed and executed successfully (e.g., status=completed) or unsuccessfully (e.g., status=failed).

[0195] The activity logging component 564 may also track or log any workflow execution errors detected within the workflow. In this regard, in addition to identifying execution errors and providing execution error information in association with the workflow simulation, the workflow execution engine 116 may also be configured to identify execution errors generated in association with executing the workflow for the image processing request 106. The workflow execution engine 116 may also generate an error message that identifies the execution error that occurred, the workflow node / task where the error occurred, and the cause of the error (e.g., an error code that identifies or indicates the cause of the error).

[0196] The activity log component 564 can also track and record information about algorithms and workflow management. For example, the activity log component 564 can track and record information about loaded, updated, and removed algorithms. The activity log component 564 can track and record information about workflow creation, activation / deactivation, updates, and removals.

[0197] The activity management component 562 may also provide administrators, via the algorithm orchestration management application, access to activity information tracked and recorded by the activity logging component 564. In some embodiments, the activity logging component 564 may generate an activity log for each received image processing request 106, which may be reviewed / viewed by an administrator to facilitate monitoring and auditing of the system. The activity log for image processing requests may include any of the activity information discussed above. For example, the activity log for image processing requests may identify the image processing request (e.g., an ID number assigned to the request via the algorithm orchestration component 112, etc.), the time / date when the request was received, the system / device from which the request was received, the workflow executed, the status of the workflow, any execution errors detected, etc. The audit component 566 may also provide an audit function using the activity log, which is used to perform root cause analysis of workflow execution errors encountered with respect to one or more of the workflows.

[0198] For example, FIG. 15A to FIG. 15B An example activity log UI is presented in accordance with one or more implementations of the disclosed subject matter. Fig.15A An example activity log UI 1501 is provided that can be generated in response to selecting the activity log tab from the upper toolbar. The activity log UI 1501 provides a searchable / filterable list view of the activity log of the image processing request 106 fulfilled by the algorithm orchestration component 110. For example, in the illustrated embodiment, a device filter 1502 is provided, and the source (e.g., the imaging provider system / device from which the request is received) can submit the request via the device filter. A search tool 1504 is also provided, and the request can be searched based on various criteria (e.g., modality, workflow type, specific workflow, algorithm type, etc.) via the search tool. The activity log identifies the execution ID (e.g., request ID) of the request, the date / time when the request was received, the modality of the imaging study (or image) processed, the access number, the study instance unified ID number (UID), the executed workflow, and the status of the request satisfaction, which can be completed, in progress, or failed. In various embodiments, if it is not determined that the workflow matches the request and / or if the workflow is not successfully completed, then the request can be assigned a "failed" status. In this example, all four requests presented are completed. You can select the workflow listed for each activity log to view detailed activity information for the executed workflow and audit workflow activity.

[0199] Fig. 15BProvide an example workflow audit UI 1511 that can be generated in response to selecting a workflow 1506 for the first image processing request in the activity log UI 1501. The workflow audit UI 1511 provides detailed information of the workflow, including the workflow name, workflow type (e.g., it can be a workflow or a sub-workflow), workflow status, workflow start time, workflow duration, and workflow version. The state of the workflow in this example is "failed" because at least one task (e.g., at least one node) in the workflow fails or is not properly executed otherwise. In this context, the state of a workflow or task does not refer to the inference / output generated by the associated algorithm, but refers to whether the workflow or task is executed or run as configured. In this regard, the workflow audit tool provided by the workflow audit UI 1511 and the audit component 566 is designed to be used by system technicians, engineers, and administrators to discover and correct technical problems in the system.

[0200] Workflow audit UI 1511 can provide a detailed interactive list view of different tasks included in the workflow, including task start time, task duration, task name, task type, input data, output data, and task status. In this context, "task" refers to any node included in the workflow, not just the task node. In order to facilitate the root cause analysis audit, tasks can be selected to view additional information about each task to determine why the task failed. The input and output data associated with each task can also be selected and viewed via workflow audit UI 1511. In this example, the sub-workflow task 1508 failed, and the "combined" task failed. In one or more embodiments, the selection of sub-workflow task 1508 can cause another workflow audit UI to be generated for the sub-workflow. The subsequent workflow audit UI of the sub-workflow can provide the same type of information as the workflow audit UI 1511, but includes information about tasks executed for the sub-workflow rather than the parent workflow. In this regard, the sub-workflow can also be studied to determine which specific task or tasks executed therein failed to cause the sub-workflow to fail. The workflow audit UI 1511 may also provide access to a simulator tool for running the workflow in test mode. For example, in the illustrated embodiment, the workflow audit UI 1511 includes a "Run Test" button 1510 that can be selected to run the failed workflow in test mode. In some embodiments, in response to selecting the "Run Test" button 1510, the interface component may present a workflow diagram for the workflow within the workflow creation UI and display a simulation thereof (e.g., Fig. 12A and 12B ).

[0201] Reference again Figure 5B, the activity management component 562 may also include an activity analysis component 568 and a dashboard reporting component 570 to help evaluate system activity and performance, and to provide an overview view of the system's activity and performance via the dashboard UI 514. In this regard, in some embodiments, the activity analysis component 568 may use various performance evaluation algorithm metrics to analyze the activity information recorded / tracked by the activity recording component 564 to track the performance / activity of the system. For example, the activity analysis component 568 may determine / track the number of requests received and processed for a specific time period (e.g., daily, weekly, etc.) based on imaging providers, based on modalities, based on image types, and other trackable criteria. In another example, the activity analysis component 568 may determine / track information about different inference models executed, such as the amount and / or frequency of executing corresponding available models. In another example, the activity analysis component 568 may determine / track information about workflow execution errors (e.g., the frequency of workflow failures, the reasons for workflow failures, etc.). In another example, the activity analysis component 568 can determine / track information about request processing timing, including throughput (e.g., the number of requests processed per hour, day, etc.), processing duration, processing flow and feedback (e.g., request processing wait time), etc.

[0202] The activity analysis component 568 can and / or the dashboard reporting component 570 can also generate a performance report including system activity and performance information presented via the activity dashboard UI. In some embodiments, the activity analysis component 568 can also evaluate the feedback received via the feedback component 515 about the accuracy of model performance, and present this information in the activity dashboard. The activity dashboard UI can also provide the results from the AI ​​model analysis relative to the actual radiology analysis of the same or similar images received via the feedback component 515. This feedback allows the physician to measure his own image analysis against potential model analysis and in view of other physicians analyzing similar images. This includes showing the number of available studies used to generate AI model analysis and false positive rate and true positive rate analysis. Compliance with the desired diagnostic criteria can be monitored in the activity dashboard, where the actual diagnosis of a given radiologist is compared with the AI ​​model output and / or other physician decisions and corresponding model decisions.

[0203] FIG. 16A to FIG. 16B An example activity dashboard UI is presented in accordance with one or more embodiments of the disclosed subject matter. Fig.16AAn example active dashboard UI 1601 is shown that provides algorithm statistics and comparative feedback with reference to radiologists' predictions and studies according to one or more examples of the disclosed subject matter. The active dashboard UI 1601 includes an overview portion 1602, which includes information about the number and percentage of applications of some different models provided by the system, including a CT hemorrhage model, a CTALVO model, and a CT schematic stroke model in this example. The overview portion 1602 also identifies the total number of studies for processing (e.g., each study corresponds to an image processing request), the total number of patients represented, the total number of sites (e.g., imaging providers) receiving the studies, and the total number of devices. The active dashboard also summarizes the studies for each diagnosis in portion 1604. In addition, the active dashboard UI also includes a model performance evaluation portion 1608, which can present information about the performance of the selected model. In this example, the model performance information compares the artificial intelligence model predictions of medical diagnostic conditions with the predictions of two selected radiologists. The active dashboard also includes a receiver operating characteristic (ROC) curve graph 1606, which summarizes the false positive diagnosis rate and the true positive diagnosis rate of all diagnostic models.

[0204] Fig. 16B Another example activity dashboard UI 1611 that can be generated and presented via the user interface component 1510 and / or the dashboard report component 570 is presented. In this example, the activity dashboard UI 1611 presents graphical information 1610 about the number of studies received for processing per modality per selected time frame (e.g., the last 30 days in this example). The activity dashboard UI 6111 also includes a model overview portion 1612 that includes information about models run in the past 30 days. In this example, the model overview portion 1612 identifies the corresponding model by name, identifies the provider, the total number of model runs and completions, the number of positive results (assuming these are diagnostic models), the number of negative results (assuming these are diagnostic models), and the number of execution failures.

[0205] Fig.17 An example algorithm orchestration system architecture 1700 is presented in accordance with one or more implementations of the disclosed subject matter. System architecture 1700 provides an example architecture for implementing algorithm orchestration system 100 and / or algorithm orchestration system 300 in accordance with one or more examples of the disclosed subject matter. For the sake of brevity, repeated descriptions of similar elements are omitted.

[0206] In this example implementation, the architecture 1700 includes three layers (data layer, service layer, and client layer) to perform various orchestration components described herein, including workflow development and processing. The data layer stores data from the service layer, including data provided by one or more AO databases 120, file sharing data sources 132; and includes a scheme for transferring files between the service layer and the data layer. The service layer includes an orchestration director component 112, a workflow execution engine 116, and an algorithm execution engine 118, and provides execution of various workflows and algorithms provided in the data layer, including a service layer for executing algorithm orchestration and various algorithms described herein. The service layer also includes a DICOM routing component 1702, which provides for transferring DICOM image files between direct attached storage (DAS) 1712 elements provided by the PACS 1714 and the workflow execution engine 116 and the algorithm execution engine 118. The DAS 1712 stores medical images and metadata. The client layer includes an administrator user interface (UI) 1710 for developing and executing workflows according to an external PACS system 1714, which includes an administrator desktop 1708 (e.g., corresponding to administrator device 104) that interacts with an administrator UI 1706 via an API gateway 1704 that provides an interface between the client layer and the service layer. The PACS system 714 also includes a study processing component 1710 for retrieving medical study data from the DAS 1712 for medical imaging analysis.

[0207] The API gateway 1704 interfaces with the orchestration director component 112, which orchestrates the execution of the workflow execution engine 116 and the algorithm execution engine 118 and the various services provided by the algorithm orchestration component 110. The workflow execution engine 116 can communicate with the DICOM router 1702, which transfers images and metadata from the DAS 1712. The workflow execution engine 116 and the algorithm execution engine 118 can enable the AO database 120 to communicate with the file sharing data source 132 to execute the selected algorithm according to the workflow instructions. In various embodiments, the data layer can operate as a structured query language (SQL) database and includes an orchestration solution for interfacing with the orchestration service and an engine solution for interfacing with the orchestration director component 112. Various interface signals couple the components depicted in the system. These signals include queue signals, representational state transfer (REST) ​​signals, and hypertext transfer protocol (HTTP) signals as well as DICOM signals.

[0208] In an implementation example, when the PACS 1714 has a new study to be processed through orchestration, this will send an HTTP request to the REST API exposed by the API Gateway 1704, which is called a "study process notification" containing study metadata in the payload. The API Gateway 1704 will forward the request to the orchestration director component 112, which can validate the request payload and respond with an execution ID (e.g., request ID) and status. The orchestration director component 112 can direct the workflow external engine to call the available / applicable workflows included in the workflow and task registry data 122. The corresponding workflows can be executed as separate threads. Some workflows will start by validating the DICOM metadata to see if they match the workflow conditions (e.g., modality, view position, study description, etc.), and in the case of a match, the study data can be transferred from the PACS 1714 to the internal file storage device 1716. When the transfer is complete, the algorithm execution engine 118 can execute the algorithm defined in the workflow. For each algorithm to be executed, the workflow execution engine 116 calls the algorithm execution engine 118 and waits for a response. When the response of the selected algorithm is available, the orchestration director component 112 can transfer the output files generated by the algorithm back to the PACS 1714 and send a notification message indicating that the processing of the study is complete. This notification message should also include a list of algorithms executed by the workflow execution engine 116 and the execution results of the selected corresponding algorithm.

[0209] FIG. 18A to FIG. 18B Another example algorithm orchestration system architecture 1800 is presented in accordance with one or more implementations of the disclosed subject matter. System architecture 1800 provides another example architecture for implementing algorithm orchestration system 100 and / or algorithm orchestration system 300 in accordance with one or more examples of the disclosed subject matter. For the sake of brevity, repeated descriptions of similar elements are omitted.

[0210] FIG. 18A to FIG. 18BAn example system architecture 1800 from front-end to back-end is presented for a user to access the system to initiate a workflow execution. In this example implementation, the system architecture 1800 includes four layers, a client layer 1802, an algorithm orchestration (AO) back-end layer 1810, a DICOM service layer 1812, and a workflow orchestrator layer 1820. The client layer 1801 may include an administrator UI 1706, a web service 1802 configured to interface with an API gateway (e.g., a PACS viewer with orchestration services), a data distribution service (DDS) server 1804, a storage SCU 1806, and a mobile SCP 1808. Depending on which APIs are supported by the client system / device, there are different ways to start the same workflow execution. In one implementation, a client system / device (e.g., a web service client 1802) can send a web service call (e.g., an HTTP call) through the API gateway 1704 to provide an image processing request. Administrators can also access the AO backend layer 1810 and the algorithm orchestrator layer 1820 via the administrator UI 1706 and the API gateway 1704 .

[0211] The system architecture 1800 also supports access to the AO backend layer 1810 and the orchestration layer 1820 via the DICOM service layer 1812. In this example, the DICOM service layer 1812 may include: a DICOM database 1814 including medical images to be processed via the workflow, a DICOM routing component 1702, a DICOM service class provider (SCP) 1816, and a DICOM service class user (SCU). This channel can be used by any DICOM compliant system to push DICOM images / studies to the AO backend layer 1810 and the workflow orchestrator layer 1820 to start the same workflow logic.

[0212] With this example implementation, workflow execution begins at the workflow orchestrator layer 1820. When a client system / device calls the orchestrator through the web service 1802, the channel passes through the AO backend layer 1810, which then invokes the algorithm execution engine 118, the workflow execution engine 116, and the HTTP task execution 1822 as orchestrated / performed by the orchestration director component. In various implementations, the HTTP task execution engine 1822 can be or correspond to a version of the algorithm execution engine 118 that is configured to manipulate the execution of asynchronous HTTP workflow tasks and jobs defined in the file sharing data source 132, including the execution of third-party inference tasks 1824 involving invoking and running third-party inference models (e.g., third-party algorithms 136), internal inference tasks 1826 involving invoking and running internal inference models (e.g., internal algorithms 134), and job execution tasks 1828. When a client system / device submits an image processing request 106 via the DICOM API, the (backend) DICOM service layer 1812 will send a notification to the DICOM routing component 1702 , which in turn will call the AO backend layer 1810 and ultimately access the workflow orchestrator layer 1820 .

[0213] Various operation commands / signals describing the interaction between corresponding system components are also provided. These operation commands / signals include: manage, status (notify), execute, move, store, notify, start execution, run workflow, run, discover and read / write.

[0214] Fig.19A block diagram of a workflow program 1900 and computational stages for analyzing medical images and metadata is shown in accordance with one or more embodiments of the disclosed subject matter. In this example implementation, before the image / study is delivered to a first storage bucket 1910, the image / study 1902 and associated metadata are delivered from a medical image storage system 306 (e.g., a PACS, a VNA, etc.) to a healthcare interest group (HCIG) database and processor for pre-processing and formatting. In this regard, at 1904, the PACS may automatically route the image / study to the HCIG database. At 1908, the HCIG database transfers the image / study to the first storage bucket 1910 and notifies the algorithm orchestration component 110. At 1912, the algorithm orchestration component opens an image processing request 1914 for the image / study and assigns it to an execution ID. At 1916, the algorithm orchestration component directs the workflow execution engine 116 to execute the associated workflow for the image processing request and provides the result 1920 to a second storage bucket 1922. The results 1920 from the storage bucket can be viewed using the image viewer application 304, which can also be sent back as result feedback to the medical image storage system 306 (i.e., PACS). In this regard, at 1924, the second storage bucket sends the results to the viewer application and returns to the PACS.

[0215] Fig. 20 2000 and the interaction of the algorithm orchestration component with other components of the workflow according to one or more embodiments of the disclosed subject matter. At 2010, study metadata is retrieved from the PACS (e.g., by the orchestration director component 112), and a decision is performed at 2012 to determine whether the study matches a first workflow / algorithm criterion (e.g., whether the modality is MR / CR / DX). If false at 2012, the procedure 2000 proceeds to 2014, where a customized summary message is generated that is provided back to the other PACS 216. If true at 2012, the procedure proceeds to 2020. At 2020, the procedure performs another decision to determine whether the image meets a second decision criterion (e.g., to see if the location is AP / PA). If false at 2020, the procedure proceeds to 2024 and generates an error message, and then proceeds to 2028 and generates a summary message. If true at 2020, the program proceeds to 2030 and 2034 in parallel. At 2030, the program performs a move operation and then infers the image at 2036. At 2034, a decision is made about the patient's age. If not greater than 18 at 2034, the program proceeds to 2024 and generates an error message. If greater than 18 at 2034, the program proceeds to 2030.

[0216] After the inference at 2036, the program proceeds to 2040 where it is determined whether a given medical problem has been identified. If yes at 2040, the program proceeds to 2044 and performs a move operation followed by a store operation at 2046 and then a store overwrite operation at 2050 which ends with a PACS move at 2054. If no medical problem has been identified at 2040, the program proceeds to 2060 where a quality metric is run to determine the quality of the medical decision determined by the inference engine at 2036. If the quality is adequate, the program proceeds to 2064 and generates a warning message before proceeding to 2028. If the quality metric is not adequate, the program proceeds to 2028 and generates a summary message indicating that a problem could not be detected.

[0217] Fig.21 2100 and the interaction of the algorithm management component with other components of the program according to one or more embodiments of the disclosed subject matter. At 2110, a PACS acquisition for a medical image (or images) is initiated, wherein a determination is performed at 2114 whether multiple images (e.g., studies / series) are involved. If a single image is acquired at 2114, the program proceeds to 2120 to create an image uploader identifier. If multiple images are detected at 2114, the images are grouped into a single upload identifier at 2124. After creating the identifiers at 2120 and 2124, respectively, the program proceeds to an inference AI service corresponding to using a workflow execution engine 116 and an algorithm execution engine 118 to execute one or more AI models on image data according to one or more applicable workflows. The output from the AI ​​service 2130 is fed to an overview message generator at 2134 and processed by a decision at 2140. At 2140, a determination is made whether to check the model run on each AI image. If no at 2130, the program proceeds to 2144 and creates a summary including failure and warning messages, followed by a PACS update at 2150. If yes at 2140, the program proceeds to 2154 where it is determined whether all images have been processed. If no at 2154, the program proceeds to 2144. If yes at 2154, the program proceeds to 2160 and performs a store to the PACS of the image. The output of 2160 is fed to 2164 where a PACS move is performed. As shown, the PACS administrator has control over components 2150 and 2164, respectively.

[0218] Fig. 22An example of a workflow implementation 2200 for analyzing a chest x-ray for a pneumothorax (PTX) condition according to one or more embodiments of the disclosed subject matter is shown. At 2202, the workflow begins. At 2204, a decision is made to determine whether a study description of a chest or chest image will be analyzed. At 2206, a decision is made to determine whether the patient age is greater than or equal to 18. If not, then the workflow 2200 proceeds to 2208, where a notification is generated indicating that the patient is not old enough for analysis by the workflow 2200. After 2208, the program 2200 proceeds to 2210 and generates an overview notification, and then ends at 2212. If the decision is true at 2206, then the program 2200 proceeds to 2220, where a decision is made regarding the type of modality in which the image was captured (e.g., DX, CR). At 2224, the program 2200 generates a notification that the workflow is in progress. At 2228, a sub-workflow is executed to determine the modality, and a check is performed to determine whether the CR modality was determined at 2220. At 2230, a decision is performed to determine whether the viewing location of the image is an AP or a PA. If either is false, the program proceeds to 2234 and generates a notification that no AI model is available.

[0219] If the decision is true at 2230, the program 2200 proceeds to 2236 and executes a sub-workflow to check the modality and VP. At 2240, a decision is made to see if the position is Ap or PA. If true, the program proceeds to 2244 and a move is performed to transfer data. At 2246, the program 2200 proceeds to 2246 and determines if the move is complete. If false, the program 2200 proceeds to 2250, where a notification of a failed move is generated. If true at 2246, the program proceeds to 2254 and runs a chest frontal model. After the model execution at 2254, the program 2200 proceeds to 2256, where a decision is made if the model analysis run at 2254 exceeds a probability threshold. If false at 2256, the program 2200 proceeds to 2260 and reports that no AI is available. If true at 2256, then the fork instruction creates parallel model branches 2270 and 2272.

[0220] After executing the PTX model at 2270, the program 2200 proceeds to 2274, where a decision is made regarding the prediction made by the PTX model at 2270. If false (e.g., below a probability threshold), the program proceeds to 2210 and generates a summary notification of a false (non-positive) result. If true at 2274, the program continues to perform a storage operation and ends at a combination operation 2280. At branch 2272, the patient position model is executed in parallel with the branch at 2270. At 2282, it is determined that the patient position is greater than or equal to the patient position threshold. If true at 182, the branch ends at combination 2280. If false at 2282, a warning is added at 2284, and the branch then ends at combination 2280. The output from combination 2280 is passed to 2210, where a summary notification (e.g., a report) is generated.

[0221] Fig.23 A high-level flow chart of another example computer-implemented method 2300 for executing workflow instructions to analyze medical images and associated metadata is presented in accordance with one or more embodiments of the disclosed subject matter. At 2302, the method 2300 includes receiving, by a system (e.g., system 100, system 300, system 1700, system 1800, etc.) operably coupled to a processor, a plurality of diagnostic algorithms to analyze at least one medical image of a patient and metadata describing attributes of the at least one medical image (e.g., via the algorithm orchestration component 110). At 2304, the method 2300 includes executing, by the system (e.g., via the workflow execution engine 116), the workflow instructions describing a computer-executable diagnostic routine to be applied to the at least one medical image and the metadata. At 2306, the method 2300 includes applying, by the system in response to the workflow instructions, at least one diagnostic algorithm (e.g., via the algorithm execution engine 118) to perform a diagnostic routine based on the at least one medical image and the metadata. At 2308 , method 2300 includes generating, by the system in response to the workflow instructions, a diagnosis for the patient based on executing the selected at least one diagnostic algorithm (eg, where the algorithm result includes the diagnosis).

[0222] Fig.24 A high-level flow chart of an example computer-implemented method 2400 for workflow algorithm management is presented in accordance with one or more embodiments of the disclosed subject matter.

[0223] At 2402, a system (e.g., system 100, system 300, system 1700, system 1800, etc.) operably coupled to a processor provides an algorithm catalog (e.g., algorithm catalog data 124), the algorithm catalog including algorithm information identifying algorithms that can be used to process medical images, the algorithm information including algorithm execution instructions for executing the algorithms as a web service. At 2404, the system loads (e.g., via loading component 518) the algorithm information into the algorithm catalog in response to receiving the algorithm information via a loading user interface (e.g., algorithm loading UI 901) of an algorithm management application (e.g., an algorithm management application providing management tool 114), wherein based on including the algorithm information in the algorithm catalog, the algorithm is made available (e.g., by algorithm orchestration component 110) to be incorporated into a workflow (e.g., a workflow included in workflow and task registry data 122) for executing the algorithm on the medical image.

[0224] Fig.25 A high-level flow diagram of an example computer-implemented method 2500 for orchestrating an algorithm for performing pairwise medical image execution is presented in accordance with one or more embodiments of the disclosed subject matter.

[0225] At 2502, a system (e.g., system 100, system 300, system 1700, system 1800, etc.) operably coupled to a processor receives an image processing request from a medical imaging provider via a network (e.g., via the algorithm orchestration component 110). At 2504, the system identifies (e.g., via the orchestration director component 112 and / or the workflow execution engine 116) a workflow applicable to a medical image associated with the medical image processing request, the workflows each including one or more medical image inference models (e.g., one or more internal algorithms 134 and / or one or more third-party algorithms 136). At 2506, the system executes the workflow on the medical image in association with receiving the image processing request (e.g., using the workflow execution engine 116 and / or the algorithm execution engine 118).

[0226] Fig.26 A high-level flow chart of an example computer-implemented method 2600 for orchestrating medical image execution algorithms according to predefined workflows and managing workflows and model execution is presented in accordance with one or more embodiments of the disclosed subject matter.

[0227] At 2602, a system (e.g., system 100, system 300, system 1700, system 1800, etc.) operably coupled to a processor receives an image processing request from a medical imaging provider via a network (e.g., via the algorithm orchestration component 110). At 2604, the system identifies (e.g., via the orchestration director component 112 and / or the workflow execution engine 116) a workflow applicable to a medical image associated with the medical image processing request, the workflow comprising one or more medical image inference models (e.g., one or more internal algorithms 134 and / or one or more third-party algorithms 136), respectively. At 2606, the system executes the workflow on the medical image in association with receiving the image processing request (e.g., using the workflow execution engine 116 and / or the algorithm execution engine 118). At 2608, the system tracks activity information about the image processing request and the workflow executed for the image processing request (e.g., via the activity logging component 564). At 2610, the system provides access to the activity information via a management application 2610 (e.g., an algorithm management application providing the management tool 114).

[0228] One or more examples may be systems, methods and / or computer program products at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to implement one or more aspects of the present examples.

[0229] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following items: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove on which instructions are recorded) and any suitable combination of the foregoing items. As used herein, a computer readable storage medium should not be understood to be a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0230] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the corresponding computing / processing device.

[0231] The computer-readable program instructions for implementing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit system, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and process programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a physical computer, partially on a physical computer, executed as an independent software package, partially on a physical computer and partially on a remote computer, or executed entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the physical computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some examples, an electronic circuit system including, for example, a programmable logic circuit system, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions to personalize the electronic circuit system by utilizing the state information of the computer-readable program instructions, so as to perform various aspects of the present invention.

[0232] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to examples of the present invention. It is understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0233] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flow chart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to function in a particular manner, so that the computer-readable storage medium with instructions stored therein includes a product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flow chart and / or block diagram.

[0234] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operating steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0235] The flow chart and block diagram in the accompanying drawings illustrate the possible implementation architecture, function and operation of the system, method and computer program product according to various examples of the present invention. In this regard, each frame in the flow chart or block diagram can represent a module, a fragment or a part of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the function indicated in the frame may not occur in the order indicated in the figure. For example, depending on the function involved, two frames of continuous display can actually be performed substantially simultaneously, or the frames can sometimes be performed in reverse order. It will also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be implemented by a system based on dedicated hardware that performs a specified function or action or implements a combination of dedicated hardware and computer instructions.

[0236] Combination Fig. 27 , the systems and procedures described below may be embodied in hardware, such as a single integrated circuit (IC) chip, multiple ICs, application specific integrated circuits (ASICs), etc. In addition, the order in which some or all of the program blocks appear in each procedure should not be considered limiting. On the contrary, it should be understood that some program blocks can be executed in various orders, not all of which can be explicitly shown in this document.

[0237] refer to Fig. 27, an example environment 2700 for implementing various aspects of the claimed subject matter includes a computer 2702. The computer 2702 includes a processing unit 2704, a system memory 2706, a codec 2735, and a system bus 2708. The system bus 2708 couples system components including, but not limited to, the system memory 2706 to the processing unit 2704. The processing unit 2704 can be any of a variety of available processors. Dual microprocessors and other multi-processor architectures can also be used as the processing unit 2704.

[0238] The system bus 2708 can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, or a local bus using any of a variety of available bus architectures including, but not limited to, Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), card bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association (PCMCIA), FireWire (IEEE 13154), and Small Computer System Interface (SCSI).

[0239] In various examples, the system memory 2706 includes a volatile memory 2710 and a non-volatile memory 2712 that may employ one or more of the disclosed memory architectures. A basic input / output system (BIOS) (containing basic routines such as transferring information between elements within the computer 2702 during startup) is stored in the non-volatile memory 2712. In addition, according to the present innovation, the codec 2735 may include at least one of an encoder or a decoder, wherein at least one of the encoder or the decoder may be composed of hardware, software, or a combination of hardware and software. Although the codec 2735 is depicted as a separate component, the codec 2735 may be contained within the non-volatile memory 2712. By way of illustration and not limitation, the non-volatile memory 2712 may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a 3D flash memory, or a resistive memory, such as a resistive random access memory (RRAM). In at least some examples, the non-volatile memory 2712 may employ one or more of the disclosed memory devices. In addition, the non-volatile memory 2712 can be a computer memory (e.g., physically integrated with the computer 2702 or its motherboard) or a removable memory. Examples of suitable removable memory with which the disclosed examples can be implemented can include a secure digital (SD) card, a compact flash (CF) card, a universal serial bus (USB) memory stick, and the like. The volatile memory 2710 includes a random access memory (RAM) that acts as an external cache memory, and one or more of the disclosed memory devices can also be employed in various examples. By way of illustration and not limitation, RAM can be provided in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (ESDRAM), and the like.

[0240] The computer 2702 may also include removable / non-removable, volatile / non-volatile computer storage media. Fig. 27For example, a disk storage device 2714 is shown. The disk storage device 2714 includes, but is not limited to, devices such as a disk drive, a solid state disk (SSD), a flash memory card, or a memory stick. In addition, the disk storage device 2714 may include storage media alone or in combination with other storage media, and the other storage media may include, but is not limited to, an optical drive, such as a CD-ROM device (CD-ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), or a digital versatile disk ROM drive (DVD-ROM). In order to facilitate connecting the disk storage device 2714 to the system bus 2708, a removable or non-removable interface, such as an interface 2716, is generally used. It should be understood that the disk storage device 2714 can store information related to the entity. Such information can be stored at a server or provided to an application running on a server or entity device. In one example, the entity may be notified (e.g., by means of an output device 2736) of the type of information stored in the disk storage device 2714 or transmitted to the server or application. An entity may be provided the opportunity to opt-in or opt-out of having such information collected or shared with the server or application (eg, via input from input device 2728).

[0241] It should be understood that Fig. 27 The software that acts as an intermediary between entities and the basic computer resources described in the appropriate operating environment 2700 is described. This software includes an operating system 2718. The operating system 2718, which may be stored on the disk storage device 2714, is used to control and allocate the resources of the computer system 2702. Application programs 2720 utilize the management of resources by the operating system 2718 through program modules 2724, and program data 2726 stored in the system memory 2706 or on the disk storage device 2714, such as a startup / shutdown transaction table. It should be understood that the claimed subject matter can be implemented with various operating systems or combinations of operating systems.

[0242] Entities enter commands or information into the computer 2702 through input devices 2728. Input devices 2728 include, but are not limited to, pointing devices such as mice, trackballs, styluses, touchpads, keyboards, microphones, joysticks, gamepads, satellite dishes, scanners, TV tuner cards, digital cameras, digital video cameras, webcams, etc. These and other input devices are connected to the processing unit 2704 through the system bus 2708 via interface ports 2730. Interface ports 2730 include, for example, serial ports, parallel ports, game ports, and universal serial buses (USB). Output devices 2736 use some of the same types of ports as input devices 2728. Thus, for example, a USB port can be used to provide input to the computer 2702 and output information from the computer 2702 to output devices 2736. Output adapters 2734 are provided to illustrate that there are some output devices 2736 such as monitors, speakers, and printers, as well as other output devices 2736 that require special adapters. By way of illustration and not limitation, output adapters 2734 include video and sound cards that provide a means of connection between output devices 2736 and system bus 2708. Note that other devices or systems of devices provide both input and output capabilities, such as remote computer 2738.

[0243] Computer 2702 can operate in a networked environment using a logical connection to one or more remote computers, such as remote computer 2738. Remote computer 2738 can be a personal computer, server, router, network PC, workstation, microprocessor-based device, peer device, smart phone, tablet computer or other network node, and typically includes many elements described with respect to computer 2702. For the purpose of simplicity, only memory storage device 2740 is shown for remote computer 2738. Remote computer 2738 is logically connected to computer 2702 through network interface 2742, and then connected via communication connection 2744. Network interface 2742 covers wired or wireless communication networks, such as local area networks (LANs) and wide area networks (WANs) and cellular networks. LAN technologies include fiber distributed data interface (FDDI), copper distributed data interface (CDDI), Ethernet, token ring, etc. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks such as integrated services digital networks (ISDN) and variants thereon, packet switching networks, and digital subscriber lines (DSL).

[0244] The communication connection 2744 refers to the hardware / software used to connect the network interface 2742 to the bus 2708. Although the communication connection 2744 is shown internal to the computer 2702 for clarity of illustration, the communication connection can also be external to the computer 2702. For example purposes only, the hardware / software required to connect to the network interface 2742 includes internal and external technologies, such as modems, including conventional telephone grade modems, cable modems and DSL modems, ISDN adapters, as well as wired and wireless Ethernet cards, hubs and routers.

[0245] The illustrated aspects of the present disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0246] refer to Fig.28 , a schematic block diagram of a computing environment 2800 according to the present disclosure is shown here, in which a subject system (e.g., system 100, system 300, system 1700, system 1800, etc.), method, and computer-readable media may be deployed. The computing environment 2800 includes one or more clients 2802 (e.g., laptop computers, smart phones, PDAs, media players, computers, portable electronic devices, tablet computers, etc.). The client 2802 may be hardware and / or software (e.g., threads, programs, computing devices). The computing environment 2800 also includes one or more servers 2804. The server 2804 may also be hardware or hardware combined with software (e.g., threads, programs, computing devices). For example, the server 2804 may accommodate threads to perform transformations by adopting various aspects of the present disclosure. In various examples, one or more components, devices, systems, or subsystems of the system 1800, system 2200, and system 2400 may be deployed as hardware and / or software at the client 2802, and / or deployed as hardware and / or software deployed at the server 2804. One possible communication between the client 2802 and the server 2804 may be in the form of data packets transmitted between two or more computer programs, where the data packets may include healthcare-related data, training data, AI models, input data for the AI ​​models, etc. For example, the data packets may include metadata, such as associated context information. The computing environment 2800 includes a communication framework 2806 (e.g., a global communication network such as the Internet, or a mobile network) that can be used to facilitate communication between the client 2802 and the server 2804.

[0247] Communication may be facilitated by wired (including fiber optic) and / or wireless technologies. Client 2802 includes or is operably connected to one or more client data stores 2808 that can be used to store information local to client 2802 (e.g., associated context information). Similarly, server 2804 may include or is operably connected to one or more server data stores 2810 that can be used to store information local to server 2804.

[0248] In one example, client 2802 can transfer an encoded file to server 2804 in accordance with the disclosed subject matter. Server 2804 can store the file, decode the file, or transmit the file to another client 2802. It should be appreciated that client 2802 can also transfer an uncompressed file to server 2804, and server 2804 can compress the file in accordance with the disclosed subject matter. Likewise, server 2804 can encode video information and transmit the information to one or more clients 2802 via communication framework 2806.

[0249] Fig.29 A schematic block diagram of another example computing environment 2900 according to the present disclosure is shown, in which a subject system (e.g., system 100, system 300, system 1700, system 1800, etc.), method, and computer-readable media may be deployed. The computing environment 2900 includes a cloud deployment architecture consisting of one or more clients 2902, which may be communicatively coupled to a system cloud 2904 via a network (e.g., the Internet). The system cloud 2904 may include a cloud load balancer, one or more application containers, one or more cloud service containers, a cloud data store, and a cloud network that communicatively couples one or more cloud components to the cloud data store. According to the cloud deployment architecture, the client 2902 may include one or more client devices (e.g., mobile devices, laptop computers, desktop computers, etc.), which may include or employ suitable applications (e.g., native mobile applications, web-based applications, thin / fat client applications, etc.) to access and employ one or more features and functions of a subject native / reconstructed medical imaging system (e.g., system 100, system 300, system 1700, system 1800, etc.) deployed in the system cloud 2904. In various implementations, one or more components of system 100 , system 300 , system 1700 , and / or system 1800 may be distributed between client 2902 and system cloud 2904 .

[0250] Fig.30A schematic block diagram of another example computing environment 3000 according to the present disclosure is shown, in which a subject system (e.g., system 100, system 300, system 1700, system 1800, etc.), method, and computer-readable media may be deployed. The computing environment 3000 includes a virtualized enterprise deployment consisting of one or more clients 2902, which may be communicatively coupled to a remote data center 3002 via a network (e.g., the Internet). The remote data center 3002 may include an application server subnet 3004, which may provide a load balancer, one or more application containers, one or more virtualized servers, and one or more rack servers. The data center 3002 may also include one or more data stores, which may be communicatively coupled to the application server subnet 3004 via a data center network. According to the virtualized enterprise deployment, the client 2902 may include one or more client devices (e.g., mobile devices, laptop computers, desktop computers, etc.) that may include or employ suitable applications (e.g., native mobile applications, web-based applications, thin / fat client applications, etc.) to access and employ one or more features and functions of the subject native / reconstructed medical imaging system (e.g., system 100, system 300, system 1700, system 1800, etc.) deployed in the data center 3002 and the application server subnet 3004. In various implementations, one or more components of the system 100, system 300, system 1700, system 1800 may be distributed between the client 2902 and the application server subnet 3004 and the data center 3002.

[0251] Fig.31A schematic block diagram of another example computing environment 3100 according to the present disclosure is shown, in which a subject system (e.g., system 100, system 300, system 1700, system 1800, etc.), method, and computer-readable media may be deployed. The computing environment 3100 includes a local enterprise deployment consisting of one or more clients 2902, which may be communicatively coupled to an application server subnet 3104 via a network (e.g., the Internet). According to this example, an application server subnet 3104 may be provided at an enterprise premises 3102 (e.g., as opposed to a remote data center 3102). The application server subnet 3104 may include a load balancer, one or more application containers, and one or more servers. The application server subnet 3104 may be communicatively coupled to one or more data stores provided at the enterprise premises 3102 via an enterprise network. Similar to cloud and virtualized enterprise deployments, the client 2902 may include one or more client devices (e.g., mobile devices, laptops, desktop computers, etc.) that may include or employ suitable applications (e.g., native mobile applications, web-based applications, thin / fat client applications, etc.) to access and employ one or more features and functions of the subject native / reconstructed medical imaging system (e.g., system 100, system 300, system 1700, system 1800, etc.) deployed at the enterprise premises 3102 and in the application server subnet 3104. In various implementations, one or more components of the system 100, system 300, system 1700, system 1800 may be distributed between the client 2902 and the application server subnet 3104 and the enterprise premises 3102.

[0252] Fig.32 A schematic block diagram of another example computing environment 3200 according to the present disclosure is shown, in which the subject systems (e.g., system 100, system 300, system 1700, system 1800, etc.), methods, and computer-readable media may be deployed. The computing environment 3200 includes a local device deployment, in which all components of the systems 100, 300, 1700, and 1800 are disposed at a single client device 3202. With this implementation, the client device 3202 may include a web-based application that may be communicatively coupled to one or more application containers via loopback. The one or more application containers may be communicatively coupled to one or more databases and / or one or more local file systems via loopback.

[0253] Although the subject matter has been described above in the general context of computer executable instructions of a computer program product running on one and / or multiple computers, it will be appreciated by those skilled in the art that the present disclosure may also or may be implemented in combination with other program modules. In general, program modules include routines, programs, components, data structures, etc. that perform specific tasks and / or implement specific abstract data types. In addition, it will be appreciated by those skilled in the art that the computer-implemented method of the present invention may be practiced with other computer system configurations, including single-processor or multi-processor computer systems, small computing devices, large computers, and computers, handheld computing devices (e.g., PDAs, phones), microprocessor-based or programmable consumer or industrial electronic devices, etc. The illustrated aspects may also be practiced in a distributed computing environment, in which tasks are performed by a remote processing device linked by a communication network. However, some (if not all) aspects of the present disclosure may be practiced on a stand-alone computer. In a distributed computing environment, program modules may be located in local and remote memory storage devices.

[0254] As used in this application, the terms "component", "system", "subsystem", "platform", "layer", "gateway", "interface", "service", "application", "device", etc. may refer to and / or may include one or more computer-related entities or entities related to an operating machine having one or more specific functions. The entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a program running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, both an application running on a server and a server may be a component. One or more components may reside within a program and / or an execution thread, and a component may be located on one computer and / or distributed between two or more computers. In another example, the corresponding component may be executed according to various computer-readable media having various data structures stored thereon. The components may communicate via local and / or remote programs, such as according to signals having one or more data packets (e.g., data from a component, which interacts with another component in a local system, a distributed system, and / or a network (such as the Internet with other systems) via a signal). As another example, a component may be a device having a specific functionality provided by a mechanical part operated by an electrical or electronic circuit system operated by a software or firmware application executed by a processor. In this case, the processor may be internal or external to the device and may execute at least a portion of the software or firmware application. As yet another example, a component may be a device that provides a specific functionality through electronic components rather than mechanical parts, where the electronic components may include a processor or other device for executing software or firmware that at least partially imparts functionality to the electronic components. In one aspect, the component may simulate the electronic component, for example, via a virtual machine within a cloud computing system.

[0255] In addition, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any natural inclusive replacement. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing cases. In addition, unless otherwise specified or clear from the context that it is for a singular form, the articles "a" and "an" used in this specification and the drawings should generally be interpreted to mean "one or more". As used herein, the terms "example" and / or "exemplary" are used to mean serving as an example, instance, or illustration, and are intended to be non-limiting. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. In addition, any aspect or design described herein as "example" and / or "exemplary" is not necessarily to be interpreted as being preferred or advantageous over other aspects or designs, nor is it intended to exclude equivalent exemplary structures and techniques known to those of ordinary skill in the art.

[0256] As used in this specification, the term "processor" may refer to substantially any computational processing unit or device, including but not limited to a single-core processor; a single processor with software multithreaded execution capability; a multi-core processor; a multi-core processor with software multithreaded execution capability; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. In addition, a processor may refer to an integrated circuit, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), a discrete gate or transistor logic, a discrete hardware component, or any combination thereof designed to perform the functions described herein. In addition, the processor may utilize a nanoscale architecture (such as, but not limited to, transistors, switches, and gates based on molecules and quantum dots) in order to optimize space usage or enhance the performance of physical equipment. The processor may also be implemented as a combination of computational processing units. In the present disclosure, terms such as "storage", "storage device", "data storage", "data storage device", "database", and substantially any other information storage component related to the operation and function of the component are used to refer to a "memory component", an entity embodied in a "memory", or a component including a memory. It should be understood that the memory and / or memory components described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory. By way of illustration and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). For example, volatile memory may include RAM that may act as an external cache memory. By way of illustration and not limitation, RAM may be provided in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). In addition, the disclosed memory components of the system or computer-implemented method herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0257] What has been described above only includes examples of systems and computer-implemented methods. Of course, it is not possible to describe every conceivable combination of components or computer-implemented methods for the purpose of describing the present disclosure, but a person of ordinary skill in the art will recognize that many other combinations and permutations of the present disclosure are possible. In addition, to the extent that the terms "including", "having", "having", etc. are used in the specific embodiments, claims, appendices, and drawings, such terms are intended to be inclusive in a manner similar to the term "including", as interpreted when "including" is used as a transitional word in the claims. Descriptions of various examples have been presented for illustrative purposes, but the descriptions are not intended to be exhaustive or limited to the disclosed examples. Many modifications and variations are obvious to a person of ordinary skill in the art without departing from the scope and spirit of the described examples. The terms used herein are selected to best explain the principles of the examples, the practical applications or technical improvements that are superior to the technologies found in the market, or to enable other persons of ordinary skill in the art to understand the examples disclosed herein.

Claims

1. A system for algorithmic orchestration of a workflow for medical care imaging diagnosis, comprising: a memory storing computer executable components; and a processor that executes the computer executable components stored in the memory, wherein the computer executable components include: an algorithm catalog, the algorithm catalog comprising algorithm information identifying algorithms applicable to medical images, the algorithm information comprising algorithm execution instructions for executing the algorithms in association with invoking the algorithms at corresponding network-accessible file locations storing the algorithms; and a workflow creation component, the workflow creation component providing a workflow creation application, the workflow creation application being used to create a workflow, the system enabling the workflow to be applied to one or more of the medical images, the workflow comprising workflow information defining a set of computer-executable processing steps to be applied to the one or more of the medical images, wherein the workflow creation application includes a graphical user interface that facilitates creating a workflow using graphical nodes in association with receiving user input for dragging and dropping graphical nodes onto a workflow creation area of ​​the graphical user interface, connecting the graphical nodes using one or more connection lines to define relationships between the graphical nodes, and configuring the graphical nodes, Wherein, the graphic nodes include different types of graphic nodes corresponding to different types of computer-executable processing steps that can be used to process the medical image, respectively, and the different types of graphic nodes include task nodes representing tasks to be performed by the workflow, and the different types of graphic nodes also include nodes selected from the group consisting of decision nodes, sub-workflow nodes, waiting nodes, fork nodes, connection nodes and function nodes, and wherein, based on the selection of the task nodes, the workflow creation application provides a selectable list of algorithms contained in the algorithm catalog, and when the algorithm execution instructions corresponding to the algorithm selected from the selectable list are imported from the algorithm catalog into the workflow information, the selected algorithm is integrated into the workflow in an associated manner.

2. The system of claim 1, wherein the computer executable component further comprises: an algorithm orchestration component that receives an image processing request from a medical imaging provider via a network; and A workflow execution component executes the workflow on the medical image in association with receiving the image processing request and returns workflow results to the medical imaging provider, the workflow results including outputs of one or more of the algorithms.

3. The system of claim 2, wherein the image processing request is received from a device associated with the medical imaging provider, and wherein the computer executable component further comprises: A security component that restricts acceptance of imaging process requests for registered devices included in the device registration table information.

4. The system of claim 3, wherein the computer executable component further comprises: An algorithm execution engine that invokes the algorithm at its network accessible file location to receive the output associated with executing the workflow.

5. The system of claim 4, wherein the computer executable component further comprises: a loading component that loads the algorithm information into the algorithm catalog in response to receiving the algorithm information via a loading user interface of an algorithm management application, wherein the algorithm is enabled for incorporation into a workflow for executing the algorithm on the medical image based on including the algorithm information in the algorithm catalog; A conversion component, the conversion component converts the algorithm format from a first format incompatible with the algorithm execution engine to a second format compatible with the algorithm execution engine in association with the loading.

6. The system of claim 5, wherein the second format is compatible with an application program interface employed by the algorithm execution engine.

7. The system of claim 1, wherein the algorithm comprises a medical image inference model.

8. The system of claim 1, wherein the algorithms include internal algorithms and third-party algorithms stored at different network-accessible file locations.

9. The system of claim 1, wherein the computer executable components further comprise: A loading component that loads the algorithm information into the algorithm catalog in response to receiving the algorithm information via a loading user interface of an algorithm management application, wherein the algorithm is enabled to be incorporated into a workflow for executing the algorithm on the medical image based on including the algorithm information in the algorithm catalog.

10. The system of claim 1, wherein the algorithm management application further provides for adjusting one or more parameters and thresholds of the algorithm in association with receiving the algorithm information.

11. The system of claim 1 , wherein the computer executable component further comprises: A simulator component runs the workflow in a test mode against one or more test images and identifies execution errors.

12. The system of claim 9, wherein the computer executable components further comprise: A workflow management component provides access via the algorithm management application to workflow information defining the workflow and workflow management functions associated with the workflow.

13. The system of claim 12, wherein the workflow management functionality includes an activation / deactivation functionality that provides for selective activation and deactivation of one or more of the workflows.

14. A method for algorithmic orchestration of a workflow for medical care imaging diagnosis, comprising: providing, by a system operatively coupled to the processor, an algorithm catalog including algorithm information identifying algorithms applicable to medical images, the algorithm information including algorithm execution instructions for executing the algorithms in association with invoking the algorithms at corresponding network accessible file locations storing the algorithms; and a workflow creation application provided by the system, the workflow creation application being used to create a workflow, the system enabling the workflow to be applied to one or more of the medical images, the workflow comprising workflow information defining a set of computer-executable processing steps to be applied to the one or more of the medical images, wherein the workflow creation application includes a graphical user interface that facilitates creating a workflow using graphical nodes in association with receiving user input for dragging and dropping graphical nodes onto a workflow creation area of ​​the graphical user interface, connecting the graphical nodes using one or more connection lines to define relationships between the graphical nodes, and configuring the graphical nodes, Wherein, the graphic nodes include different types of graphic nodes corresponding to different types of computer-executable processing steps that can be used to process the medical image, respectively, and the different types of graphic nodes include task nodes representing tasks to be performed by the workflow, and the different types of graphic nodes also include nodes selected from the group consisting of decision nodes, sub-workflow nodes, waiting nodes, fork nodes, connection nodes and function nodes, and wherein, based on the selection of the task nodes, the workflow creation application provides a selectable list of algorithms contained in the algorithm catalog, and when the algorithm execution instructions corresponding to the algorithm selected from the selectable list are imported from the algorithm catalog into the workflow information, the selected algorithm is integrated into the workflow in an associated manner.

15. The method according to claim 14, further comprising: receiving, by the system via a network, an image processing request from a medical imaging provider; executing, by the system in association with receiving the image processing request, the workflow on the medical image to produce a workflow result; as well as The workflow results are returned by the system to the medical imaging provider, the workflow results comprising outputs of one or more of the algorithms.

16. The method of claim 15, wherein the executing comprises employing an algorithm execution engine that invokes the algorithm at a network accessible file location of the algorithm to receive the output associated with executing the workflow.

17. The method according to claim 16, further comprising: loading, by the system in response to receiving the algorithm information via a loading user interface of an algorithm management application, the algorithm information into the algorithm catalog, wherein the algorithm is enabled for incorporation into a workflow for executing the algorithm on the medical image based on including the algorithm information in the algorithm catalog; The algorithm format is converted, by the system in association with the loading, from a first format incompatible with the algorithm execution engine to a second format compatible with the algorithm execution engine.

18. The method of claim 14, wherein the algorithm comprises a medical image inference model.

19. The method according to claim 14, further comprising: The algorithm information is loaded by the system into the algorithm catalog in response to receiving the algorithm information via a loading user interface of an algorithm management application, wherein the algorithm is enabled to be incorporated into a workflow for executing the algorithm on the medical image based on including the algorithm information in the algorithm catalog.

20. The method of claim 14, wherein the algorithm management application further provides for adjusting one or more parameters and thresholds of the algorithm in association with receiving the algorithm information.

21. The method according to claim 14, further comprising: running the workflow in a test mode on one or more test images by the system; as well as The system determines whether there is an execution error in the workflow based on the running.

22. The method according to claim 19, further comprising: Access to workflow information defining the workflow and workflow management functions related to the workflow is provided by the system via the algorithm management application.

23. The method of claim 22, wherein the workflow management functionality includes an activation / deactivation functionality that provides for selective activation and deactivation of one or more of the workflows.

24. A machine-readable storage medium comprising executable instructions, which when executed by a processor, perform the method for algorithm orchestration of a workflow for medical care imaging diagnosis according to any one of claims 14 to 23.

Images