Medical therapeutic intervention procedure tools, resources, and duration determination, and associated devices, sytems, and methods

Abstract

An intervention planning system receives a first user input identifying a medical professional and a second user input identifying a patient. The system retrieves first data representative of the medical professional in response to the first user input; second data representative of the patient in response to the second user input; and third data associated with a medical facility at which the medical professional can perform a therapeutic procedure on the patient. The system pre-processes the first data, the second data, and the third data to form a structured data set and provides the structured data set as an input to a predictive network. The system generates with the predictive network a procedure plan for the therapeutic procedure. The system outputs, to a display, a screen display comprising at least one of the medical tool or the medical resource.

Description

MEDICAL THERAPEUTIC INTERVENTION PROCEDURE TOOLS, RESOURCES, AND DURATION DETERMINATION, AND ASSOCIATED DEVICES, SYTEMS, AND METHODSFIELD

[0001] The subject matter described herein relates to optimization and planning of medical interventions for a patient using information about the patient and / or information about doctor(s). For example, a predictive network analyzes medical information for a given patient case (e.g., medical images, medical image reports, medical records, etc.), interventionalist preferences and characteristics, and available medical resources to generate an intervention plan.BACKGROUND

[0002] Any given medical intervention, e.g., surgeries, cancer treatment, medical imaging, etc., may be performed in multiple ways. The choice of an intervention method and tools necessary for performing that intervention depends on the knowledge, preferences, and experience of the interventionalist, but also on external requirements, such as guidelines or availability of resources (e.g., operating room space and availability, equipment availability, other medical personnel with required skills, etc.). Thus, intervention planning can prove difficult. In addition, intervention planning may require disparately housed information that is often not available or accessible to a user seeking to plan an intervention.

[0003] The information included in this Background section of the specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and is not to be regarded as subject matter by which the scope of the disclosure is to be bound.SUMMARY

[0004] Disclosed is an automatic intervention planning tool. An intervention planning request is received from a user, i.e., medical professional / interventionalist. One or more intervention plans and durations are generated by a predictive network, such as a neural network. The predictive network may also generate numerous probabilities and indexes reflecting the suitability and importance of various tools and resources for an intervention. The user may then select a particular method for an intervention, and based on the selection, the intervention planning tool may automatically generate a checklist. For example, a doctor may seek an intervention plan for inserting a stent into an artery. The intervention planning tool will generate various methods indicating various preferred catheters and other tools for the stent placement procedure. In addition, the intervention planning system will also provide an estimate of the time it will take to complete the stent placement procedure.

[0005] This intervention planning system disclosed herein has particular, but not exclusive, utility for planning an intervention and predicting the likely duration of said intervention. The intervention planning system receives an intervention planning request and automatically generates predictions for intervention methods and likely duration. The intervention planning system advantageously allows a user to select subsets of the datasets based on the similarity or difference from the user. This allows experienced interventionalists to receive intervention planning system predictions that conform to their standard intervention methods and less experienced interventionalists to receive intervention methods informed by other interventionalists as trained into the intervention planning system.

[0006] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0007] In one general aspect, the present disclosure is directed to a computer-implemented method. The computer-implemented method also includes receiving, from a user input device, a first user input identifying a medical professional; retrieving, from one of more databases, first data representative of the medical professional in response to the first user input; receiving, from the user input device, a second user input identifying a patient; retrieving, from the one or more databases, second data representative of the patient inresponse to the second user input; retrieving, from the one or more databases, third data associated with a medical facility at which the medical professional can perform a therapeutic procedure on the patient; pre-processing the first data, the second data, and the third data to form a structured data set; providing the structured data set as an input to a predictive network; generating, as an output of the predictive network, a procedure plan for the therapeutic procedure, where the procedure plan that may include at least one of a medical tool or a medical resource to be utilized by the medical professional to perform the therapeutic procedure; and outputting, to a display, a screen display that may include at least one of the medical tool or the medical resource.

[0008] In some aspects, implementations may include one or more of the following features. The method may include generating, as an output of the predictive network, a predicted duration of the therapeutic procedure. The procedure plan that may include a series of substeps, and where the predicted duration that may include a plurality of sub-step durations, one sub-step duration for each sub-step in the procedure plan. The procedure plan that may include a plurality of probabilities, each probability in the plurality of probabilities associated with the medical tool or the medical resource, where each probability represents a likelihood of use of the medical tool or the medical resource in the therapeutic procedure. The procedure plan that may include a one or more indexes of criticality, each index of criticality associated with the medical tool or the medical resource, where each index of criticality represents the importance of the medical tool or the medical resource in the outcome of the therapeutic procedure. The receiving, from a user input device, a first user input identifying a medical professional further that may include: receiving, from a user input device, a selection of a subset of medical professionals, and where the structured data is restricted to data associated with the subset of medical professionals. A large language model preprocesses the first data, the second data, and the third data to form a structured dataset. The first data that may include biographic information about the medical professional, where the second data that may include medical records of the patient, and where the third data that may include at least one of available medical tool or available medical resources at the medical facility.

[0009] In one general aspect, the present disclosure is directed to a system a processor circuit configured to: receive, from a user input device, a first user input identifying a medical professional; retrieve, from one of more databases, first data representative of the medical professional in response to the first user input; receive, from the user input device, a second user input identifying a patient; retrieve, from the one or more databases, second data representative of the patient in response to the second user input; retrieve, from the one ormore databases, third data associated with a medical facility at which the medical professional can perform a therapeutic procedure on the patient; pre-process the first data, the second data, and the third data to form a structured data set; provide the structured data set as an input to a predictive network; generate, as an output of the predictive network, a procedure plan for the therapeutic procedure, where the procedure plan that may include at least one of a medical tool or a medical resource to be utilized by the medical professional to perform the therapeutic procedure; and output, to a display, a screen display that may include at least one of the medical tool or the medical resource.

[0010] In some aspects, implementations may include one or more of the following features. The system where the processor circuit is further configured to: generate, as an output of the predictive network, a predicted duration of the therapeutic procedure. The procedure plan that may include a series of sub-steps, and where the predicted duration that may include a plurality of sub-step durations, one sub-step duration for each sub-step in the procedure plan. The procedure plan that may include a plurality of probabilities, each probability in the plurality of probabilities associated with the medical tool or the medical resource, where each probability represents a likelihood of use of the medical tool or the medical resource in the therapeutic procedure. The procedure plan that may include a one or more indexes of criticality, each index of criticality associated with the medical tool or the medical resource, where each index of criticality represents the importance of the medical tool or the medical resource in the outcome of the therapeutic procedure. The processor circuit is further configured to: receive, from a user input device, a selection of a subset of medical professionals, and where the structured data is restricted to data associated with the subset of medical professionals. A large language model preprocesses the first data, the second data, and the third data to form a structured dataset. The first data that may include biographic information about the medical professional, where the second data that may include medical records of the patient, and where the third data that may include at least one of available medical tool or available medical resources at the medical facility.

[0011] In one general aspect, the present disclosure is directed to a non-transitory machine- readable medium that may include a plurality of machine-executable instructions which. The non-transitory machine-readable medium also includes receiving, from a user input device, a first user input identifying a medical professional; retrieving, from one of more databases, first data representative of the medical professional in response to the first user input; receiving, from the user input device, a second user input identifying a patient; retrieving, from the one or more databases, second data representative of the patient in response to thesecond user input; retrieving, from the one or more databases, third data associated with a medical facility at which the medical professional can perform a therapeutic procedure on the patient; pre-processing the first data, the second data, and the third data to form a structured data set; providing the structured data set as an input to a predictive network; generating, as an output of the predictive network, a procedure plan for the therapeutic procedure, where the procedure plan that may include at least one of a medical tool or a medical resource to be utilized by the medical professional to perform the therapeutic procedure; and outputting, to a display, a screen display that may include at least one of the medical tool or the medical resource.

[0012] In some aspects, implementations may include one or more of the following features. The non-transitory machine-readable medium where the one or more processors are further caused to perform operation that may include: generating, as an output of the predictive network, a predicted duration of the therapeutic procedure. The procedure plan that may include a series of sub-steps, and where the predicted duration that may include a plurality of sub-step durations, one sub-step duration for each sub-step in the procedure plan. The procedure plan that may include a plurality of probabilities, each probability in the plurality of probabilities associated with the medical tool or the medical resource, where each probability represents a likelihood of use of the medical tool or the medical resource in the therapeutic procedure.

[0013] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. A more extensive presentation of features, details, utilities, and advantages of the automatic measurement point detection system, as defined in the claims, is provided in the following written description of various aspects of the disclosure and illustrated in the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Illustrative aspects of the present disclosure will be described with reference to the accompanying drawings, of which:

[0015] Figure l is a schematic diagram of a networked system for intervention planning, according to aspects of the present disclosure.

[0016] Figure l is a schematic diagram of a processor circuit, according to aspects of the present disclosure.

[0017] Figure 3 is a schematic diagram of a deep learning network configuration, according to aspects of the present disclosure.

[0018] Figure 4 is a schematic diagram of at least a portion of an intervention planning system, according to aspects of the present disclosure.

[0019] Figure 5 is a schematic sequence diagram of a process for intervention planning, according to aspects of the present disclosure.

[0020] Figure 6 is a schematic diagram of at least a portion of system for training a predictive network in an intervention planning tool, according to aspects of the present disclosure.

[0021] Figure 7 is a schematic sequence diagram of a process for training a predictive network in an intervention planning tool, according to aspects of the present disclosure.

[0022] Figure 8 is a schematic diagram of a patient database, according to aspects of the present disclosure.

[0023] Figure 9 is a schematic diagram of an interventionalist database, according to aspects of the present disclosure.

[0024] Figure 10 is a schematic diagram of a resource database, according to aspects of the present disclosure.

[0025] Figure 11 is an example display for an intervention planning request, according to aspects of the present disclosure.

[0026] Figure 12 is a schematic diagram of a sequence of interfaces based on predictions of the intervention planning system, according to aspects of the present disclosure.

[0027] Figure 13 is a schematic diagram of slot planning support system, according to aspects of the present disclosure.

[0028] Figure 14 is a schematic flow diagram of a method for intervention planning, according to aspects of the present disclosure.

[0029] Figure 15 is a schematic flow diagram of a method for training a predictive network in an intervention planning system, according to aspects of the present disclosure.DETAILED DESCRIPTION

[0030] In accordance with at least one aspect of the present disclosure, an intervention planning system is provided that automatically generates intervention plans. This may allow, for example, users (including medical professionals and other interventionalists) to select between a number of possible methods for an intervention. Furthermore, the system provides an automatic way to implement user preferences for preferred level of experience used to generate intervention plans.

[0031] These descriptions are provided for exemplary purposes only, and should not be considered to limit the scope of the intervention planning system. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.

[0032] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the aspects illustrated in the drawings, and specific language will be used to describe the same. It is nevertheless understood that no limitation to the scope of the disclosure is intended. Any alterations and further modifications to the described devices, systems, and methods, and any further application of the principles of the present disclosure are fully contemplated and included within the present disclosure as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one aspect may be combined with the features, components, and / or steps described with respect to other aspects of the present disclosure. For the sake of brevity, however, the numerous iterations of these combinations will not be described separately. As used herein, accessing can include querying, retrieving, sorting, etc.

[0033] The systems and methods disclosed herein provide a number of benefits. For a given interventional procedure to be scheduled, experienced interventionalists may want to stick to their workflows, while inexperienced interventionalists may benefit from receiving guidance from more experienced colleagues. Thus, the systems and methods disclosed herein allow for an interventionalist to select proposal preferences for the preferred level of experience to make intervention predictions.

[0034] It is not always easy to select the best methods and tools for a procedure and to estimate the duration of the procedure, so the systems and methods described herein provide intervention planning, including selections of methods, tools, and resources required based on historic data and on interventionalists’ profiles.

[0035] As used herein, “therapeutic plan,” “procedure plan,” and “intervention prediction” may be used synonymously.

[0036] Figure 1 is a schematic diagram of a networked system 100 for intervention planning, according to aspects of the present disclosure. The networked system 100 may for example may be used to receive an intervention planning request from a user. An intervention planning request may be transmitted to different components of the network system 100 to facilitate intervention planning. The networked system 100 may provide a user with a prediction in the form of a therapeutic plan which was output from a predictive network.

[0037] The networked system 100 is used for generating a therapeutic plan. The networked system 100 may include a network / cloud computers 110,140, user computers 160, medical imaging console 170, and medical imaging device 180. The network / cloud computers 110, 140 may be in communication with each, each sending and / or receiving data and information from the other. User computer 160 is depicted in Figure 1 as in communication with network / cloud computer 110. However, in some instances, user computer 160 may be in communication with either or both of network / cloud computers 110, 140. User computer 160 may be in communication with a medical imaging console 170 which is in communication with a medical imaging device 180. As described herein, communication between the different components may be accomplished by any numbers of connections, e.g., wired and / or wireless.

[0038] Network / cloud computer 110 may include a processor 112, input device 114, display 116, communication interface 117, and memory 118.

[0039] Processor 112 may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 112 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 112 is configured to process the instructions stored in memory 118. The processor 112 is connected to the communication interface 117.

[0040] Input device 114 allows a user to make selections or provide instructions to the network / cloud computer 110. The input device, may be a mouse, touch screen, touch pad etc.

[0041] The display 116 is coupled to the processor 112. The display 116 may be a monitor or any suitable display. The display 116 is configured to display intervention planner 150output, data structuring network 155 output, slot planning support 126 output, information in databases 120, 122, 124, network performance, or any other system diagnostic information.

[0042] The communication interface 117 is coupled to the processor 112. The communication interface 117 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 117 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 117 can be referred to as a communication device or a communication interface module.

[0043] The memory 118 is coupled to the processor 112. The memory 118 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 112), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0044] The memory 118 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to ultrasound images, ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical scan window, a probe orientation, and / or the subject position during an imaging procedure. The memory 118 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting / segmenting anatomy, image quantification algorithms, and / or image acquisition guidance algorithms, including those described herein.

[0045] In some instances, memory 118 includes data and stored instructions for modules, including slot planning support 126. Slot planning support 126 may utilize predicted durations and other data included in user intervention planning requests to plan and reserve medical facility resources to schedule an intervention. For example, slot planning support 126may reserve a room for particular period of time based on the predicted duration of an intervention and also assign staff and other medical professionals needed for the intervention. Slot planning support 126 may create instructions and orders to provision a room, e.g., an operating room, with the necessary tools and resources for an intervention as selected by an interventionalist from the output of the intervention planner 150.

[0046] Network / cloud computer 140 may include a processor 142, input device 144, display 146, communication interface 147, and memory 148.

[0047] Processor 142 may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 142 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 142 is configured to process the instructions stored in memory 148. The processor 142 is connected to the communication interface 147.

[0048] Input device 144 allows a user to make selections or provide instructions to the network / cloud computer 140. The input device, may be a mouse, touch-screen, touch pad etc.

[0049] The display 146 is coupled to the processor 142. The display 146 may be a monitor or any suitable display. The display 146 is configured to display intervention planner 150 output, data structuring network 155 output, slot planning support 126 output, information in databases 120, 122, 124, network performance, or any other system diagnostic information.

[0050] The communication interface 147 is coupled to the processor 142. The communication interface 147 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 147 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 147 can be referred to as a communication device or a communication interface module.

[0051] The memory 148 is coupled to the processor 142. The memory 148 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 142), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flashmemory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0052] The memory 148 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to ultrasound images, ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical scan window, a probe orientation, and / or the subject position during an imaging procedure. The memory 148 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting anatomy, segmenting anatomy (e.g., defining contours of the anatomy), image quantification algorithms, and / or image acquisition guidance algorithms, including those described herein.

[0053] In some instances, memory 148 includes data and stored instructions for modules, including intervention planner 150, including predictive network 152, and data structuring network 155. As depicted in Fig. 1, the intervention planner 150 and the data structuring network 155 are located on a separate network / cloud computer from databases 120, 122, 124 and slot planning support 126. During intervention planning, data from databases 120, 122, 124 may be sent to the network / cloud computer 140. Intervention planner 150 may generate intervention predictions for the received data which may then be sent back to one or more of the computers and devices of networked system 100.

[0054] The predictive network 152 and data structuring network 155 may be any neural network-based model or rule-based programming model, e.g., as described with respect to and as depicted in Fig. 3. For example, networks 152, 155 can be Convolutional Neural Networks (CNN), decision tree models, support vector machines (SVM), generative image- to-image transformers, generative image-to-text transformers (GIT), or other transformerbased models.

[0055] User computer 160 may include a processor 162, input device 164, display 166, communication interface 167, and memory 168.

[0056] Processor 162 may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmablegate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 162 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 162 is configured to process the instructions stored in memory 168. The processor 162 is connected to the communication interface 167.

[0057] Input device 164 allows a user to make selections or provide instructions to the user computer 160. The input device, may be a mouse, touch-screen, touch pad, etc.

[0058] The display 166 is coupled to the processor 162. The display 166 may be a monitor or any suitable display. The display 166 is configured to display intervention predictions from intervention planner 150 as described herein. Furthermore, the display 166 may be configured to display the images from the data generated by medical imaging device 180.

[0059] The communication interface 167 is coupled to the processor 162. The communication interface 167 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 167 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 167 can be referred to as a communication device or a communication interface module.

[0060] The memory 168 is coupled to the processor 162. The memory 168 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 162), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0061] The memory 168 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to ultrasound images, ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subjectinformation may include parameters related to an imaging procedure such as an anatomical scan window, a probe orientation, and / or the subject position during an imaging procedure. The memory 168 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting anatomy, segmenting anatomy, image quantification algorithms, and / or image acquisition guidance algorithms, including those described herein.

[0062] Medical imaging console 170 may include a processor 172, input device 174, display 176, communication interface 177, and memory 178.

[0063] Processor 172 may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 172 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 172 is configured to process the instructions stored in memory 178. The processor 172 is connected to the communication interface 177.

[0064] Input device 174 allows a user to make selections or provide instructions to the medical imaging console 170. The input device, may be a mouse, touch-screen, touch pad, etc.

[0065] The display 176 is coupled to the processor 172. The display 176 may be a monitor or any suitable display. The display 166 is configured to display intervention predictions from intervention planner 150 as described herein. Furthermore, the display 176 may be configured to display the images from the data generated by medical imaging device 180.

[0066] The communication interface 177 is coupled to the processor 172. The communication interface 177 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 177 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 177 can be referred to as a communication device or a communication interface module.

[0067] The memory 178 is coupled to the processor 172. The memory 178 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 172),random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0068] The memory 178 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to ultrasound images, ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical scan window, a probe orientation, and / or the subject position during an imaging procedure. The memory 178 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting anatomy, segmenting anatomy, image quantification algorithms, and / or image acquisition guidance algorithms, including those described herein.

[0069] Medical imaging device 180 may generate medical images 185. The medical imaging device is in communication with medical imaging console 170, e.g., through the communication interface 177. The medical images generated by data from the medical imaging device 180 may be shown on the display 176. Medical images 185 may also be communicated to any of the computers, networks / clouds 110, 140, 160.

[0070] In some aspects, aspects of the present disclosure can be implemented with medical images 185 of subjects obtained using any suitable medical imaging device 180 and / or modality. Examples of medical images and medical imaging devices include x-ray images (angiographic images, fluoroscopic images, images with or without contrast) obtained by a medical imaging device such as an ultrasound imaging device, X-ray imaging device, computed tomography (CT) images obtained by a CT imaging device, positron emission tomography-computed tomography (PET-CT) images obtained by a PET-CT imaging device, magnetic resonance images (MRI) obtained by an MRI imaging device, single-photon emission computed tomography (SPECT) images obtained by a SPECT imaging device, optical coherence tomography (OCT) images obtained by an OCT imaging device, andintravascular photoacoustic (IVPA) images obtained by an IVPA imaging device. The medical imaging device 180 can obtain the medical images while positioned outside the subject body, spaced from the subject body, adjacent to the subject body, in contact with the subject body, and / or inside the subject body.

[0071] Training dataset 195 may be stored on network / cloud computer 110, network / cloud computer 140, or any other network or cloud computing environment. In some embodiments, the systems described herein may be provided as a cloud service. Client sites may connect to the cloud service to provide training data and to receive intervention planning services. Training dataset comprises historic information, which may include past interventions, results of the interventions, patient details, interventionalist details, and tools, resources, and methods used for the intervention. Training dataset 195 may comprise both structured and / or unstructured data. As described herein, unstructured data may be (pre-)processed before used as input to the model or for comparison with the output of the model.

[0072] It should be noted that the examples described above are provided for purposes of illustration, and are not intended to be limiting. Other devices and / or device configurations may be utilized to carry out the operations described herein.

[0073] Figure l is a schematic diagram of a processor circuit 250, according to aspects of the present disclosure. The processor circuit 250 may be implemented in the network / cloud computer 110, 140, user computer 160, imaging console 170, or other devices or workstations (e.g., third-party workstations, network routers, etc.), or on a cloud processor or other remote processing unit, as necessary to implement the method. As shown, the processor circuit 250 may include a processor 260, a memory 264, and a communication interface 268. These elements may be in direct or indirect communication with each other, for example via one or more buses.

[0074] The processor 260 may include a central processing unit (CPU), a digital signal processor (DSP), an ASIC, a controller, or any combination of general-purpose computing devices, reduced instruction set computing (RISC) devices, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other related logic devices, including mechanical and quantum computers. The processor 260 may also comprise another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 260 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0075] The memory 264 may include a cache memory (e.g., a cache memory of the processor 260), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, other forms of volatile and non-volatile memory, or a combination of different types of memory. In an aspect, the memory 264 includes a non-transitory computer-readable medium. The memory 264 may store instructions 266. The instructions 266 may include instructions that, when executed by the processor 260, cause the processor 260 to perform the operations described herein. Instructions 266 may also be referred to as code. The terms “instructions” and “code” should be interpreted broadly to include any type of computer-readable statement(s). For example, the terms “instructions” and “code” may refer to one or more programs, routines, subroutines, functions, procedures, etc. “Instructions” and “code” may include a single computer-readable statement or many computer-readable statements.

[0076] The communication interface 268 can include any electronic circuitry and / or logic circuitry to facilitate direct or indirect communication of data between the processor circuit 250, and other processors or devices. In that regard, the communication interface 268 can be an input / output (I / O) device. In some instances, the communication interface 268 facilitates direct or indirect communication between various elements of the processor circuit 250 and / or computers of networked system 100. The communication interface 268 may communicate within the processor circuit 250 through numerous methods or protocols. Serial communication protocols may include but are not limited to United States Serial Protocol Interface (US SPI), Inter-Integrated Circuit (I2C), Recommended Standard 232 (RS- 232), RS-485, Controller Area Network (CAN), Ethernet, Aeronautical Radio, Incorporated 429 (ARINC 429), MODBUS, Military Standard 1553 (MIL-STD-1553), or any other suitable method or protocol. Parallel protocols include but are not limited to Industry Standard Architecture (ISA), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI), Institute of Electrical and Electronics Engineers 488 (IEEE-488), IEEE-1284, and other suitable protocols. Where appropriate, serial and parallel communications may be bridged by a Universal Asynchronous Receiver Transmitter (UART), Universal Synchronous Receiver Transmitter (USART), or other appropriate subsystems.

[0077] External communication (including but not limited to software updates, firmware updates, model sharing between the processor and central server, or outputs generated by themodel selection and prediction system described herein) may be accomplished using any suitable wireless or wired communication technology, such as a cable interface such as a universal serial bus (USB), micro USB, Lightning, or FireWire interface, Bluetooth, Wi-Fi, ZigBee, Li-Fi, or cellular data connections such as 2G / GSM (global system for mobiles) , 3G / UMTS (universal mobile telecommunications system), 4G, long term evolution (LTE), WiMax, or 5G. For example, a Bluetooth Low Energy (BLE) radio can be used to establish connectivity with a cloud service, for transmission of data, and for receipt of software patches. The controller may be configured to communicate with a remote server, or a local device such as a laptop, tablet, or handheld device, or may include a display capable of showing status variables and other information. Information may also be transferred on physical media such as a USB flash drive or memory stick.

[0078] Figure 3 is a schematic diagram of a deep learning network configuration, according to aspects of the present disclosure. The configuration 300 can be implemented by a deep learning network. The configuration 300 includes a deep learning network 310, which may include one or more CNNs 312. The CNN 312 is one example of a type of predictive model, predictive network 152 or data structuring network 155. For simplicity of illustration and discussion, Fig. 3 illustrates one CNN 312. However, the embodiments can be scaled to include any suitable number of CNNs 312 (e.g., about 2, 3 or more). The configuration 300 can be trained for identification of various anatomy (organs, tissue, bone) and / or other features (natural and / or man-made) within a patient anatomy. The configuration 300 can be further trained for segmenting human anatomy, diagnosis medical conditions or any number of other diagnostic or medical tasks.

[0079] The CNN 312 may include a set of N convolutional layers 320 followed by a set of K fully connected layers 330, where N and K may be any positive integers. The convolutional layers 320 are shown as 320(1) to 320(N). The fully connected layers 330 are shown as 330(1) to 330(K). Each convolutional layer 320 may include a set of filters 322 configured to extract features from an input 302 (e.g., x-ray venogram images or other additional data). The values N and K and the size of the filters 322 may vary depending on the embodiments. In some instances, the convolutional layers 320(1) to 320(N) and the fully connected layers 330(1) to 330(K-l) may utilize a leaky rectified non-linear (ReLU) activation function and / or batch normalization. The fully connected layers 330 may be non-linear and may gradually shrink the high-dimensional output to a dimension of the prediction result 340 (e.g., location for an object detection box or other the classification output). The fully connected layers 330may also be referred to as a classifier. In some embodiments, the fully convolutional layers 320 may additionally be referred to as perception or perceptive layers.

[0080] When the prediction result 340 takes the form of classification output, it may indicate a confidence score (e.g., a probability) for each class 342 based on the input image 302. The classes 342 are shown as 342a, 342b, . . . , 342c. For example, when the CNN 312 is trained for regions of stenosis or general venous compression, the classes 342 may indicate an inguinal ligament class 342a, a crossover class 342b, a pelvic bone notch class 342c, a region of blood flow restriction class 342d, or any other suitable class. A class 342 indicating a high confidence score indicates that the input image 302 or a section or pixel of the image 302 is likely to include an anatomical object / feature of the class 342. Conversely, a class 342 indicating a low confidence score indicates that the input image 302 or a section or pixel of the image 302 is unlikely to include an anatomical object / feature of the class 342.

[0081] The CNN 312 can also output a feature vector 350 at the output of the last convolutional layer 320(N). The feature vector 350 may indicate objects detected from the input medical image 302 or other data. For example, the feature vector 350 may indicate regions associated with diseased tissue as identified from the image 302, which may be a patient medical image. The feature vector 350 may indicate the pixels in a medical image associated with the location of diseased tissue, as described herein.

[0082] The deep learning network 310 may implement or include any suitable type of learning network. For example, in some embodiments and as described in relation to Fig. 3, the deep learning network 310 could include a convolutional neural network 312. In addition, the convolutional neural network 310 may additionally or alternatively be or include a multiclass classification network, an encoder-decoder type network, a fully connected deep learning network, or any suitable network or means of identifying features within an image.

[0083] In an embodiment in which the deep learning network 310 includes a fully-connected neural network, the fully connected neural network may transform the data not related to a medical image or it may transform data derived from an image (e.g., detected objects) generated by another network, program, or human annotator. For example, the fully connected neural network may transform information about a patient, such as age, weight, or other low-dimensional data.

[0084] In an embodiment in which the deep learning network 310 includes an encoderdecoder network, the network may include two paths. One path may be a constricting path, in which a large image, such as the image 302, may be convolved by several convolutional layers 320 such that the size of the image 302 changes in relation to the depth of the networklayer. The image 302 may then be represented in a low dimensional space, or a flattened space. From this flattened space, an additional path may expand the flattened space to the original size of the image 302. In some embodiments, the encoder-decoder network implemented may also be referred to as a principal component analysis (PCA) method. In some embodiments, the encoder-decoder network may segment the image 302 into patches. In an additional embodiment of the present disclosure, the deep learning network 310 may include a multi-class classification network. In such an embodiment, the multi-class classification network may include an encoder path. For example, the image 302 may be of a high dimensional image. The image 302 may then be processed with the convolutional layers 320 such that the size is reduced. The resulting low dimensional representation of the image 302 may be used to generate the feature vector 350 shown in Fig. 3. The low dimensional representation of the image 302 may additionally be used by the fully connected layers 330 to regress and output one or more classes 342. In some regards, the fully connected layers 330 may process the output of the encoder or convolutional layers 320. The fully connected layers 330 may additionally be referred to as task layers or regression layers, among other terms.

[0085] Any suitable combination or variations of the deep learning network 310 described is fully contemplated. For example, the deep learning network may include fully convolutional networks or layers or fully connected networks or layers or a combination of the two. In addition, the deep learning network may include a multi-class classification network, an encoder-decoder network, or any combination of networks.

[0086] In some embodiments, a data structuring network 155 may include a large language model (LLM). The LLM may be used to generate structured data from unstructured data. For example, input into the LLM could be extensive text contained in medical report written by medical professionals. The LLM may generate from the unstructured medical reports structured data including diagnosis, frequency of patient interactions, response to treatment, etc. LLMs may have a variety of network architectures, including autoencoders and transformer-based models.

[0087] Figure 4 is a schematic diagram of at least a portion of an intervention planning system 400, according to aspects of the present disclosure. Intervention planning system 400 generates one or more therapeutic plans (i.e., intervention predictions 425) and predicts an expected duration 420 for the therapeutic plans. User input (e.g., intervention planning request 410) and selections may take the form of medical images, patient information, medical records, intervention objectives, interventionalist experience, medical resource availability (e.g., infrastructure, available scheduling time, staff resources, etc.) and theoutput may be a therapeutic plan 425 and the expected duration 420 of the intervention associated with the therapeutic plan. A user may be any of a number of medical professionals. In some instances, selections by a user may cause the system 400 to automatically gather additional information based on the selections from various databases. A user may also restrict predictions for therapeutic plans by interventionalist experience. For example, a user may prefer that interventions from all interventionalists be considered, interventions by interventionalists with similar experience to the user, or only interventions previously done by the user. The intervention planning system 400 may include data structuring network 155, intervention planner 150, and database 415.

[0088] The intervention planning system 400 may receive an intervention planning request 410 from a user computer 160 or medical imaging console 170. Intervention planning request 410 may comprise an interventionalist identifier and input 412, intervention objectives 414, and patient details 416. In some instances, interventionalist identifier and input 412 may be populated by an interventionalist using a selection interface, depicted and further described with respect to Fig. 11, allowing an interventionalist to select proposal preferences. In some instances, intervention objectives 414 are goals set by an interventionalist, e.g., 30% reduction in arterial blockage. In some instances, patient details may include identifying information of a patient so that medical records related to the patient may be retrieved from one or more databases. An intervention planning request may be structured using a data structuring network 155 and serve as query to database 415 for additional patient-, interventionalist-, and / or resource-related information.

[0089] Data structuring network 155 may receive unstructured portions of the intervention planning request. Data structuring network 155 may include one or more neural networks suitable for structuring data. The input to the data structuring network may be natural language, e.g., an interventionalist descriptions of the goals for an intervention. The output of the data structuring network is structured data, e.g., diagnoses conforming to a standardized format. In some embodiments, the structured data output from the data structuring network takes the form of an n-dimensional vector, wherein the location of entries in the n- dimensional vector correspond to different parts of the input context. For relatively simple contexts, the n-dimensional vector may have lower dimension, e.g., 1 to 10, whereas for more complicated contexts (e.g., a greater variety of available input data) the n-dimensional vector may have higher dimensions, e.g., greater than 10. As an example, a data structuring network 155 may be a large language model (LLM). The term “Large Language Model” (LLM) may refer to a neural network based deep learning system designed to understand and generatenatural language. An LLM may possess a Transformer architecture, which often includes a significant number of parameters (weights in a neural network). For example, LLM such as Generative Pre-trained Transformer (GPT) 3 has 175 billion parameters, Text-to-Text Transfer Transformers (T5) has around 11 billion parameters.

[0090] Database 415 may include a patient database 120, interventionalist database 122, or a resource database 124. Information contained in intervention planning request 410 may be used to retrieve certain information from database 415. For example, patient details 416 may be used to retrieve all or a portion of a patient’s medical history contained in patient database 120; interventionalist identifier and input 412 may be used to information about the interventionalist from an interventionalist database 122; or information in the intervention planning request 410 to retrieve the required information from the resource database 124, e.g., particular medical facilities and staff along with their capability and experience, respectively. If unstructured data is contained and retrieved from database 415 based on the intervention planning request 410, then data structuring network 155 may be used to generate a structured form of the data from database 415. For example, patient database 120 may include medical professionals notes and / or medical reports (e.g., radiology reports) containing written language which may be processed by the data structuring network 155 into a standard format.

[0091] Intervention planner 150 may receive the structured data from the intervention planning request 410 and database 415. Using a predictive network 152, intervention planner 150 generates an intervention prediction 425 and an associated predicted duration 420 for the intervention prediction. In some embodiments, predictive network 152 is capable of receiving all of the input received by the intervention planner 150 and produce a multi -parametric regression as an output. In some embodiments, predictive network 152 may comprise a deep neural network comprising multiple network layers with all-to-all connections between nodes in sequential layers. In some embodiments, predictive network 152 may include one or more neural network models capable of receiving medical images, e.g., a CNN as described with respect to Fig. 3. In addition to neural networks, the predictive network may also be a Support Vector Machines, Random Forest model, or a decision tree model.

[0092] Intervention prediction 425 may comprise one or more different possible interventions, e.g., different treatment procedures using different equipment or the same equipment in a different way. In some embodiments, intervention prediction 425 may take the form as shown in Table 1, below. Table 1 shows an intervention prediction 425 that proposes 3 possible methods for intervention. For each method the probability of requiring aparticular tool or resource is shown in the corresponding entry of the table. For example, Method 1 has probability of 20% to require Tool 1, probability of 80% to require Tool 2, probability of 94% to require Resource 1, and a probability of 50% to require Resource 2. Tools could be various pieces of medical equipment such as an intravenous catheter, and resources could be rooms of various sophistications, e.g., examination rooms or operating rooms, or various staffing requirements, e.g., a method requiring an anesthesiologist. The information about a tool requirement can also be augmented by a respective use (e.g., removal of a lesion) and time (e.g., preparation of removal) or a more detailed timeline. In addition, information about the underlying reasoning for a probability generated by the predictive network may be provided to a user increase acceptance of the proposed method(s) and tools. For example, the suitability of a particular method of intervention may be primarily determined by a patient’s advanced age, which rules out a number of alternative methods of intervention.

[0093] Intervention prediction 425 may include any numbers of methods, tools, and resources as appropriate for the context and the intervention planning request 410. For example, only one method may be possible for an intervention and so only one method would be predicted. However, for an intervention where many procedures are possible, a large number of methods may be suggested. In some embodiments, subOsteps of a method may be listed in intervention prediction 425. For example, a stent placement procedure may require postplacement imaging, which could be listed as a sub-step. In some embodiments, an index of criticality may be included in the intervention prediction 425. The index of criticality reflects how critical a certain resource is for a certain method. In other words, a high index of criticality for a tool may indicate that the absence of tool has a significant impact on the outcome of the intervention. In some embodiments, an index of criticality may be generated for each tool or resource for different sub-steps / phases of the intervention. The index of criticality in combination with the probability of using a certain resource may be used to prioritize or de-prioritize these critical resources in a hospital scheduling system.Table 1

[0094] Predicted duration 420 of an intervention may be an estimate of the time it takes to complete an intervention, e.g., a number of minutes or hours and minutes. Predicted duration 420 may be used by slot planning support 126 to reserve and schedule equipment and resources for the intervention at a medical facility, such as a hospital. In some embodiments, the estimation of the duration of the intervention further depends on the selected method, which can also include variations depending on the expertise of the interventionalist. In this case, separate estimation results are output, one for each possible method. In some embodiments, predicted duration 420 may also include an estimate of the variance (e.g., an error bar) of the duration of the planned intervention. In some embodiments, the predicted duration may include estimates of the time and / or associated variances (e.g., error bars) in the time for sub-steps / phases of the intervention.

[0095] In some embodiments, the proposed tools and resources can be compared with the availability of tools and resources in a hospital. If required tools or resources are unavailable, the suitability of the respective methods can be adjusted accordingly or a rescheduling of the intervention if the resources are only temporarily unavailable.

[0096] The intervention prediction 425 or predicted duration 420 may be provided to a user computer 160 or medical imaging console 170. As further described with respect to Fig. 14, intervention prediction 425 may be provided to an interventionalist through a user interface, allowing the interventionalist to select between the proposed methods of intervention. Selections by an interventionalist, along with the predicted duration, may be provided to a slot planning support 126. Slot planning support 126 may be used by an organization to plan and reserve all the necessary infrastructure and resources for an intervention.

[0097] An interventionalist may provide feedback 430 on the intervention predictions 425 through user computer 160 or medical imaging console 170. In some embodiments, feedback 430 may be a quality indicator reflecting an interventionalist’ s evaluation of the suggested interventions. In addition, feedback 430 may provide include intervention results to the extent they can be ascertained after completion of the intervention. In some embodiments, the feedback loop may be an automated feedback loop that collects the information about tools, resources, and durations from IT systems, or may be a manual feedback loop that acquires feedback about the intervention from the interventionalist after the procedure, or it may be a combination of the two. In some embodiments a checklist, e.g., as shown in Fig. 12, may contain the options “ready” and “not required” for each element. Selections in the checklist may be recorded when the proposed tools and resources were actually used in the intervention / procedure. The checklist may also include an option to add required tools orresources manually that have not been predicted by the system. In some instances, the tool and / or resource usage may be automatically recorded using known device / motion tracking technology, including via the use of a CNN for object detection.

[0098] Figure 5 is a schematic sequence diagram of a process for intervention planning system 400, according to aspects of the present disclosure. The intervention planning system can include slot planning support 126, user computer 160 or medical imaging console 170, data structuring network 155, intervention planner 150, and database 415.

[0099] At step 502, the data structuring network 155 may receive an intervention planning request from user computer 160 or medical imaging console 170 and database 415 may be queried for information associated with the intervention planning request. Database 415 may include patient database 120, interventionalist database 122, or resource database 124 as described herein.

[0100] At step 504, data associated with the intervention planning request is retrieved from the database 415 and provided to the data structuring network 155.

[0101] At step 506, the data structuring network 155 may (pre-)process unstructured data contained in the intervention planning request and in the data associated with the intervention planning request from database 415. In some embodiments, data from database 415 may be structured and therefore not require structuring by the data structuring network 155.

[0102] At step 508, the structured intervention planning request and structured data from the database is provided to the intervention planner 150.

[0103] At step 510, intervention planner 150 generates a therapeutic plan and predicted duration. In some embodiments, request 410 may be received at intervention planner 150 without any intermediate structuring by the data structuring network 155.

[0104] At step 512, the therapeutic plan and predicted duration is provided to the user computer 160 or medical imaging console 170 and slot planning support 126.

[0105] At step 514, the therapeutic plan is output to a display for user evaluation and selection.

[0106] At step 516, user selections are provided to slot planning support 126

[0107] At step 518, slot planning support schedules the intervention. As described herein, scheduling the intervention may comprise reserving various infrastructures (e.g., surgical rooms), resources (e.g., surgical tools), and staff.

[0108] At step 520, an interventionalist may provide feedback through interaction with user computer 160 or medical imaging console 170 to the intervention planner 150.

[0109] One skilled in the art will appreciate that steps in the process may be performed separately or concurrently and in other arrangements.

[0110] Figure 6 is a schematic diagram of at least a portion of a training system 600 for a predictive network 152, according to aspects of the present disclosure. Training system 600 trains the predictive network 152 to accurately predict interventions and durations for those interventions from the context provided at inference. Predictive network 152 is trained by providing historic information of the choice of methods, tools, and resources, as well as the duration of intervention, alongside the respective input features described in intervention planning system 400, e.g., the contents of intervention planning request 410 and data retrieved from database 415. The information about tools and methods may be extracted from reporting documents or may be recorded manually after each intervention. The information about the duration of the intervention may be extracted from hospital IT systems or may be recorded manually after each intervention. Training system 600 includes data structuring network 155, intervention planner 150, and model objectives / functions 615.

[0111] Training data 602 may be contained in the training dataset 195. Training data 602 may include unstructured training data 604, structured training data 606, and ground truth labels and target predictions 608. Unstructured data 604 may be in the form of natural language from medical reports, objectives for an intervention, and other sources as described herein. Furthermore, unstructured may include images with or without annotations or classifier. Structured data may include things such as patient details like sex, weight, height, medications, etc. Ground truth labels and predictions 608 may include preferred methods for intervention given a context, the outcome of the preferred intervention, and / or actual duration times for the intervention.

[0112] Data structuring network 155 may receive the training data 602 and generate structured training data. Data structuring network 155 may include one or more neural networks suitable for structuring unstructured data, e.g., as described with respect to Fig. 3. The input to the data structuring network may be natural language, e.g., an interventionalist descriptions of the goals for an intervention. The output of the data structuring network is structured data, e.g., diagnoses conforming to a standardized format. In some embodiments, the structure data output from the data structuring network takes the form of an n-dimensional vector, wherein the location of entries in the n-dimensional vector correspond to different parts of the input context. For relatively simple contexts, the n-dimensional vector may have lower dimension, e.g., 1 to 10, whereas for more complicated contexts (e.g., a greater variety of available input data) the n-dimensional vector may have higher dimensions, e.g., greaterthan 10. As an example, a data structuring network 155 may be a large language model (LLM). the term “Large Language Model” (LLM) may refer to a neural network based deep learning system designed to understand and generate natural language. An LLM may possess a Transformer architecture, which often includes a significant number of parameters (weights in a neural network). For example, LLM such as Generative Pre-trained Transformer (GPT) 3 has 175 billion parameters, Text-to-Text Transfer Transformers (T5) has around 11 billion parameters.

[0113] Intervention planner 150 may receive the structured training data from the data structuring network 155. Intervention planner 150 includes a predictive network 152, similar to the predictive network described with respect to Figs. 3 and 4. Using the predictive network 152, the intervention planner 150 generates intervention prediction and duration predictions 610 which are similar in structure and content to predicted duration 420 and intervention prediction 425 as described with respect to Fig. 4.

[0114] Using model objectives / functions 615 training system 600 compares the intervention prediction and duration predictions 610 with the associated ground truth labels and predictions 608. Model objectives / functions 615 may include objectives / functions which penalize to a greater or lesser extent predicted durations which are further or closer to the ground truth labels and prediction, respectively. Similarly, model objectives / functions 615 may include objectives / functions that penalize selecting the wrong tool or resource with higher probability than the ground truth prediction. Model objectives / functions 615 may include objectives / functions that penalize a prediction based on an incorrect ranking of methods. Losses may be chosen based on the task and / or data type. Some examples of loss may be the LI and L2 loss, mean-squared error loss, cross-entropy losses, Kullback Leibler divergence loss, etc.

[0115] Comparisons from the model objectives / functions 615 may update parameters 620 of the predictive network 152. In some instances, updating may be accomplished using gradient of the objective functions and backpropagation to update the parameters of the predictive network 152.

[0116] In some embodiments, training and / or finetuning can be done for newly introduced or experimental devices (e.g., within clinical trials). For instance, the provider of a newly approved device could make the data from the tests and trials of the device available for model training, thus introducing the device into the (potential) workflow.

[0117] Figure 7 is a schematic sequence diagram of a process for a system for training a predictive network 600, according to aspects of the present disclosure. The system fortraining a predictive network can include training dataset 195, data structuring network 155, intervention planner 150, and model objective / functions 615.

[0118] At step 702, training data from training dataset 195 is provided to the data structuring network 155. In some embodiments, training data may include ground truth labels or target predictions.

[0119] At step 704, data structuring network 155 (pre-)processes unstructured training data. In some instances, training data may be structured and not require processing by the data structuring network.

[0120] At step 706, the structured training data 706 is provided to the intervention planner 150.

[0121] At step 708, the intervention planner 150 generates predictions. In some embodiments predictions may include a therapeutic plan and expected duration for the therapeutic plan.

[0122] At step 710, predictions and ground truth labels or target predictions are provided to model objectives / functions 615.

[0123] At step 712, model objectives / functions 615 are used to compare predictions with ground truth labels or target predictions.

[0124] At step 714, the comparisons of predictions and ground truth labels or target predictions are provided to the intervention planner 150.

[0125] At step 716, intervention planner 150 may update network parameters based on the comparisons. In some embodiments, updating involves the use of backpropagation of model objective / function gradients to modify the parameters of a predictive network contained in intervention planner 150.

[0126] One skilled in the art will appreciate that steps in the process may be performed separately or concurrently and in other arrangements.

[0127] Figures 8-10 are schematic diagrams of various databases. Each database may share data in common with another. For example, an interventionalist may have performed an intervention on a patient in the past and that information could be reflected both in the patient database 120 and the interventionalist database 122. Each item in these databases may be associated with a scheduling system to allow effective assignment of infrastructure, equipment, staff, and resources, avoiding double-booking.

[0128] Figure 8 is a schematic diagram of a patient database 120, according to aspects of the present disclosure. Patient database 120 may include patient details 810, medical images 820, and medical reports 830. Patient details 810 may be information that impacts the methods or tools used for the intervention, which may include sex, body weight, allergies, age, healthconditions, special requirements, medication, results from previous interventions. Medical images 820 and medical reports 830 may include lab reports, medical device data (e.g., ECG, EEG) and any prior medical images or data containing information that may affect how the intervention is performed. In some instances, such information can be found in Electronic Medical Records (EMRs) and Electronic Health Records (HER). The images can also be augmented by radiology reports of the medical images.

[0129] Figure 9 is a schematic diagram of an interventionalist database 122, according to aspects of the present disclosure. Interventionalist database 122 may include a collection of past procedures 910 performed by one or more interventionalists (including the interventionalist who may place the intervention planning request as described with respect to Fig. 4) and interventionalist details 920. Past procedures 910 may be a structured or unstructured list of historical interventions performed by an interventionalist, including the outcome resources used and other information associated with the intervention.Interventionalist details 920 may include information such educations, experience, specialties, and preferences for each interventionalist.

[0130] Figure 10 is a schematic diagram of a resource database 124, according to aspects of the present disclosure. Resource database 124 may include facility infrastructure 1010, Equipment 1020, and staff resources 1030. Facility infrastructure 1010 may include a listing of rooms and associated capabilities for one or more medical facilities. For example, operating rooms will have different capabilities compared to examination rooms. There may be many other room types, reflecting different specialized capabilities. Equipment 1020 may include any number of devices, e.g., catheters, EKG, CT, MRI, etc. Equipment 1020 contains the tools that may comprise part of the output of the intervention planning system 400. Staff resources 1030 may including a listing of staff that includes specialties and experience. For example, some interventions may require minimal staff support beyond the interventionalist, whereas a surgery may require a number of staff with different specialties and experience.

[0131] Figure 11 is an example display 1100 for an intervention planning request, according to aspects of the present disclosure. Example display 1100 may be a user interface provided through a display at the user computer 160 or medical imaging console 170 to an interventionalist. The display 1100 may include prompts for an interventionalist to identify a patient 1105, the intervention objective 1110, and identify the interventionalist 1115. Patient slot 1105 may receive a patient identifier (e.g., by a unique identifier or name). Intervention objective 1110 may be a natural language description provided by the interventionalist describing the diagnosis for the patient and the goal of the intervention. In someembodiments, this will be used by the intervention planning system to set the context of the planned intervention. Interventionalist slot 1115 may receive an interventionalist identifier (e.g., a unique alphanumeric identifier or name). Interventionalist slot 1115 may be automatically populated based on an interventionalist’ s credentials used to log in to user computer 160 or medical imaging console 170.

[0132] Furthermore, an interventionalist may be prompted to selected to domain of experience over which interventions may be generated by the intervention planning system, reflected by the “Choose Proposal Preferences” sub-window 1120. Probabilities generated by intervention planning system 400 may depend on the interventionalist profile, since they may have different preferences or expertise. The intervention planning system 400 may output separate probability sets for: (1) The interventionalist scheduled to perform the procedure (i.e., based on status or training data from the interventionalist); (2) Peers of the interventionalist scheduled to perform the procedure (i.e., based on status or training data from a set of interventionalists with similar background, experience, and preferences); or (3) All interventionalists (i.e., based on the full training data set).

[0133] Figure 12 is a schematic diagram of a sequence of interfaces based on predictions on the intervention planning system, according to aspects of the present disclosure. Using the intervention prediction 425, an example given in Table 1 and reproduced in Fig. 12, a user selection interface 1205 may be generated presented to an interventionalist on a display of user computer 160 or medical imaging console 170. The user selection interface 1205 includes a structured presentation of the intervention prediction 425. In addition to the percentages shown for each tool and resource, user selection interface 1205, includes suitability percentages 1207-1209. Suitability percentages 1207-1209 indicate how appropriate the method is for the planned intervention. For example, Method 1 has the highest suitability percentage of 90%. In some embodiments, an index of criticality may also be included with the probabilities for each tool and resource.

[0134] After an interventionalist selects one of the presented methods in the user selection interface 1205, a preparation checklist 1210 may generated. Preparation checklist 1210 includes a list of tools and resources associated with the selected method. Checklist 1210 may have boxes for marking whether a particular tool or resource is ready or not ready. In some embodiments, checklist 1210 may be used to mark whether a tool or resource was used during an intervention. A completed checklist may be used to provide feedback to the intervention planning system 400 and update the predictive network as described herein.

[0135] Figure 13 is a schematic diagram of slot planning support system 1300, according to aspects of the present disclosure. Slot planning support system 1300 allows a medical facility or other healthcare organization to plan for interventions and reserve facility space, such as an operating room. Slot planning support system 1300 includes slot planning support 126. Slot planning support 126 may receive projected duration 420 and generate infrastructure and staff reservations 1305. Infrastructure and staff reservations 1305 may include use of medical facility IT systems to reserve one or more rooms and to assign one or more staff to assist the intervention in that room at the scheduled time. In some embodiments, slot planning support 126 may also receive the predicted intervention. Slot planning support 126 may utilize the predictions for the likelihood of using various tools and resources to ensure the tools are available at the time of the intervention in the correct room.

[0136] Figure 14 is a schematic flow diagram of a method 1400 for intervention planning, according to aspects of the present disclosure. It is understood that the steps of method 1400 may be performed in a different order than shown in Figure 14, additional steps can be provided before, during, and after the steps, and / or some of the steps described can be replaced or eliminated in other aspects. One or more of steps of the method 1400 can be carried out by one or more devices and / or systems described herein, such as components of the networked system 100 and / or processor circuit 250.

[0137] In step 1402, the method 1400 includes receiving, from a user input device, a first user input identifying a medical professional. For example, a user entering an interventionalist identifier in response to prompt 1115 through a user computer 160. In some embodiments, the method 1400 further includes receiving, from a user input device, a selection of a subset of medical professionals (e.g., 1120 in Fig. 11).

[0138] In step 1404, the method 1400 includes retrieving, from one of more databases, first data representative of the medical professional in response to the first user input. For example, interventionalist details from interventionalist database 122. In some embodiments, the first data comprises biographic information (e.g., stored in interventionalist database 122) about the medical professional.

[0139] In step 1406, the method 1400 includes receiving, from the user input device, a second user input identifying a patient. For example, a user input to prompt 1105 though a user computer 160.

[0140] In step 1408, the method 1400 includes retrieving, from the one or more databases, second data representative of the patient in response to the second user input. For example,patient information from patient database 120. In some embodiments, the second data comprises medical records (e.g., stored in patient database 120) of the patient.

[0141] In step 1410, the method 1400 includes retrieving, from the one or more databases, third data associated with a medical facility at which the medical professional can perform a therapeutic procedure on the patient. For example, resource information from resource database 124. In some embodiments, the third data comprises at least one of available medical tool or available medical resources (e.g., stored in resource database 124) at the medical facility.

[0142] In step 1412, the method 1400 includes pre-processing the first data, the second data, and the third data to form a structured data set. For example, pre-processing with the data structuring network 155 the intervention planning request 410 and data from database 415.

[0143] In step 1414, the method 1400 includes providing the structured data set as an input to a predictive network. For example, providing the output of the data structuring network 155 to intervention planner 150, containing predictive network 152. In some embodiments, method 1400 further includes restricting the structured data to data associated with the subset of medical professionals (e.g., based on a selection by a user in 1120).

[0144] In step 1416, the method 1400 includes generating, as an output of the predictive network, a procedure plan for the therapeutic procedure, wherein the procedure plan comprises at least one of a medical tool or a medical resource to be utilized by the medical professional to perform the therapeutic procedure. For example, the intervention planner 150 outputting intervention prediction 425. In some embodiments, method 1400 may further include generating, using the predictive network 152, a predicted duration 420 of the therapeutic procedure. In some instances, the procedure plan 425 may comprise a series of sub-steps and a predicted duration 420 may comprise a plurality of sub-step durations, each associated with each of the sub-steps in the procedure plan. In some embodiments, the procedure plan 425 comprises a plurality of probabilities, each probability in the plurality of probabilities associated with the medical tool or the medical resource (e.g., the probabilities depicted in Table 1), wherein each probability represents a likelihood of use of the medical tool or the medical resource in the therapeutic procedure. In some embodiments, the procedure plan 425 comprises a one or more indexes of criticality, each index of criticality associated with the medical tool or the medical resource, wherein each index of criticality represents the importance of the medical tool or the medical resource in the outcome of the therapeutic procedure.

[0145] In step 1418, the method 1400 includes outputting, to a display, a screen display comprising at least one of the medical tool or the medical resource. For example, a display 166 on user computer 160 displaying the user selection interface 1205.

[0146] Figure 15 is a schematic flow diagram of a method 1500 for training a predictive network 152 in an intervention planning system, according to aspects of the present disclosure. It is understood that the steps of method 1500 may be performed in a different order than shown in Figure 15, additional steps can be provided before, during, and after the steps, and / or some of the steps described can be replaced or eliminated in other aspects. One or more of steps of the method 1500 can be carried by one or more devices and / or systems described herein, such as components of the networked system 100 and / or processor circuit 250.

[0147] In step 1502, the method 1500 includes retrieving, from one or more databases, training data, including inputs for a predictive network and ground truth outputs (an intervention plan and / or duration). For example, training data 602 retrieved from training dataset 195.

[0148] In step 1504, the method 1500 includes pre-processing the training data into structured training data. For example, preprocessing training data 602 with data structuring network 155.

[0149] In step 1506, the method 1500 includes providing the structured training data as an input to a predictive network. For example, providing the output of data structuring network 155 to the intervention planner 150, including the predictive network 152.

[0150] In step 1508, the method 1500 includes generating, as an output of the predictive network, a procedure plan for the therapeutic procedure and an expected duration, wherein the procedure plan comprises at least one of a medical tool or a medical resource to be utilized by the medical professional to perform the therapeutic procedure. For example, the intervention planner 150, including predictive network 152, generating intervention predication and durations predictions 610.

[0151] In step 1510, the method 1500 includes comparing, using objective functions and / or losses, the procedure plan and expected duration with corresponding ground truth outputs. For example, model objective s / functions 615 comparing intervention prediction and durations predictions 610 with associated ground truth labels and predictions 608.

[0152] In step 1512, the method 1500 includes updating predictive network based on the comparison. For example, using the comparison from model objectives / functions 615 update parameters 620 of the predictive network 152.

[0153] Furthermore, the technology disclosed herein is also applicable to other medical imaging modalities obtained from a medical imaging device where 3D data is available, such as other ultrasound applications, camera-based videos, X-ray videos, and 3D volume images, such as computer aided tomography (CT) scans, magnetic resonance imaging (MRI) scans, optical coherence tomography (OCT) scans, or intravenous ultrasound (IVUS) pullback sequences. The technology described herein can be used in a variety of settings including emergency department, intensive care, inpatient, and out-of-hospital settings.

[0154] Accordingly, the logical operations making up the aspects of the technology described herein are referred to variously as operations, steps, objects, layers, elements, components, algorithms, or modules. Furthermore, it should be understood that these may occur or be performed or arranged in any order, unless explicitly claimed otherwise or a specific order is inherently necessitated by the claim language.

[0155] All directional references e.g., upper, lower, inner, outer, upward, downward, left, right, lateral, front, back, top, bottom, above, below, vertical, horizontal, clockwise, counterclockwise, proximal, and distal are only used for identification purposes to aid the reader’s understanding of the claimed subject matter, and do not create limitations. Connection references, e.g., attached, coupled, connected, joined, or “in communication with” are to be construed broadly and may include intermediate members between a collection of elements and relative movement between elements unless otherwise indicated. As such, connection references do not necessarily imply that two elements are directly connected and in fixed relation to each other. The term “or” shall be interpreted to mean “and / or” rather than “exclusive or.” The word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. Unless otherwise noted in the claims, stated values shall be interpreted as illustrative only and shall not be taken to be limiting.

[0156] The above specification, examples and data provide a complete description of the structure and use of exemplary aspects of the intervention planning system as described herein. Although various aspects of the claimed subject matter have been described above with a certain degree of particularity, or with reference to one or more individual aspects, those skilled in the art could make numerous alterations to the disclosed aspects without departing from the spirit or scope of the claimed subject matter.

[0157] Still other aspects are contemplated. It is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative only of particular aspects and not limiting. Changes in detail or structure may bemade without departing from the basic elements of the subject matter as defined in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method, comprising: receiving (502), from a user input device (164), a first user input identifying a medical professional (1115); retrieving (504), from one of more databases (415), first data representative of the medical professional in response to the first user input; receiving (502), from the user input device (164), a second user input identifying a patient (1105); retrieving (504), from the one or more databases (415), second data representative of the patient in response to the second user input; retrieving (504), from the one or more databases (415), third data associated with a medical facility at which the medical professional can perform a therapeutic procedure on the patient; pre-processing (506) the first data, the second data, and the third data to form a structured data set; providing (508) the structured data set as an input to a predictive network (152); generating (510), as an output of the predictive network, a procedure plan (425) for the therapeutic procedure, wherein the procedure plan comprises at least one of a medical tool or a medical resource to be utilized by the medical professional to perform the therapeutic procedure; and outputting (514), to a display (166), a screen display (1205) comprising at least one of the medical tool or the medical resource.

2. The method of claim 1, further comprising: generating (510), as an output of the predictive network (152), a predicted duration (420) of the therapeutic procedure.

3. The method of claim 2, wherein the procedure plan (425) comprises a series of sub-steps, and wherein the predicted duration (420) comprises a plurality of sub-step durations, one sub-step duration for each sub-step in the procedure plan.

4. The method of claim 1, wherein the procedure plan (425) comprises a plurality of probabilities, each probability in the plurality of probabilities associated with the medical tool or the medical resource, wherein each probability represents a likelihood of use of the medical tool or the medical resource in the therapeutic procedure.

5. The method of claim 4, wherein the procedure plan (425) comprises a one or more indexes of criticality, each index of criticality associated with the medical tool or the medical resource, wherein each index of criticality represents the importance of the medical tool or the medical resource in the outcome of the therapeutic procedure.

6. The method of claim 1, wherein the receiving (502), from a user input device (164), a first user input identifying a medical professional (1120) further comprises: receiving (502), from a user input device (164), a selection of a subset of medical professionals (1120), and wherein the structured data is restricted to data associated with the subset of medical professionals.

7. The method of claim 1, wherein a large language model (155) preprocesses the first data, the second data, and the third data to form a structured dataset.

8. The method of claim 1, wherein the first data comprises biographic information about the medical professional (920), wherein the second data comprises medical records of the patient (12), and wherein the third data comprises at least one of available medical tool or available medical resources at the medical facility (124).

9. A system (400), comprising: a processor circuit (250) configured to: receive (502), from a user input device (164), a first user input identifying a medical professional (1115); retrieve (504), from one of more databases (415), first data representative of the medical professional in response to the first user input;receive (502), from the user input device (164), a second user input identifying a patient (1105); retrieve (504), from the one or more databases (415), second data representative of the patient in response to the second user input; retrieve (504), from the one or more databases (415), third data associated with a medical facility at which the medical professional can perform a therapeutic procedure on the patient; pre-process (506) the first data, the second data, and the third data to form a structured data set; provide (508) the structured data set as an input to a predictive network (152); generate (510), as an output of the predictive network, a procedure plan (425) for the therapeutic procedure, wherein the procedure plan comprises at least one of a medical tool or a medical resource to be utilized by the medical professional to perform the therapeutic procedure; and output (514), to a display (166), a screen display (1205) comprising at least one of the medical tool or the medical resource.

10. The system of claim 9, wherein the processor circuit is further configured to: generate (510), as an output of the predictive network (152), a predicted duration(420) of the therapeutic procedure.

11. The system of claim 10, wherein the procedure plan (425) comprises a series of sub-steps, and wherein the predicted duration (420) comprises a plurality of sub-step durations, one sub-step duration for each sub-step in the procedure plan.

12. The system of claim 9, wherein the procedure plan (425) comprises a plurality of probabilities, each probability in the plurality of probabilities associated with the medical tool or the medical resource, wherein each probability represents a likelihood of use of the medical tool or the medical resource in the therapeutic procedure.

13. The system of claim 12, wherein the procedure plan (425) comprises a one or more indexes of criticality, each index of criticality associated with the medical tool or the medicalresource, wherein each index of criticality represents the importance of the medical tool or the medical resource in the outcome of the therapeutic procedure.

14. The system of claim 9, wherein the processor circuit is further configured to: receive (502), from a user input device (164), a selection of a subset of medical professionals (1120), and wherein the structured data is restricted to data associated with the subset of medical professionals.

15. The system of claim 9, wherein a large language model (155) preprocesses the first data, the second data, and the third data to form a structured dataset.

16. The system of claim 9, wherein the first data comprises biographic information about the medical professional (920), wherein the second data comprises medical records of the patient (120), and wherein the third data comprises at least one of available medical tool or available medical resources at the medical facility.

17. A non-transitory machine-readable medium comprising a plurality of machineexecutable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising: receiving (502), from a user input device (164), a first user input identifying a medical professional (1115); retrieving (504), from one of more databases (415), first data representative of the medical professional in response to the first user input; receiving (502), from the user input device (164), a second user input identifying a patient (1105); retrieving (504), from the one or more databases (415), second data representative of the patient in response to the second user input; retrieving (504), from the one or more databases (415), third data associated with a medical facility at which the medical professional can perform a therapeutic procedure on the patient;pre-processing (506) the first data, the second data, and the third data to form a structured data set; providing (508) the structured data set as an input to a predictive network (152); generating (510), as an output of the predictive network, a procedure plan (425) for the therapeutic procedure, wherein the procedure plan comprises at least one of a medical tool or a medical resource to be utilized by the medical professional to perform the therapeutic procedure; and outputting (514), to a display (166), a screen display (1205) comprising at least one of the medical tool or the medical resource.

18. The non-transitory machine-readable medium of claim 17, wherein the one or more processors are further caused to perform operations comprising: generating (510), as an output of the predictive network (152), a predicted duration (420) of the therapeutic procedure.

19. The non-transitory machine-readable medium of claim 18, wherein the procedure plan (425) comprises a series of sub-steps, and wherein the predicted duration (420) comprises a plurality of sub-step durations, one sub-step duration for each sub-step in the procedure plan.

20. The non-transitory machine-readable medium of claim 17, wherein the procedure plan (425) comprises a plurality of probabilities, each probability in the plurality of probabilities associated with the medical tool or the medical resource, wherein each probability represents a likelihood of use of the medical tool or the medical resource in the therapeutic procedure.