Automatic identification of anatomical checkpoints for system control, and associated devices, sytems, and methods

Abstract

An automatic checkpoint identification system receives a user input including patient health information or pre-operative images. The system generates, as an output of a patient model estimator, a 3D patient model with one or more anatomical checkpoints from the user input. The system receives, from an external imaging system, one or more imaging frames tracking a medical procedure. The system detects, by a controller, progress of the medical procedure based on a proximity of an intravascular device to one of the one or more anatomical checkpoints. The system generates, by a controller, an action recommendation based on the proximity.

Description

AUTOMATIC IDENTIFICATION OF ANATOMICAL CHECKPOINTS FOR SYSTEM CONTROL, AND ASSOCIATED DEVICES, SYTEMS, AND METHODSFIELD

[0001] The present disclosure relates generally to endovascular procedures (e.g., balloon angioplasty, stenting, etc.). In particular, to automatic identification of anatomical checkpoints for system control during endovascular procedures, e.g., to provide userguidance during an endovascular procedure.BACKGROUND

[0002] Endovascular procedures are typically performed with the support of several systems that are designed to provide different utilities at different phases of the procedure. The main system typically used to guide these procedures is an imaging system (e.g., X-ray imaging system). Different procedure phases that are typically encountered at different anatomical locations require different X-ray imaging settings including different X-ray dose, different amounts of collimation, and so on. Similarly, in robot-assisted procedures, different phases require different robot maneuvers, movements at different speeds, etc. Various clinical personnel must be available to adjust these parameters on systems in order to optimally and safely use the systems during a procedure. Adjusting system parameters through the various phases of the procedure can distract clinical personnel from their focus on the patient and suboptimal system parameters can make procedure steps difficult to complete (e.g., the wrong imaging protocol can generate poor quality images making navigation or treatment delivery difficult). Thus, configuring and reconfiguring various support systems can be challenging.

[0003] The information included in this Background section of the specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and is not to be regarded as subject matter by which the scope of the disclosure is to be bound.SUMMARY

[0004] Disclosed is an automatic checkpoint identification system. Patient health information and / or pre-operation images are received from a user, i.e., a medical professional or interventionalist. A patient model with anatomical checkpoints identified is generated by a patient model estimator. The patient model with anatomical checkpoints may be registered with an image from a procedure support system, e.g., x-ray imaging frames. An action controller may detect / determine when an endovascular procedure is nearing one of the anatomical checkpoints and generate a notification for a user. The notification may include a suggested action item, e.g., reduce x-ray dosage, change collimation settings, reduce navigation speed of the endovascular device, etc. In some instances, upon detecting the proximity of an endovascular device to an anatomical checkpoint, the action controller may automatically adjust the settings of the imaging support system and / or the endovascular medical device system, while also notifying a user of the changes.

[0005] This automatic checkpoint identification system disclosed herein has particular, but not exclusive, utility for supporting endovascular procedures. The automatic checkpoint identification system receives patient health information and automatically generates patient models with relevant anatomical checkpoints that may be used to monitor the progress of an endovascular procedure. The automatic checkpoint identification system advantageously updates system settings, including support systems such as x-ray imaging, thus reducing patient exposure to radiation. Also, the automatic checkpoint identification system advantageously alerts users to the proximity to an anatomical checkpoint where an endovascular device has an increased likelihood of deviating from the planned path through the vasculature, thus allowing users to take action to prevent deviation (e.g., update X-ray view). Also, the automatic checkpoint identification system advantageously alerts users to the proximity to an anatomical checkpoint where endovascular devices must be exchanged, thus allowing staff time to prepare new devices.

[0006] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0007] In one general aspect, the present disclosure is directed to a computer-implemented method. The computer-implemented method also includes receiving, from a user input device, a user input including patient health information or pre-operative images; generating, as an output of a patient model estimator, a 3D patient model with one or more anatomical checkpoints from the user input. The computer-implemented method also includes receiving, from an external imaging system, one or more imaging frames tracking a medical procedure. The computer-implemented method also includes detecting, by a controller, progress of the medical procedure based on a proximity of an intravascular device to one of the one or more anatomical checkpoints. The computer-implemented method also includes generating, by a controller, an action recommendation based on the proximity.

[0008] In some aspects, implementations may include one or more of the following features. The method may include: registering the one or more imaging frames with the 3D patient model; and outputting, to a display, a screen display that may include the one or more imaging frames, a notification corresponding to the action recommendation, and an anatomical checkpoint. The imaging frames may include X-ray frames, and where the action recommendation may include changing at least one of a navigation speed, an X-ray dosage, or X-ray collimation setting. The method may include: generating, by the patient model estimator, an updated 3D patient model from the user input and at least one of the one or more imaging frames. The method may include: automatically controlling, by the controller, the external imaging system or the intravascular device based on the action recommendation.

[0009] In some aspects, implementations may include one or more of the following features. The generating the 3D patient model further may include: generating, via the patient model estimator, the 3D patient model from one or more shape features, where the shape features include a mean patient shape and one or more shape variation modes; and mapping, via the patient model estimator, one or more reference checkpoints on the mean patient shape to the one or more anatomical checkpoints on the 3D patient model. The patient model estimator that may include a neural network-based variational autoencoder and where the generating the 3D patient model with the one or more anatomical checkpoints further that may include: generating, via the patient model estimator, a sample from a latent shape distribution parametrized by one or more shape features, where the shape features include a mean and a variance; and generating, as an output of a decoder in the neural network-based variational autoencoder, the 3D patient model from the sample and the user input; performing registration of the 3D patient model and a mean patient shape including one or more reference checkpoints; and mapping, via the patient model estimator, the one or morereference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model and the mean patient shape. The patient model estimator may include a neural network-based diffusion model and where the generating the 3D patient model with the one or more anatomical checkpoints further that may include: generating, via the neural network-based diffusion model, a 3D patient model from a noisy mean patient shape and the user input; performing registration of the 3D patient model and a mean patient shape including one or more reference checkpoints; and mapping, via the patient model estimator, the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model and the mean patient shape.

[0010] In one general aspect, the present disclosure is directed to a system a processor circuit configured to: receive, from a user input device, a user input including patient health information or pre-operative images; generate, as an output of a patient model estimator, a 3D patient model with one or more anatomical checkpoints from the user input; receive, from an external imaging system, one or more imaging frames tracking a medical procedure; detect, by a controller, progress of the medical procedure based on a proximity of an intravascular device to one of the one or more anatomical checkpoints; and generate, by a controller, an action recommendation based on the proximity.

[0011] In some aspects, implementations may include one or more of the following features. The system where the processor circuit is further configured to: register the one or more imaging frames with the 3D patient model; and output, to a display, a screen display that may include the one or more imaging frames, a notification corresponding to the action recommendation, and an anatomical checkpoint. The imaging frames may include X-ray frames, and where the action recommendation that may include changing at least one of a navigation speed, an X-ray dosage, or X-ray collimation setting. The processor circuit is further configured to: generate, by the patient model estimator, an updated 3D patient model from the user input and at least one of the one or more imaging frames. The processor circuit is further configured to: automatically control, by the controller, the external imaging system or the intravascular device based on the action recommendation.

[0012] In some aspects, implementations may include one or more of the following features. The generating the 3D patient model further may include: generating, via the patient model estimator, the 3D patient model from one or more shape features, where the shape features include a mean patient shape and one or more shape variation modes; and mapping, via the patient model estimator, one or more reference checkpoints on the mean patient shape to theone or more anatomical checkpoints on the 3D patient model. The patient model estimator may include a neural network-based variational autoencoder and where the generating the 3D patient model with the one or more anatomical checkpoints further that may include: generating, via the patient model estimator, a sample from a latent shape distribution parametrized by one or more shape features, where the shape features include a mean and a variance; and generating, as an output of a decoder in the neural network-based variational autoencoder, the 3D patient model from the sample and the user input; performing registration of the 3D patient model and a mean patient shape including one or more reference checkpoints; and mapping, via the patient model estimator, the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model and the mean patient shape. The patient model estimator may include a neural network-based diffusion model and where the generating the 3D patient model with the one or more anatomical checkpoints further that may include: generating, via the neural network-based diffusion model, a 3D patient model from a noisy mean patient shape and the user input; performing registration of the 3D patient model and a mean patient shape including one or more reference checkpoints; and mapping, via the patient model estimator, the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3d patient model and the mean patient shape.

[0013] In one general aspect, the present disclosure is directed to a non-transitory machine- readable medium that may include a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations including receiving, from a user input device, a user input including patient health information or pre-operative images; generating, as an output of a patient model estimator, a 3D patient model with one or more anatomical checkpoints from the user input. The non-transitory machine-readable medium also includes receiving, from an external imaging system, one or more imaging frames tracking a medical procedure. The non- transitory machine-readable medium also includes detecting, by a controller, progress of the medical procedure based on a proximity of an intravascular device to one of the one or more anatomical checkpoints. The non-transitory machine-readable medium also includes generating, by a controller, an action recommendation based on the proximity.

[0014] In some aspects, implementations may include one or more of the following features. The non-transitory machine-readable medium where the one or more processors are further caused to perform operations that may include: registering the one or more imaging frameswith the 3D patient model; and outputting, to a display, a screen display that may include the one or more imaging frames, a notification corresponding to the action recommendation, and an anatomical checkpoint. The imaging frames that may include X-ray frames, and where the action recommendation may include changing at least one of a navigation speed, an X-ray dosage, or X-ray collimation setting. The one or more processors are further caused to perform operations that may include: generating, by the patient model estimator, an updated 3D patient model from the user input and at least one of the one or more imaging frames. The one or more processors are further caused to perform operations that may include: automatically controlling, by the controller, the external imaging system or the intravascular device based on the action recommendation.

[0015] In some aspects, implementations may include one or more of the following features. The generating the 3D patient model further that may include: generating, via the patient model estimator, the 3D patient model from one or more shape features, where the shape features include a mean patient shape and one or more shape variation modes; and mapping, via the patient model estimator, one or more reference checkpoints on the mean patient shape to the one or more anatomical checkpoints on the 3D patient model. The patient model estimator may include a neural network-based variational autoencoder and where the generating the 3D patient model with the one or more anatomical checkpoints further that may include: generating, via the patient model estimator, a sample from a latent shape distribution parametrized by one or more shape features, where the shape features include a mean and a variance; and generating, as an output of a decoder in the neural network-based variational autoencoder, the 3D patient model from the sample and the user input; performing registration of the 3D patient model and a mean patient shape including one or more reference checkpoints; and mapping, via the patient model estimator, the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model and the mean patient shape. The patient model estimator may include a neural network-based diffusion model and where the generating the 3D patient model with the one or more anatomical checkpoints further that may include: generating, via the neural network-based diffusion model, a 3D patient model from a noisy mean patient shape and the user input; performing registration of the 3D patient model and a mean patient shape including one or more reference checkpoints; and mapping, via the patient model estimator, the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model and the mean patient shape.

[0016] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. A more extensive presentation of features, details, utilities, and advantages of the automatic measurement point detection system, as defined in the claims, is provided in the following written description of various aspects of the disclosure and illustrated in the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Illustrative aspects of the present disclosure will be described with reference to the accompanying drawings, of which:

[0018] Figure l is a schematic diagram of a networked system for automatic checkpoint identification, according to aspects of the present disclosure.

[0019] Figure l is a schematic diagram of a processor circuit, according to aspects of the present disclosure.

[0020] Figure 3 is a schematic diagram of a deep learning network configuration, according to aspects of the present disclosure.

[0021] Figure 4 illustrates a portion of human vasculature.

[0022] Figure 5 is a schematic diagram of at least a portion of an automatic checkpoint identification system, according to aspects of the present disclosure.

[0023] Figure 6 is a schematic diagram of at least a portion of a training system for a patient model estimator, according to aspects of the present disclosure.

[0024] Figure 7 is a second diagrammatic schematic view of a training system for an untrained patient model estimator, according to aspects of the present disclosure.

[0025] Figure 8 is a diagram of a patient model estimator, according to aspects of the present disclosure.

[0026] Figure 9 is a schematic diagram of a variational autoencoder, according to aspects of the present disclosure.

[0027] Figure 10 depicts a simplified example control dictionary, according to aspects of the present disclosure.

[0028] Figure 11 depicts operation of the action controller during an endovascular procedure, according to aspects of the present disclosure.

[0029] Figure 12 is a schematic flow diagram of a method for automatic checkpoint identification, according to aspects of the present disclosure.

[0030] Figure 13 is an example display for displaying the output of an action controller, according to aspects of the present disclosure.DETAILED DESCRIPTION

[0032] In accordance with at least one aspect of the present disclosure, an automatic checkpoint identification system is provided that automatically generates patient body models with anatomical checkpoints identified. This may allow, for example, users (including medical professionals and other medical personnel) to be notified when a checkpoint is reached during an endovascular procedure or other procedures. Furthermore, the system provides an automatic way to notify users of the proximity to a checkpoint and suggest adjustment of settings of various systems utilized during a procedure, e.g., endovascular devices and x-ray imaging support systems. In some instances, a controller directly controls system settings instead of providing a recommendation for a user to act on.

[0033] These descriptions are provided for exemplary purposes only, and should not be considered to limit the scope of the automatic checkpoint identification system. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.

[0034] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the aspects illustrated in the drawings, and specific language will be used to describe the same. It is nevertheless understood that no limitation to the scope of the disclosure is intended. Any alterations and further modifications to the described devices, systems, and methods, and any further application of the principles of the present disclosure are fully contemplated and included within the present disclosure as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one aspect may be combined with the features, components, and / or steps described with respect to other aspects of the present disclosure. For the sake of brevity, however, the numerous iterations of these combinations will not be described separately. As used herein, accessing can include querying, retrieving, sorting, etc.

[0035] The systems and methods disclosed herein provide a number of benefits. During an endovascular procedure, a patient may be exposed to considerable radiation doses from an imaging support system, e.g., X-rays. Thus, the systems and methods disclosed herein improve the control of system settings that allow for effective procedure while also reducing radiation exposure. Furthermore, reduces the cognitive demands of the medical devices and supporting imaging systems on medical personnel, allowing improved focus on the patient and other aspects of the procedure.

[0036] As used herein, registration may include establishing a spatial relationship / correspondence between the generated patient model and the mean patient model such that a location in the generated patient model matches a corresponding location in the mean patient model (and vice versa) or between the patient model and x-ray image(s) such that a location in the patient model matches a corresponding location in the x-ray image(s) and vice versa.

[0037] Figure 1 is a schematic diagram of a networked system 100 for automatic checkpoint identification, according to aspects of the present disclosure. The networked system 100 may, for example, may be used to receive patient health information from a user. Patient health information may be transmitted to different components of the networked system 100 to generate a patient model and register the patient model with images from a medical device console 140. The networked system 100 may provide a user with a notification and / or a recommended action based on the real-time analysis of an endovascular procedure and external imaging that has been registered with a patient body model.

[0038] The networked system 100 is used for generating a notification for a user, recommending an action to a user or automatically controlling external imaging system settings and / or medical device settings during a procedure. The networked system 100 may include a network / cloud computers 110, medical device console 140, medical device 150, user computer 160, medical imaging console 170, and medical imaging device 180. As depicted, each of the network / cloud computers 110, medical device console 140, user computer 160, medical imaging console 170, may be in communication with each other. As described herein, communication between the different components may be accomplished by any number of connections, e.g., wired and / or wireless.

[0039] Network / cloud computer 110 may include a processor 112, input device 114, display 116, communication interface 117, and memory 118.

[0040] Processor 112 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 112 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 112 is configured to process the instructions stored in memory 118. The processor 112 is connected to the communication interface 117.

[0041] Input device 114 allows a user to make selections or provide instructions to the network / cloud computer 110. The input device, may be a mouse, touch screen, touch pad etc.

[0042] The display 116 is coupled to the processor 112. The display 116 may be a monitor or any suitable display. The display 116 is configured to patient model estimator 120 output, patient information 122, information in training database 124, preoperative images 126, network performance, or any other system diagnostic information.

[0043] The communication interface 117 is coupled to the processor 112. The communication interface 117 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 117 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 117 can be referred to as a communication device or a communication interface module.

[0044] The memory 118 is coupled to the processor 112. The memory 118 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 112), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0045] The memory 118 can be configured to store patient information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to X-ray or ultrasound images, X-ray or ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical field of view, a device orientation, and / or the subject position during an imaging procedure. The memory 118 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting / segmenting anatomy, image quantification algorithms, and / or procedure guidance algorithms, including those described herein.

[0046] In some instances, memory 118 includes data and stored instructions for modules, including patient model estimator 120. Patient model estimator 120 may utilize patient health information and other patient-related images to generate a 3D model of the patient. For example, using a patient’s age, height, and weight, the patient model estimator 120 may generate a tessellated surface which serves as a model of a patient’s body shape.

[0047] Medical device console 140 may include a processor 142, input device 144, display 146, communication interface 147, and memory 148.

[0048] Processor 142 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 142 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 142 is configured to process the instructions stored in memory 148. The processor 142 is connected to the communication interface 147.

[0049] Input device 144 allows a user to make selections or provide instructions to the medical device console 140. The input device, may be a mouse, touch-screen, touch pad etc. With the input device 144, a user may control device setting such as a robot speed, where robotic control of the medical device 150 is employed. For example, robotic control may control the navigation speed of a catheter through vasculature.

[0050] The display 146 is coupled to the processor 142. The display 146 may be a monitor or any suitable display. The display 146 is configured to display any of the outputs generated within the network system 100, including patient model, interoperative images with anatomical checkpoint, medical imaging device settings, or any other settings associated with a procedure.

[0051] The communication interface 147 is coupled to the processor 142. The communication interface 147 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 147 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 147 can be referred to as a communication device or a communication interface module.

[0052] The memory 148 is coupled to the processor 142. The memory 148 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 142), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0053] The memory 148 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to X-ray or ultrasound images, X-ray or ultrasound videos, X-ray data in case of fiber optic or any number of guidewire / catheter- based procedures / technol ogies, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical field of view, a device orientation, and / or the subject position during an imaging procedure. The memory 148 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting anatomy, segmenting anatomy (e.g., defining contours of the anatomy), image quantification algorithms, and / or procedure guidance algorithms, including those described herein.

[0054] In some instances, memory 148 includes data and stored instructions for modules, including patient model estimator 120 and action controller 520. As depicted in Fig. 1, the patient model estimator 120 is located on the network / cloud computer 110; however, it may also be stored on medical device console 140, user computer 160, and medical imaging console 170. During automatic checkpoint identification, data and / or images may be sent from the medical imaging console 170 to the user computer 160, network / cloud computer 110, and / or medical device console 140. Patient model estimator 120 and action controller 520 may generate patient models, checkpoints, and / or action recommendation for the received data which may then be displayed on a display device or be sent back to one or more of the computers and devices of networked system 100 (e.g., to the medical imaging console 170 to change the imaging dose or framerate).

[0055] The patient model estimator 120 and action controller 520 may be any statistical model, machine learning model, neural network-based model or rule-based programming model, e.g., as described with respect to and as depicted in Figs. 3 and 9. For example, patient model estimator 120 and action controller 520 can implement Convolutional Neural Networks (CNN), variational autoencoders (VAEs), decision tree models, support vector machines (SVM), generative image-to-image transformers, generative image-to-text transformers, or other machine learning models.

[0056] Medical device 150 may be used in procedures for treatment or imaging, including endovascular procedures, e.g., stenting. Medical device 150 may be devices used under X-ray guidance such as guidewires and catheters, modified devices such as guidewires or catheters with embedded optical fibers in the case of shape sensing (e.g., Philips Fiber Optic Real Shape or FORS), etc. The medical device may be in communication with a medical device console 140, e.g., through the communication interface 147. When medical device 150 is an imaging device, the medical images generated by the medical device 150 may be shown on the display 146. Medical images may also be communicated to any of the computers, networks / clouds 110, 160, 170.

[0057] User computer 160 may include a processor 162, input device 164, display 166, communication interface 167, and memory 168.

[0058] Processor 162 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 162 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 162 is configured to process the instructions stored in memory 168. The processor 162 is connected to the communication interface 167.

[0059] Input device 164 allows a user to make selections or provide instructions to the user computer 160. The input device, may be a mouse, touch-screen, touch pad, etc.

[0060] The display 166 is coupled to the processor 162. The display 166 may be a monitor or any suitable display. The display 166 is configured to display any of the outputs generated within the network system 100, including patient model, interoperative images with anatomical checkpoint, medical imaging device settings, or any other settings associated with a procedure.

[0061] The communication interface 167 is coupled to the processor 162. The communication interface 167 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 167 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 167 can be referred to as a communication device or a communication interface module.

[0062] The memory 168 is coupled to the processor 162. The memory 168 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 162), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0063] The memory 168 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to X-ray or ultrasound images, X-ray or ultrasound videos, X-ray data in case of fiber optic or any number of guidewire / catheter- based procedures / technol ogies, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical field of view, a device orientation, and / or the subject position during an imaging procedure. The memory 168 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting anatomy, segmenting anatomy, image quantification algorithms, and / or procedure guidance algorithms, including those described herein.

[0064] Medical imaging console 170 may include a processor 172, input device 174, display 176, communication interface 177, and memory 178.

[0065] Processor 172 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardwaredevice, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 172 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 172 is configured to process the instructions stored in memory 178. The processor 172 is connected to the communication interface 177.

[0066] Input device 174 allows a user to make selections or provide instructions to the medical imaging console 170. The input device, may be a mouse, touch-screen, touch pad, etc. For example, a user may input different imaging settings to control X-ray dosage, collimation settings, framerate, etc.

[0067] The display 176 is coupled to the processor 172. The display 176 may be a monitor or any suitable display. The display 176 configured to display any of the outputs generated within the network system 100, including patient model, interoperative images with anatomical checkpoint, medical imaging device settings, or any other settings associated with a procedure.

[0068] The communication interface 177 is coupled to the processor 172. The communication interface 177 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 177 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals to other devices in the networked system 100. The communication interface 177 can be referred to as a communication device or a communication interface module.

[0069] The memory 178 is coupled to the processor 172. The memory 178 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 172), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0070] The memory 178 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. Subject information may include measurements, data, files,other forms of medical history, such as but not limited to X-ray or ultrasound images, X-ray or ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical field of view, a device orientation, and / or the subject position during an imaging procedure. The memory 178 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting anatomy, segmenting anatomy, image quantification algorithms, and / or procedure guidance algorithms, including those described herein.

[0071] Medical imaging device 180 may generate medical images, e.g., X-ray images. The medical imaging device is in communication with medical imaging console 170, e.g., through the communication interface 177. The medical images generated by data from the medical imaging device 180 may be shown on the display 176. Medical images 185 may also be communicated to any of the networks / clouds, consoles, computers 110, 140, 160. In some aspects, medical imaging device may include fixed monoplane / biplane C-arm X-ray imaging system, mobile C-arm X-ray imaging system, etc. In some instances, the C-arm can be tracked and used with interoperative images to update checkpoints and patient models.

[0072] In some aspects, aspects of the present disclosure can be implemented with medical images of patients obtained using any suitable medical imaging device 180 and / or modality. Examples of medical images and medical imaging devices include x-ray images (angiographic images, fluoroscopic images, images with or without contrast) obtained by a medical imaging device such as an X-ray imaging device, computed tomography (CT) images obtained by a CT imaging device, positron emission tomography-computed tomography (PET-CT) images obtained by a PET-CT imaging device, magnetic resonance images (MRI) obtained by an MRI imaging device, single-photon emission computed tomography (SPECT) images obtained by a SPECT imaging device, optical coherence tomography (OCT) images obtained by an OCT imaging device, and ultrasound images obtained by an ultrasound imaging device. The medical imaging device 180 can obtain the medical images while positioned outside the subject body, spaced from the subject body, adjacent to the subject body, in contact with the subject body, and / or inside the subject body.

[0073] Training database 124 may be stored on network / cloud computer 110 or any other network or cloud computing environment. In some aspects, the systems described herein may be provided as a cloud service. Training dataset comprises historic information, which may include preoperative images and / or measurements, interoperative images and / ormeasurements, ground truth patient models with user-annotated checkpoints, patient health information (e.g., patient electronic medical records (EMR), age, sex, weight, etc.), interventionalist details, and tools, resources, and methods used for the intervention.

[0074] It should be noted that the examples described above are provided for purposes of illustration, and are not intended to be limiting. Other devices and / or device configurations may be utilized to carry out the operations described herein.

[0075] Figure l is a schematic diagram of a processor circuit 250, according to aspects of the present disclosure. The processor circuit 250 may be implemented in the network / cloud computer 110, medical device console 140, user computer 160, medical imaging console 170, or other devices or workstations (e.g., third-party workstations, network routers, etc.), or on a cloud processor or other remote processing unit, as necessary to implement the method. As shown, the processor circuit 250 may include a processor 260, a memory 264, and a communication interface 268. These elements may be in direct or indirect communication with each other, for example via one or more buses.

[0076] The processor 260 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a controller, or any combination of general-purpose computing devices, reduced instruction set computing (RISC) devices, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other related logic devices, including mechanical and quantum computers. The processor 260 may also comprise another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 260 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0077] The memory 264 may include a cache memory (e.g., a cache memory of the processor 260), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, other forms of volatile and non-volatile memory, or a combination of different types of memory. In an aspect, the memory 264 includes a non-transitory computer-readable medium. The memory 264 may store instructions 266. The instructions 266 may include instructions that, when executed by the processor 260, cause the processor 260 to perform the operations described herein. Instructions 266 may also be referred to as code. The terms “instructions” and “code” shouldbe interpreted broadly to include any type of computer-readable statement(s). For example, the terms “instructions” and “code” may refer to one or more programs, routines, subroutines, functions, procedures, etc. “Instructions” and “code” may include a single computer-readable statement or many computer-readable statements.

[0078] The communication interface 268 can include any electronic circuitry and / or logic circuitry to facilitate direct or indirect communication of data between the processor circuit 250, and other processors or devices. In that regard, the communication interface 268 can be an input / output (I / O) device. In some instances, the communication interface 268 facilitates direct or indirect communication between various elements of the processor circuit 250 and / or computers of networked system 100. The communication interface 268 may communicate within the processor circuit 250 through numerous methods or protocols. Serial communication protocols may include but are not limited to United States Serial Protocol Interface (US SPI), Inter-Integrated Circuit (I2C), Recommended Standard 232 (RS- 232), RS-485, Controller Area Network (CAN), Ethernet, Aeronautical Radio, Incorporated 429 (ARINC 429), MODBUS, Military Standard 1553 (MIL-STD-1553), or any other suitable method or protocol. Parallel protocols include but are not limited to Industry Standard Architecture (ISA), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI), Institute of Electrical and Electronics Engineers 488 (IEEE-488), IEEE-1284, and other suitable protocols. Where appropriate, serial and parallel communications may be bridged by a Universal Asynchronous Receiver Transmitter (UART), Universal Synchronous Receiver Transmitter (USART), or other appropriate subsystems.

[0079] External communication (including but not limited to software updates, firmware updates, model sharing between the processor and central server, or outputs generated by the model selection and prediction system described herein) may be accomplished using any suitable wireless or wired communication technology, such as a cable interface such as a universal serial bus (USB), micro USB, Lightning, or FireWire interface, Bluetooth, Wi-Fi, ZigBee, Li-Fi, or cellular data connections such as 2G / GSM (global system for mobiles) , 3G / UMTS (universal mobile telecommunications system), 4G, long term evolution (LTE), WiMax, or 5G. For example, a Bluetooth Low Energy (BLE) radio can be used to establish connectivity with a cloud service, for transmission of data, and for receipt of software patches. The controller may be configured to communicate with a remote server, or a local device such as a laptop, tablet, or handheld device, or may include a display capable ofshowing status variables and other information. Information may also be transferred on physical media such as a USB flash drive or memory stick.

[0080] Figure 3 is a schematic diagram of a deep learning network configuration, according to aspects of the present disclosure. The configuration 300 can be implemented by a deep learning network. The configuration 300 includes a deep learning network 310, which may include one or more CNNs 312. The CNN 312 is one example of a type of predictive model and / or machine learning model, which may be used in patient model estimator 120. For simplicity of illustration and discussion, Fig. 3 illustrates one CNN 312. However, any suitable number of CNNs 312 (e.g., about 2, 3 or more) may be included. The configuration 300 can be trained for identification of various anatomy (organs, tissue, bone) and / or other features (natural and / or man-made) within a patient anatomy. The configuration 300 can be further trained for segmenting human anatomy, diagnosis of medical conditions or any number of other diagnostic or medical tasks.

[0081] The CNN 312 may include a set of N convolutional layers 320 followed by a set of K fully connected layers 330, where N and K may be any positive integers. The convolutional layers 320 are shown as 320(1) to 320(N). The fully connected layers 330 are shown as 330(1) to 330(K). Each convolutional layer 320 may include a set of filters 322 configured to extract features from an input 302 (e.g., x-ray images or other data). The values N and K and the size of the filters 322 may vary depending on the use of the CNN. In some instances, the convolutional layers 320(1) to 320(N) and the fully connected layers 330(1) to 330(K-l) may be interspersed with rectified non-linear (ReLU) or leaky ReLU or other activation functions and / or batch normalization layers. The fully connected layers 330 may gradually shrink the high-dimensional output to a lower dimension or the dimension of the predicted result 340 (e.g., location for an object detection landmark, or location and dimension for an object detection box, or number of classes for a classification output). The fully connected layers 330 may also be referred to as a classifier. In some aspects, the fully convolutional layers 320 may additionally be referred to as representation or encodings or features.

[0082] When the prediction result 340 takes the form of classification output, it may indicate a confidence score (e.g., a probability) for each class 342 based on the input image 302. The classes 342 are shown as 342a, 342b, . . . , 342c. For example, when the CNN 312 is trained for classification of present anatomical features, the classes 342 may indicate a first anatomical feature class 342a, a second anatomical feature class 342b, a third anatomical feature class 342c, a fourth anatomical feature class 342d, or any other suitable class. A class 342 indicating a high confidence score indicates that the input image 302 or a section or pixelof the image 302 is likely to include an anatomical object / feature of the class 342. Conversely, a class 342 indicating a low confidence score indicates that the input image 302 or a section or pixel of the image 302 is unlikely to include an anatomical object / feature of the class 342.

[0083] The CNN 312 can also output a feature vector 350 at the output of the last convolutional layer 320(N), though any of the layers in the CNN are feature vectors. A feature vector 350 or encodings or representations may encode some representation of objects detected from the input medical image 302 or other data. For example, the feature vector 350 may encode representations of the patient model or of regions associated with anatomical checkpoints as identified from the image 302, which may be a patient medical image, e.g., an x-ray frame. The feature vector 350 may encode some representation of the pixels in a medical image associated with the location of anatomy associated with an anatomical checkpoint, e.g., a relevant subset of the vasculature, as described herein. These representations can be decoded using a reversed CNN where the fully connected layers expand the low-dimensional representation to a higher dimension and transposed convolutional layers can expand feature layers up to the size of the original input, where pixel-wise outputs (e.g., binary segmentation map, multi-class segmentation map) can be generated.

[0084] The deep learning network 310 may implement or include any suitable type of learning network. For example, in some aspects, and as described in relation to Fig. 3, the deep learning network 310 could include a convolutional neural network 312. In addition, the deep learning network 310 may additionally or alternatively be or include a multi-class classification network, an encoder-decoder type network, a fully connected deep learning network, or any suitable network or means of identifying features within an image.

[0085] In some aspects, when the deep learning network 310 includes a fully-connected neural network, the fully connected neural network may transform the data not related to a medical image or it may transform data derived from an image (e.g., detected objects) generated by another network, program, or human annotator. Data for detected objects may be concatenated into an layer (e.g., as an additional channel. For simple numeric features like age, weight, and other data - these can be concatenated into fully connected layers. For example, the fully connected neural network may transform information about a patient, such as age, weight, or other low-dimensional data.

[0086] In some aspects, when the deep learning network 310 includes an encoder-decoder network, the network may include two components. One component may be a constrictingcomponent or encoder, in which a large image, such as the image 302, may be convolved by several convolutional layers 320 such that the size of the image 302 changes in relation to the depth of the network layer. For instance, the CNN 312 may be the encoder. The image 302 may then be represented in a low dimensional space, or a flattened space. From this flattened space, an additional component or decoder may expand the flattened space to the original size of the image 302. For instance, the reverse of the CNN 312 may be the decoder. In some aspects, the encoder-decoder network may reconstruct the input image 302. In some aspects, the encoder-decoder network may segment the image 302 into patches. In some aspects of the present disclosure, the deep learning network 310 may include a multi-class classification network. In that aspect, the multi-class classification network may include an encoder path. For example, the image 302 may be a high dimensional image. The image 302 may then be processed with the convolutional layers 320 such that the size is reduced. The resulting low dimensional representation of the image 302 may be used to generate the feature vector 350 shown in Fig. 3. The low dimensional representation of the image 302 may additionally be used by the fully connected layers 330 to regress and output one or more classes 342. In some regards, the fully connected layers 330 may process the output of the convolutional layers 320. The fully connected layers 330 may additionally be referred to as task layers or regression layers, among other terms.

[0087] Any suitable combination or variations of the deep learning network 310 described is fully contemplated. For example, the deep learning network may include fully convolutional networks or layers or fully connected networks or layers or a combination of the two. In addition, the deep learning network may include a multi-class classification network, an encoder-decoder network, or any combination of networks. The process of training the deep learning network 310 includes adjusting its parameters, or more particularly the weights and biases, which control the operation of activation functions in the neural network. In supervised learning, the training process automatically adjusts the weights and the biases, such that when presented with the input data, the neural network accurately provides the corresponding expected output data. In order to do this, the value of the loss functions, or errors, are computed based on a difference between predicted output data and the expected output data. The value of the loss function may be computed using functions such as the negative log-likelihood loss, the mean squared error, or the Huber loss, or the cross-entropy loss. During training, the value of the loss function is typically minimized. Various methods are known for solving the loss minimization problem such as gradient descent, Quasi-Newton methods, and so forth. Various algorithms have been developed to implement these methodsand their variants including but not limited to Stochastic Gradient Descent “SGD”, batch gradient descent, mini-batch gradient descent, Gauss-Newton, Levenberg Marquardt, Momentum, Adam, Nadam, Adagrad, Adadelta, RMSProp, and Adamax “optimizers” These algorithms compute the derivative of the loss function with respect to the model parameters using the chain rule. This process is called backpropagation since derivatives are computed starting at the last layer or output layer, moving toward the first layer or input layer. These derivatives inform the algorithm how the model parameters must be adjusted in order to minimize the error function. The training process is performed iteratively by making adjustments to the weights and biases in each iteration. Training is terminated when the error, or difference between the predicted output data and the expected output data, is within an acceptable range for the training data, or for some validation data. Subsequently the neural network may be deployed, and the trained neural network makes predictions on new input data using the trained values of its parameters. If the training process was successful, the trained neural network accurately predicts the expected output data from the new input data.

[0088] Figure 4 illustrates a portion of human vasculature 400. Portions of the vasculature 400 may be diseased, requiring any number of different medical interventions. In some instances, a build-up of plaque may restrict flow within a portion of the vasculature 400. In such cases, an endovascular procedure may be required to restore flow. An endovascular procedure may begin with the insertion of a catheter into the vasculature at an access site 402, near the groin. From the access site 402, the catheter proceeds through the vasculature until it reaches the treatment area, i.e., the diseased tissue. Various anatomical checkpoints may be passed through along the way to the treatment site. For example, proceeding from the access site 402, the catheter may pass an iliac checkpoint 404, a renal checkpoint 406, an aortic arch checkpoint 405, a neck checkpoint 410, and a head checkpoint 412. These checkpoints are only exemplary, there may be fewer / more depending on the procedure and the location of the diseased tissue.

[0089] As depicted in Fig. 4, the anatomical checkpoints may mark areas where increased care must be taken in progressing the catheter. For example, branching vasculature may increase the probability of the catheter diverging from the planned path through the vasculature. The proximity to a certain anatomical checkpoint may inform how various parameters associated with the procedure should be altered. For example, external x-ray imaging is often employed for guidance purposes. Near the checkpoints, higher quality x-ray imaging may be needed, which could be achieved through a higher dose of radiation or higher framerate. Near such checkpoints, an alternative view of the anatomy may be needed,which is achieved through repositioning of the X-ray imaging arms. Other parameter settings such as collimation settings, i.e., changing the shape of the x-ray beam, and robot speed of a robot used to mechanically move the catheter through the vasculature may also be changed. Other parameter settings configured, monitored, and changed during an endovascular procedure are also contemplated by the present disclosure.

[0090] Figure 5 is a schematic diagram of at least a portion of an automatic checkpoint identification system 500, according to aspects of the present disclosure. Automatic checkpoint identification system 500 generates a patient model with checkpoints 512 and determines an action decision and / or recommendation(s) 525. User input (e.g., patient health information 504 and / or image-derived measurements 506) and selections may take the form of pre-operative medical images (i.e., images taken before the planned procedure), patient information, medical records, type of procedure and the output may be a patient model with checkpoints 512 and action decision and / or recommendation(s) 525 at each checkpoint. A user may be any of a number of medical professionals. In some instances, selections by a user may cause the system 500 to automatically gather additional information based on the selections from various databases, e.g., based on a patient identifier. The automatic checkpoint identification system 500 may include patient model estimator 120 and action controller 520. Though depicted as two distinct blocks, patient model estimator 120 and action controller 520 may comprise a single module.

[0091] The automatic checkpoint identification system 500 may receive patient measurements 502 from a medical device console 140, user computer 160, or medical imaging console 170. Patient measurements 502 may comprise patient health information 504 and / or image-derived measurements 506. In some instances, patient health information 504 may be entered by a user using a selection interface, allowing the doctor to identify a patient by a patient identifier so that patient health information may be retrieved from a database or allowing the doctor to directly enter patient health information. Patient health information may include patient height, weight, BMI, age, etc. In some instances, image-derived measurements 506 may be derived from any number of imaging modalities, e.g., ultrasound applications, camera-based videos, X-ray videos, and 3D volume images, such as computed tomography (CT) scans, 3D rotational angiography (3DRA) scans, magnetic resonance imaging (MRI) scans, optical coherence tomography (OCT) scans, or intravenous ultrasound (IVUS) pullback sequences. Imaging modalities where anatomy can be measured to scale may be particularly useful, e.g., CT, MRI, etc. Preoperative images from these modalities may include medical image reports with image-derived measurements already determined.Alternatively, pre-operative images may be passed through a convolution neural network or other image processing system to derive measurements from the images. For example, the convolutional neural network may detect objects and / or segment objects of known shape or measurements in the pre-operative image, which may be used to scale anatomical measurements.

[0092] Patient model estimator 120 may receive patient measurements 502. Patient model estimator 120 may include shape features 508. Patient model estimator 120 may comprise a number of different structures, including a linear / polynomial regression model, e.g., principal component analysis, or non-linear regression model, e.g., a neural network-based variational autoencoder, a neural network-based diffusion model. Shape features 508 may include learned shape features such as components describing the mean shape (P) or mean over learned representations (p) of shapes and modes of variation in shapes ( ) or variance over learned representations (o) of shapes. Shape features may vary depending on the type of patient model estimator. In some aspects, patient model estimator 120 may also receive interoperative images, measurements, and / or imaging settings 522, e.g., X-ray images, measurements derived from X-ray images, and settings of imaging device used for X-ray imaging (e.g., medical imaging console 170 and associated device 180). As X-ray images are acquired during a procedure, new measurements of the patient may become available (e.g., if a 3DRA of brain vasculature is acquired, the size of patient head may also become available). These new measurements may be used to update the estimated shape of the patient such that the new shape matches the new measurements in addition to the measurements that were available before. This allows the automatic checkpoint identification system to keep updating the estimated patient shape and checkpoints as more information (e.g., more measurements) becomes available. This is also applicable to other imaging modalities, e.g., CT in CT-guided procedures.

[0093] As depicted in Figure 8, patient model estimator 120 may receive patient health information 804 and measurements 806 derived from pre-operative images and / or images taken during the procedure (804 and 806 similar to 504 and 506 as depicted in Fig. 5). For example, measurements 806 giving the relative hip width of a patient to a known table width may be received by the patient model estimator 120. From these inputs, patient model estimator 120 may generate a 3D patient model 810 with a number of marked example anatomical checkpoints: access site 812, iliac 814, renal 816, aortic arch 818, neck 820, and head 822. Though identified on the patient model 810, the anatomical checkpoints areassociated with medical procedure-relevant checkpoints in the real vasculature / anatomy, e.g., as described with respect to Fig. 4.

[0094] In some aspects, patient model estimator 120 includes a linear / polynomial regression model. Using the regression model, patient model estimator 120 generates an estimate of the current patient shape from the measurements and learned shape features.

[0095] For instance, patient measurement measurements, X, along with principal component analysis coefficients, Y, can be used to estimate regression coefficients, B, via a linear regression model:Y = XB (1)

[0096] An estimate P of the patient shape, P, may be computed from patient measurement information using:P = P + YD = P + XBD (2)

[0097] where P represents the mean 3D shape computed over all shapes, D is a subset of eigenvectors corresponding to the n highest eigenvalues (representing the modes of variation in the 3D shape dataset; also referred to as shape variation modes), and Y represents the n PCA coefficients (or mode weights that describe how each 3D shape varies from the mean shape). In some aspects, these features and parameters are the shape features 508. This is an example of how 3D shapes can be represented in a parametric space learned using PCA. The shape of a new patient (as well as checkpoints / landmarks) can be estimated within this parametric space. B may be estimated via linear or polynomial regression according to:B « B = (X'X')~1X'Y') (3)

[0098] The PCA coefficients, Y, for a new patient (without shape information) can be estimated, for instance, via gradient descent. A first estimate of Y may be made (e.g., an n vector with values close to zero so that the estimation is close to the mean shape P) and then iteratively improved based on errors between the known patient measurements, X, and estimated measurements, X, extracted from the patient shape, P, estimated using equation (2). As this is repeated, the estimated patient shape, P, improves in accuracy and, consequently, as do the checkpoints, described herein, estimated from the patient shape. The estimated Y can also be used to directly estimate patient shape using the first equality of equation (2), bypassing the estimation of B, with measurements, X, only used to iteratively improve theestimated Y and, therefore, also the patient shape, P, but not directly in the estimation of P, as in right-hand side of equation (2).

[0099] In some aspects, patient model estimator 120 may include one or more non-linear regression models or neural networks suitable for generating a patient model, e.g., a collection of vertices (and possibly triangles, squares, and other polygons, or a combination thereof, associated with a tessellation comprising the 3D patient model) representative of the shape of a patient. In some aspects, patient model estimator 120 includes a neural networkbased variational autoencoder (VAE), as described in Fig. 9.

[0100] Figure 9 is a schematic diagram of a variational autoencoder 900. Another example of learning 3D shape representation in a parametric space is by using neural network architectures such as a variational autoencoder 900. During training, a VAE 900 may encode a patient model 902 using an encoder 904. The encoder 904 encodes the patient model into a probability distribution 906, which may be sampled over the latent representation 908. The latent representation 908 may then be decoded using a decoder 910 to generate a reconstructed patient model 912. Such a model may be trained through updating the encoder and decoder to reproduce the original patient model 902 as the predicted output 912. Over the course of training, a VAE 900 learns a mean, p, and variance, o, i.e., learns a distribution 906 (over the latent representations describing the training data) from which a latent representation of the input shape 908, z~JV'(p, o), is sampled and decoded to reconstruct the input shape 912. In some aspects, mean, p, and variance, o, may be shape features 508. Patient measurements may be combined with learned features via concatenation with the sampled latent representation or with one of the earlier (e.g., linear layers) layers in the decoder 910. Once the distribution 906 has been learned and the decoder 910 has been trained, then they are used at inference to generate a patient model. In other words, patient health information and other measurements would be input, the distribution 906 would be sampled and the decoder, guided by the input, would generate a patient model.

[0101] Using the variational autoencoder components, p and o, a z may be sampled from the distribution (described by p and o) over the latent representation space and decoded along with the concatenated measurements, X, to decode a shape such that it matches the patient measurements, X. Patient measurements, X, include for instance preoperative patient measurements 502 and / or intraoperative patient measurements 522. An initial z may describe a shape close to the mean, p, since the patient shape is unknown. The decoder 910 may be fine-tuned via gradient descent (as described herein) using errors between the knownmeasurements, X, and the estimated measurements, X (e.g., height), from the estimated or decoded patient shape to iteratively improve the estimated shape. A relevant loss function such as mean squared error (MSE) or mean absolute error (MAE) may be used. The errors may be backpropagated through the decoder layers to update z such that a more accurate patient shape is estimated by decoding the updated z (i.e., error between X and X is smaller). The variational autoencoder architecture is further described with respect to Fig. 9. Neural network architectures are also described with respect to Fig. 3. The VAE used at inference may serve as a patient model estimator 120.

[0102] In some aspects, patient model estimator 120 includes a neural network-based diffusion model. Conditioned diffusion models may be used to estimate 3D patient shapes from measurements, where the generated output is conditioned on known patient measurements, X. Since the estimated patient shape should be constrained with the range of feasible human shapes, the shape estimate may start with a mean patient shape, P. In diffusion models, noise is added to an initial estimate (e.g., mean shape), which is then denoised. By conditioning the denoising on patient specific measurements, X, and using reconstruction loss (e.g., MSE, MAE, etc.) during training to compare the estimated and ground truth patient shape, the diffusion model can be trained to associate measurements with shape characteristics (e.g., height with the head-to-toe distance of the patient shape). The trained diffusion model may receive a noisy initial shape (e.g., noisy mean patient shape, P) along with patient measurements, X. The conditioned diffusion model may guide the diffusion sampling process (i.e., the process of estimating the noise to be removed) toward the conditioning information, X, and iteratively estimating the patient shape such that the error between the known measurements, X, and the estimated measurements, X, from the estimated patient shape are minimized. The known patient measurements may include and / or be derived from patient health information, pre-operative images / measurements, and / or interoperative images / measurements. All models (PCA, VAE, diffusion, etc.) described above could be modified to directly estimate checkpoints rather than patient shape.

[0103] Furthermore, patient model estimator 120 may include a checkpoint generator 510 to generate checkpoints with respect to a patient body model, e.g., anatomical checkpoints as described herein. Known reference checkpoints from the mean shape can be transferred to the estimated shape either using equation (2) if using PCA, or via registration with the mean shape if using VAE or diffusion models. The user may identify a first checkpoint (e.g., “access site” 402 in Fig. 4) and all other checkpoints may be estimated relative to the locationof this checkpoint. Whether the subsequent checkpoints have been reached may be inferred via system data, where system may include an imaging system such as a C-arm imaging system or a robotic system. For instance, if the C-arm or patient table has been translated by a distance, d, from the “access site” checkpoint which matches the distance between checkpoints “access site” and “iliac” (see Figs. 4 and 8), then the system can infer that “iliac” checkpoint has been reached. Alternatively, checkpoints can also be inferred by tracking device movement in X-ray images (e.g., using image processing or tracked devices including electromagnetic or EM tracking, shape sensing such as Philips Fiber Optics RealSense or FORS) or by tracking robot movement (if robot has pushed a device by a distance, d, for instance). In some instances, as X-ray images are acquired of an expected checkpoint, images can be analyzed to assess whether the checkpoint was where it was expected to be. For example, once X-ray images of the aortic arch are acquired, the now known distance a C-arm has traveled between the access site and the aortic arch can be compared with the predicted distance. If the error is larger than some threshold, then this measurement can be used in additional to previous measurements to refine the estimated shape of the patient as well as all subsequent checkpoints.

[0104] In some aspects, depth sensors (e.g., infrared or IR sensors, hyperspectral cameras), RGB cameras, or other sensors in the procedure room may also be used to estimate patient measurements and used in the 3D shape estimation. Patients are typically under drapes, however, so specific features such as the patient head or other features that are not draped may be used to estimate some measurements. Alternatively, a model may be trained to estimate patient measurements (or patient shape, directly) through the drapes.

[0105] In some aspects, a first checkpoint may also be identified using image processing techniques. For instance, if no checkpoints have been identified, then any X-ray images acquired may be classified into one of several access sites (e.g., left femoral, right femoral, left radial, right radial, etc.). Other checkpoints may be estimated relative to this checkpoint.

[0106] Action controller 520 may receive the patient model with anatomical checkpoints generated by the patient model estimator 120 and interoperative images, measurements, and / or image settings 522. Action controller 520 may comprise a dictionary of system control decisions to suggest or trigger tasks or actions relevant to the current procedure at each checkpoint. An example of a control dictionary is depicted in Fig. 10, described below.

[0107] Figure 10 depicts a simplified example control dictionary 1000, according to aspects of the present disclosure. Control dictionary includes conditions 1005 and associated actions 1010. In some aspects, conditions may include distance thresholds. For example, the firstcondition may be formulated as “within 2cm of iliac checkpoint,” and when this condition is satisfied, a recommendation or a control signal is generated to reduce dosage (e.g., of X-rays) or, relatedly, to reduce the framerate of X-ray imaging. In some aspects, the action controller 520 is able to compare the checkpoints generated with the patient model to the progress of a medical device through the vasculature by first registering the patient body model with the X- ray (or other intraoperative imaging that is being employed). Once registered, a distance may be calculated between the position of a portion of the medical device traversing the vasculature and the position of a checkpoint.

[0108] The following are examples of the conditions to be met for associated recommendations / actions. At the “iliac” checkpoint, e.g., 814, during a neurointerventional procedure, reduce dosage / framerate of X-ray image acquisition such that the image quality is sufficient to visualize the device, but anatomical features may not be very clear since images here are simply required to ensure navigation of devices is in the right direction but not for diagnosis of anatomical features or for treatment delivery. At the “iliac” checkpoint, e.g., 814, during a neurointerventional procedure, move vertical collimation shutters inward to cut off anatomy away from the vasculature (i.e., anatomy that does not need to be visualized for successful navigation). At the “iliac” checkpoint, e.g., 814, during a neurointerventional procedure, increase robot navigation speed since navigation through these large vessels is relatively easy. At the “iliac” checkpoint, e.g., 814, during an endovascular aneurysm repair (EVAR) procedure, decrease robot navigation speed to slowly approach “renal” checkpoint so that the renal artery can be cannulated. At the “renal” checkpoint, e.g., 816, during a neurointerventional procedure, maintain robot navigation speed (relative to “iliac” checkpoint) as long as device is moving in a straight line (i.e., device has not accidentally cannulated the renal artery). At the “aortic arch” checkpoint, e.g., 818, during a neurointerventional procedure, decrease robot navigation speed to carefully cannulate the appropriate branch (e.g., vertebral or carotid) from the arch. At the “head” checkpoint, e.g., 822, during a neurointerventional procedure, increase dosage of X-ray image acquisition since this is a critical region where treatment is delivered, and high image quality is required to administer optimal treatment. At the “head” checkpoint, e.g., 822, during a neurointerventional procedure, change collimation / digital zoom settings such that the imaging field of view is focused around the treatment target. These are merely a few examples; control dictionaries for other procedures and other imaging devices are also conceived of by the present disclosure.

[0109] Returning to Fig. 5, action controller 520 selects or generates one or more action decisions and one or more recommendations 525. Recommendations may be provided to a user by means of a display 530 When action controller 520 is configured to directly control and reconfigure the medical imaging devices and medical devices (e.g., 140, 150, 170, and / or 180 as depicted in Fig. 1), an action decision may contain instructions for changing parameter settings of various systems being utilized during the procedure, for instance, an X-ray imaging support system or the robot speed controlling the progress of the catheter traversing the vasculature. Even if the action controller is configured to control the systems associated with the procedure, a user may still be notified through a screen display of the action taken, possibly with an opportunity to override to change.

[0110] A screen display may be generated for display on any number of devices, e.g., displays 116, 146, 166, or 176. Screen display may include interoperative images 522, e.g., X-rays, and an action decision and / or recommendation 525, as described herein. An example screen display is provided in Fig. 13.[OHl] The automatic checkpoint identification system 500 may also estimate a confidence measure. Confidence may be related to the quality of the data (e.g., number of measurements provided with fewer measurements related to lower confidence, image quality with low dose images related to lower confidence, etc.). Confidence may also be computed using dropout or other techniques that may be used in a neural network implementation. Dropout ignores outputs from different nodes (chosen at random) in neural network layers during the different training epochs. At inference, data flows through the trained network multiple times where, again, different nodes are ignored during different inference runs generating different outputs. If the network is confident, the difference in outputs is small. However, if the network is not confident, the difference in outputs is large.

[0112] Figure 6 is a schematic diagram of at least a portion of a training system 600 for a patient model estimator 120, according to aspects of the present disclosure. Training system 600 trains the patient model estimator 120 to accurately predict patient models and checkpoints from the context provided at inference. Patient model estimator 120 is trained by providing known patient models 627 (e.g., 3D shape models made up of vertices and triangles) with user-annotated checkpoints 629 and associated patient health information 621, pre-operative images and / or measurements 623, and interoperative images and / or measurements 625. For example, pre-operative images could be CTs or MRIs and interoperative images may be X-rays or ultrasound scans. The known patient models 627 may be obtained from pre-operative images, e.g., volumetric images obtained from CT or MRIscans. Measurements associated with both preoperative and interoperative images may include various anatomical measurements discernible within the field-of-view of the image (e.g., the hip to table width ratio as depicted in Fig. 8). This information may be recorded from earlier procedures. It is not necessary that all types of information be available for each patient. Training system 600 includes patient model simulator or estimator 120 and model objectives / functions 650.

[0113] Training data 610 may be stored in the training database 124 of network / cloud computer 110. Training data 602 may include for each patient, one or more of a patient model 627 with user-annotated checkpoints 629, patient health information 621, pre-operative images / measurements 623, and interoperative images / measurements 625. The patient model 627 and user-annotated checkpoints 629 represent the ground truth that the patient model estimator 120 is attempting to predict / reproduce during the operation of the training system 600. User-annotated checkpoints may alternatively only be annotated on one patient model in the database or on the mean shape or mean patient model. Assuming all patient models are registered, these checkpoints can be transferred between all patient models.

[0114] Patient model estimator 120 receives the patient health information (e.g., patient EMR, age, sex, weight, height, etc.), preoperative images and / or measurements 623, and / or interoperative images and / or measurements 625 and generates a predicted 3D patient model and one or more predicted anatomical checkpoints. The predicted patient model and checkpoints 640 may be generated as described herein, and in particular with respect to Fig.5. Using model objectives / functions 650, the predicted model and checkpoints 640 may be compared with the ground truth patient model 627 and user-annotated checkpoints 629. Model objectives / functions 650 may include objectives / functions which penalize to a greater or lesser extent predicted patient models and checkpoints 640 which are further or closer to the ground patient model 627 and user-annotated checkpoints 629, respectively. In some aspects, the model objectives / functions 650 may be a mean squared error or mean absolute error.

[0115] Comparisons from the model objectives / functions 650 may update parameters 660 of the patient model estimator 120. In some instances, updating may be accomplished using gradients of the objective functions and backpropagation to update the parameters of the patient model estimator 120.

[0116] Similarly, action controller 520 may be trained to learn a control dictionary. The training data may be stored in the training database 124 of network / cloud computer 110. The training data for the action controller may include imaging data (e.g., X-ray images showingdevice movement, collimation settings, X-ray framerate, etc.), robot data (e.g., speed at which devices were maneuvered, etc.), and information used from any other devices. During training, action controller 520 may receive the data for a large number of patients and extract relevant information (e.g., by parsing relevant information like X-ray framerate, collimation settings, robot velocity, etc., and / or using traditional image processing methods or machine / deep learning methods to segment devices from images, etc. to infer device type or device speed over X-ray frames, and so on). And then aggregate extracted information (using any method including simple average or more complex modelling) to build a dictionary (e.g., similar to Fig. 10) of system control decisions or actions to suggest or trigger at each checkpoint for a given procedure.

[0117] Figure 7 is a second diagrammatic schematic view of a training system 600 for an untrained patient model estimator 730, according to aspects of the present disclosure. After training, as described in Fig. 6, is complete an untrained patient model estimator with parameters A 730 is transformed into a trained patient model estimator with parameters B 740. Parameters A and B differ between the trained and untrained estimators because over the course of training, parameters in the patient model estimator are updated (e.g., 660 in Fig. 6) based on comparisons between ground truth labels and predictions.

[0118] Figure 11 depicts operation of the action controller during an endovascular procedure, according to aspects of the present disclosure. A time sequence 1100 is an example time sequence with three example times 1102, 1104, 1106 selected and associated during different stages of an endovascular procedure. At each time slice, an image is generated by the supporting imaging system (e.g., medical imaging console 170 and device 180), which may be an X-ray system.

[0119] At the first time 1102, interoperative image 1110, e.g., an X-ray image, depicts an endovascular device 1112 (e.g., 150) traversing the vasculature starting at an access site. Interoperative image 1110 is received by the action controller 520 along with the associated 3D patient model and checkpoints 1114. In some aspects, the action controller 520 may need to co-register the interoperative image 1110 and 3D patient model 1114. As described herein, the action controller generates a notification / alert and a recommended action 1116. At the first time 1102, the endovascular device 1112 is approaching the iliac checkpoint, reflected in the alert from the action controller 520: “Alert: within 4cm of iliac checkpoint.” The action controller 520 may also generate the recommended action: “Recommended: reduce X-ray dosage.”

[0120] At the second time 1104, interoperative image 1120, e.g., an X-ray image, depicts an endovascular device 1112 (e.g., 150) traversing the vasculature starting at an access site. As depicted in image 1120, the procedure has progressed to a new stage with the device 1112 having travelled further through the vasculature compared to the first time 1102. Interoperative image 1120 is received by the action controller 520 along with the associated 3D patient model and checkpoints 1124. The 3D patient model and checkpoints 1124 may differ from an earlier model and checkpoints, e.g., 1114, because of an updated model produced by a patient model estimator 120 from new measurements, as described herein. In some aspects, the action controller 520 may only need to receive the 3D patient model and checkpoints 1124 once. In some aspects, the action controller 520 may need to co-register the interoperative image 1120 and 3D patient model 1124. As described herein, the action controller generates a notification / alert and a recommended action 1126. At the second time 1104, the endovascular device 1112 is approaching the renal checkpoint, reflected in the alert from the action controller 520: “Alert: within 2cm of renal checkpoint.” The action controller 520 may also generate the recommended action: “Recommended: maintain navigation speed.”

[0121] At the third time 1106, interoperative image 1130, e.g., an X-ray image, depicts an endovascular device 1112 (e.g., 150) traversing the vasculature starting at an access site. As depicted in image 1130, the procedure has progressed to a new stage with the device 1112 having travelled further through the vasculature compared to the first time 1102 and second time 1104. Interoperative image 1130 is received by the action controller 520 along with the associated 3D patient model and checkpoints 1134. The 3D patient model and checkpoints 1134 may differ from an earlier model and checkpoints, e.g., 1114, 1124, because of an updated model produced by a patient model estimator 120 from new measurements, as described herein. In some aspects, the action controller 520 may only need to receive the 3D patient model and checkpoints 1134 once. In some aspects, the action controller 520 may need to co-register the interoperative image 1130 and 3D patient model 1134. As described herein, the action controller generates a notification / alert and a recommended action 1136. At the third time 1106, the endovascular device 1112 is near the aortic arch checkpoint, reflected in the alert from the action controller 520: “Alert: at aortic arch checkpoint.” The action controller 520 may also generate the recommended action: “Recommended: reduce navigation speed.”

[0122] The alerts and recommended actions are merely illustrative, and many others are possible depending on the procedure, the medical devices, and stage of the procedure.

[0123] Figure 12 is a schematic flow diagram of a method 1200 for automatic checkpoint identification, according to aspects of the present disclosure. It is understood that the steps of method 1200 may be performed in a different order than shown in Figure 12, additional steps can be provided before, during, and after the steps, and / or some of the steps described can be replaced or eliminated in other aspects. One or more of steps of the method 1200 can be carried by one or more devices and / or systems described herein, such as components of the networked system 100 and / or processor circuit 250.

[0124] In step 1202, the method 1200 includes receiving, from a user input device, a user input including patient health information or pre-operative images. For example, a doctor may input patient health information 504 into a user computer 160 via an input device 164. In some aspects, may upload preoperative images of a patient saved locally on user computer or may select the relevant images stored elsewhere, such as on network / cloud computer 110.

[0125] In step, 1204, the method 1200 includes generating, as an output of a patient model estimator, a 3D patient model with one or more anatomical checkpoints from the user input. For example, patient model estimator 120 generates a patient model and anatomical checkpoints 512 from patient measurements 502. In some aspects, patient model estimator 120 may generate a patient model and checkpoint from interoperative images and / or measurements 522.

[0126] In step 1206, the method 1200 includes receiving, from an external imaging system, one or more imaging frames tracking a medical procedure. For example, medical imaging console 170 and medical imaging device 180 may be an external X-ray imaging system with a C-arm whose movements may be tracked and recorded. The external X-ray imaging system may generate X-ray frames and send them to other devices in the networked system 100.

[0127] In step 1208, the method 1200 includes detecting, by a controller, the progress of the medical procedure based on a proximity of an intravascular device to one of the one or more anatomical checkpoints. For example, action controller 520 may receive the patient model and checkpoints 512 and interoperative images from, for example, an external X-ray imaging system 170, 180 and compare the progress of an endovascular device, e.g., 1112, through a patient’s vasculature, as described in Fig. 11.

[0128] In step 1210, the method 1200 includes generating, by a controller, an action recommendation based on the proximity. For example, action controller’s 520 action dictionary, e.g., 1000, may be searched to determine if a condition 1005 has been met, and if so, selecting / outputting the associated action 1010 to the met condition 1005.

[0129] In step 1212, the method 1200 includes displaying a screen display comprising the one or more imaging frames, a notification corresponding to the action recommendation, and an anatomical checkpoint. For example, display 1300 depicts imaging frame 1310, an alert 1340, an action recommendation 1345, and an anatomical checkpoint 1315.

[0130] Figure 13 is an example display 1300 for displaying the output of an action controller 520, according to aspects of the present disclosure. Example display 1300 may be a user interface provided through a display at the medical device console 140, user computer 160, or medical imaging console 170 to medical personnel. The display 1300 may include a, possibly live, interoperative imaging frame 1310, such as an X-ray, an anatomical checkpoint 1315, settings 1332-1334-1336 associated with the medical imaging console 170 and device 180 (e.g., X-ray), a notification / alert 1340, and a recommended action 1345. Interoperative imaging frame 1310 may depict the endovascular device 1112 (e.g., similar to 150 in networked system 100) approaching the iliac checkpoint 1315. The iliac checkpoint 1315 is marked with a flag on the imaging frame 1310. Settings A, B, and C 1332, 1334, 1336 may be various imaging settings of the external imaging system, e.g., X-ray, and / or the endovascular device. For example, setting A 1332 may be the X-ray dosage, setting B 1334 may be collimation settings, and setting C 1336 may be the robot speed of the endovascular device, where the robot speed controls how quickly the endovascular device navigates the vasculature. In some aspects, settings may be changed by a user using an input device on the relevant device in the networked system 100.

[0131] In some aspects, an action controller may directly control the medical imaging device and / or medical device 150 (e.g., the X-ray system and / or endovascular device) based on the recommended action. In such instances, the action taken by the controller is provided as a notification on the display 1300 and not as a recommended action for a user to implement.

[0132] Furthermore, the technology disclosed herein is also applicable to other medical imaging modalities obtained from a medical imaging device where 3D data is available, such as other 3D ultrasound applications, camera-based videos with post-processed 3D reconstructions, X-ray images from multiple views, fiber optic shape sensing, and 3D volume images, such as computer aided tomography (CT) scans, magnetic resonance imaging (MRI) scans. The technology described herein can be used in a variety of settings including emergency department, intensive care, inpatient, and out-of-hospital settings.

[0133] Accordingly, the logical operations making up the aspects of the technology described herein are referred to variously as operations, steps, objects, layers, elements, components, algorithms, or modules. Furthermore, it should be understood that these may occur or beperformed or arranged in any order, unless explicitly claimed otherwise or a specific order is inherently necessitated by the claim language.

[0134] All directional references e.g., upper, lower, inner, outer, upward, downward, left, right, lateral, front, back, top, bottom, above, below, vertical, horizontal, clockwise, counterclockwise, proximal, and distal are only used for identification purposes to aid the reader’s understanding of the claimed subject matter, and do not create limitations. Connection references, e.g., attached, coupled, connected, joined, or “in communication with” are to be construed broadly and may include intermediate members between a collection of elements and relative movement between elements unless otherwise indicated. As such, connection references do not necessarily imply that two elements are directly connected and in fixed relation to each other. The term “or” shall be interpreted to mean “and / or” rather than “exclusive or.” The word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. Unless otherwise noted in the claims, stated values shall be interpreted as illustrative only and shall not be taken to be limiting.

[0135] The above specification, examples and data provide a complete description of the structure and use of exemplary aspects of the automatic checkpoint identification system as described herein. Although various aspects of the claimed subject matter have been described above with a certain degree of particularity, or with reference to one or more individual aspects, those skilled in the art could make numerous alterations to the disclosed aspects without departing from the spirit or scope of the claimed subject matter.

[0136] Still other aspects are contemplated. It is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative only of particular aspects and not limiting. Changes in detail or structure may be made without departing from the basic elements of the subject matter as defined in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method, comprising: receiving, from a user input device, a user input including patient health information or pre-operative images; generating, as an output of a patient model estimator, a 3D patient model with one or more anatomical checkpoints from the user input; receiving, from an external imaging system, one or more imaging frames tracking a medical procedure; detecting, by a controller, progress of the medical procedure based on a proximity of an intravascular device to one of the one or more anatomical checkpoints; and generating, by a controller, an action recommendation based on the proximity.

2. The method of claim 1, further comprising: registering the one or more imaging frames with the 3D patient model; and outputting, to a display, a screen display comprising the one or more imaging frames, a notification corresponding to the action recommendation, and an anatomical checkpoint.

3. The method of claim 2, wherein the imaging frames comprises X-ray frames, and wherein the action recommendation comprises changing at least one of a navigation speed, an x-ray dosage, or X-ray collimation setting.

4. The method of claim 1, further comprising: generating, by the patient model estimator, an updated 3D patient model from the user input and at least one of the one or more imaging frames.

5. The method of claim 1, further comprising: automatically controlling, by the controller, the external imaging system or the intravascular device based on the action recommendation.

6. The method of claim 1, wherein the generating the 3D patient model further comprises: generating, via the patient model estimator, the 3D patient model from one or more shape features, wherein the shape features include a mean patient shape and one or more shape variation modes; and mapping, via the patient model estimator, one or more reference checkpoints on the mean patient shape to the one or more anatomical checkpoints on the 3D patient model.

7. The method of claim 1, wherein the patient model estimator comprises a neural network-based variational autoencoder and wherein the generating the 3D patient model with the one or more anatomical checkpoints further comprises: generating, via the patient model estimator, a sample from a latent shape distribution parametrized by one or more shape features, wherein the shape features include a mean and a variance; and generating, as an output of a decoder in the neural network-based variational autoencoder, the 3D patient model from the sample and the user input; performing registration of the 3D patient model and a mean patient shape including one or more reference checkpoints; and mapping, via the patient model estimator, the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model and the mean patient shape.

8. The method of claim 1, wherein the patient model estimator comprises a neural network-based diffusion model and wherein the generating the 3D patient model with the one or more anatomical checkpoints further comprises: generating, via the neural network-based diffusion model, a 3D patient model from a noisy mean patient shape and the user input; performing registration of the 3D patient model and a mean patient shape including one or more reference checkpoints; and mapping, via the patient model estimator, the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model and the mean patient shape.

9. A system, comprising: a processor circuit configured to: receive, from a user input device, a user input including patient health information or pre-operative images; generate, as an output of a patient model estimator, a 3D patient model with one or more anatomical checkpoints from the user input; receive, from an external imaging system, one or more imaging frames tracking a medical procedure; detect, by a controller, progress of the medical procedure based on a proximity of an intravascular device to one of the one or more anatomical checkpoints; and generate, by a controller, an action recommendation based on the proximity.

10. The system of claim 9, wherein the processor circuit is further configured to: register the one or more imaging frames with the 3D patient model; and output, to a display, a screen display comprising the one or more imaging frames, a notification corresponding to the action recommendation, and an anatomical checkpoint.

11. The system of claim 10, wherein the imaging frames comprises X-ray frames, and wherein the action recommendation comprises changing at least one of a navigation speed, an x-ray dosage, or X-ray collimation setting.

12. The system of claim 9, wherein the processor circuit is further configured to: generate, by the patient model estimator, an updated 3D patient model from the user input and at least one of the one or more imaging frames.

13. The system of claim 9, wherein the processor circuit is further configured to: automatically control, by the controller, the external imaging system or the intravascular device based on the action recommendation.

14. The system of claim 9, wherein the generating the 3D patient model further comprises:generating, via the patient model estimator, the 3D patient model from one or more shape features, wherein the shape features include a mean patient shape and one or more shape variation modes; and mapping, via the patient model estimator, one or more reference checkpoints on the mean patient shape to the one or more anatomical checkpoints on the 3D patient model.

15. The system of claim 9, wherein the patient model estimator comprises a neural network-based variational autoencoder and wherein the generating the 3D patient model with the one or more anatomical checkpoints further comprises: generating, via the patient model estimator, a sample from a latent shape distribution parametrized by one or more shape features, wherein the shape features include a mean and a variance; and generating, as an output of a decoder in the neural network-based variational autoencoder, the 3D patient model from the sample and the user input; performing registration of the 3D patient model and a mean patient shape including one or more reference checkpoints; and mapping, via the patient model estimator, the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model and the mean patient shape.

16. The system of claim 9, wherein the patient model estimator comprises a neural network-based diffusion model and wherein the generating the 3D patient model with the one or more anatomical checkpoints further comprises: generating, via the neural network-based diffusion model, a 3D patient model from a noisy mean patient shape and the user input; performing registration of the 3D patient model and a mean patient shape including one or more reference checkpoints; and mapping, via the patient model estimator, the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model and the mean patient shape.

17. A non-transitory machine-readable medium comprising a plurality of machineexecutable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising: receiving, from a user input device, a user input including patient health information or pre-operative images; generating, as an output of a patient model estimator, a 3D patient model with one or more anatomical checkpoints from the user input; receiving, from an external imaging system, one or more imaging frames tracking a medical procedure; detecting, by a controller, progress of the medical procedure based on a proximity of an intravascular device to one of the one or more anatomical checkpoints; and generating, by a controller, an action recommendation based on the proximity.

18. The non-transitory machine-readable medium of claim 17, wherein the one or more processors are further caused to perform operations comprising: registering the one or more imaging frames with the 3D patient model; and outputting, to a display, a screen display comprising the one or more imaging frames, a notification corresponding to the action recommendation, and an anatomical checkpoint.

19. The non-transitory machine-readable medium of claim 18, wherein the imaging frames comprises X-ray frames, and wherein the action recommendation comprises changing at least one of a navigation speed, an x-ray dosage, or X-ray collimation setting.

20. The non-transitory machine-readable medium of claim 17, wherein the one or more processors are further caused to perform operations comprising: automatically controlling, by the controller, the external imaging system or the intravascular device based on the action recommendation.

Images