Automatic identification of anatomical examination points for system control, and related equipment, systems, and methods.

By generating patient models through an automated checkpoint identification system and registering them with the process support system, the challenges of adjusting imaging system parameters in intravascular procedures are addressed, improving the safety and effectiveness of the process and reducing radiation exposure and the cognitive burden on medical staff.

CN123124944APending Publication Date: 2026-11-17KONINKLIJKE PHILIPS NV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202580027161.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-04-10
Filing Date
2025-04-07
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In intravascular procedures, clinicians need to frequently adjust imaging system and robot manipulation parameters, which can lead to distraction and suboptimal system parameters that may make it difficult to complete the procedure. In particular, incorrect imaging protocols may generate poor-quality images, making navigation or treatment delivery difficult.

Method used

An automatic checkpoint identification system is employed, which generates a patient model with anatomical checkpoints through a patient model estimator and registers it with the images of the workflow support system. When the motion controller detects approach to the checkpoint, it automatically adjusts the imaging support system and intravascular device settings, providing notifications and suggested actions.

Benefits of technology

It reduces patients' exposure to radiation, reduces the need for medical staff to understand system settings, improves the effectiveness and safety of the process, and ensures that intravascular devices navigate along the planned path.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123124944A_ABST
    Figure CN123124944A_ABST
Patent Text Reader

Abstract

An automated checkpoint identification system receives user input including patient health information or preoperative images. The system generates a 3D patient model as the output of a patient model estimator, the 3D patient model having one or more anatomical checkpoints from the user input. The system receives one or more imaging frames from an external imaging system to track a medical procedure. The system detects the progress of the medical procedure via a controller based on the proximity of an intravascular device to one of the one or more anatomical checkpoints. The system generates action suggestions based on the proximity via the controller.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure generally relates to intravascular procedures (e.g., balloon angioplasty, stent placement, etc.). Specifically, it relates to the automatic identification of anatomical checkpoints used for system control during intravascular procedures (e.g., to provide user guidance during intravascular procedures). Background Technology

[0002] Endovascular procedures are typically performed with the support of several systems designed to provide different utilities at different stages of the procedure. The primary system used to guide these procedures is usually an imaging system (e.g., an X-ray imaging system). Different stages of the procedure, encountered at different anatomical locations, often require different X-ray imaging settings (including different X-ray doses, different collimation doses, etc.). Similarly, in robot-assisted procedures, different stages require different robot manipulations, movement at different speeds, etc. Various clinicians must constantly adjust these system parameters to ensure optimal and safe use of the system throughout the procedure. Adjusting system parameters at various stages of the procedure can distract clinicians from the patient, and suboptimal system parameters can make procedural steps difficult to complete (e.g., incorrect imaging protocols can produce poor-quality images, making navigation or treatment delivery difficult). Therefore, configuring and reconfiguring various support systems can be challenging.

[0003] Information included in the background section of this specification, including any references cited herein and any descriptions or discussions thereof, is included for technical reference purposes only and should not be considered as subject matter limiting the scope of this disclosure. Summary of the Invention

[0004] An automated checkpoint identification system is disclosed. Patient health information and / or pre-procedure images are received from a user (i.e., a medical professional or interventional physician). A patient model with identified anatomical checkpoints is generated by a patient model estimator. The patient model with anatomical checkpoints can be registered with images (e.g., X-ray imaging frames) from a procedure support system. A motion controller can detect / determine when an intravascular procedure approaches an anatomical checkpoint and generate a notification for the user. The notification may include suggested actions (e.g., reducing X-ray dose, changing collimation settings, reducing the navigation speed of the intravascular device, etc.). In some cases, upon detecting the approach of an intravascular device to an anatomical checkpoint, the motion controller can automatically adjust the settings of the imaging support system and / or the intravascular medical device system, while also notifying the user of these changes.

[0005] The automated checkpoint marking system disclosed herein has specific, but non-exclusive, utility for supporting endovascular procedures. The system receives patient health information and automatically generates a patient model with relevant anatomical checkpoints, which can be used to monitor the progress of the endovascular procedure. The system advantageously updates system settings, including, for example, support systems for X-ray imaging, thereby reducing patient radiation exposure. Furthermore, the system advantageously warns the user of approaching an anatomical checkpoint where the likelihood of the endovascular device deviating from its planned path through the vascular system increases, allowing the user to take action to prevent deviation (e.g., updating the X-ray view). Additionally, the system advantageously warns the user of anatomical checkpoints where endovascular equipment must be replaced, allowing staff time to prepare new equipment.

[0006] A system of one or more computers can be configured to perform a specific operation or action by installing software, firmware, hardware, or a combination thereof on the system, which in operation causes or causes the system to perform the action. One or more computer programs can be configured to perform a specific operation or action by including instructions that, when executed by a data processing device, cause the device to perform the action.

[0007] In one general aspect, this disclosure relates to a computer-implemented method. The computer-implemented method further includes: receiving user input from a user input device, the user input including patient health information or preoperative images; generating a 3D patient model as the output of a patient model estimator, the 3D patient model having one or more anatomical checkpoints from the user input. The computer-implemented method further includes: receiving one or more imaging frames tracking a medical procedure from an external imaging system. The computer-implemented method further includes: detecting the progress of the medical procedure by a controller based on the proximity of an intravascular device to one of the one or more anatomical checkpoints. The computer-implemented method further includes: generating action suggestions by the controller based on the proximity.

[0008] In some aspects, implementations may include one or more of the following features. The method may include: registering the one or more imaging frames with the 3D patient model; and outputting a screen display to a display, the screen display including the one or more imaging frames, notifications corresponding to the action suggestions, and anatomical checkpoints. The imaging frames may include X-ray frames, and the action suggestions may include at least one of: changing navigation speed, X-ray dose, or X-ray collimation settings. The method may include: generating an updated 3D patient model by the patient model estimator based on the user input and at least one of the one or more imaging frames. The method may include: automatically controlling the external imaging system or the intravascular device by the controller based on the action suggestions.

[0009] In some aspects, implementations may include one or more of the following features. Generating the 3D patient model may further include: generating the 3D patient model via the patient model estimator based on one or more shape features, wherein the shape features include an average patient shape and one or more shape variation patterns; and mapping one or more reference checkpoints on the average patient shape to the one or more anatomical checkpoints on the 3D patient model via the patient model estimator. The patient model estimator may include a neural network-based variational autoencoder, and generating the 3D patient model having the one or more anatomical checkpoints may further include: generating samples via the patient model estimator based on a latent shape distribution parameterized by one or more shape features, wherein the shape features include a mean and a variance; and generating the 3D patient model as the output of a decoder in the neural network-based variational autoencoder based on the samples and the user input; performing registration of the 3D patient model with an average patient shape including one or more reference checkpoints; and mapping the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model via the patient model estimator based on the registration of the 3D patient model with the average patient shape. The patient model estimator may include a neural network-based diffusion model, and wherein generating the 3D patient model having the one or more anatomical checkpoints may further include generating the 3D patient model via the neural network-based diffusion model based on a noisy average patient shape and the user input; performing registration of the 3D patient model with an average patient shape including one or more reference checkpoints; and mapping the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model via the patient model estimator based on the registration of the 3D patient model with the average patient shape.

[0010] In one general aspect, this disclosure relates to a system including processor circuitry configured to: receive user input from a user input device, the user input including patient health information or preoperative images; generate a 3D patient model as the output of a patient model estimator, the 3D patient model having one or more anatomical checkpoints from the user input; receive one or more imaging frames from an external imaging system tracking a medical procedure; detect the progress of the medical procedure by a controller based on the proximity of an intravascular device to one of the one or more anatomical checkpoints; and generate action suggestions by the controller based on the proximity.

[0011] In some aspects, the implementation may include one or more of the following features. In this system, the processor circuitry is further configured to: register the one or more imaging frames with the 3D patient model; and output a screen display to a display, the screen display including the one or more imaging frames, a notification corresponding to the action suggestion, and anatomical checkpoints. The imaging frames may include X-ray frames, and the action suggestion may include at least one of the following: changing navigation speed, X-ray dose, or X-ray collimation setting. The processor circuitry is further configured to: generate an updated 3D patient model by the patient model estimator based on the user input and at least one of the one or more imaging frames. The processor circuitry is further configured to: automatically control the external imaging system or the intravascular device by the controller based on the action suggestion.

[0012] In some aspects, implementations may include one or more of the following features. Generating the 3D patient model may further include: generating the 3D patient model via the patient model estimator based on one or more shape features, wherein the shape features include an average patient shape and one or more shape variation patterns; and mapping one or more reference checkpoints on the average patient shape to the one or more anatomical checkpoints on the 3D patient model via the patient model estimator. The patient model estimator may include a neural network-based variational autoencoder, and generating the 3D patient model having the one or more anatomical checkpoints may further include: generating samples via the patient model estimator based on a latent shape distribution parameterized by one or more shape features, wherein the shape features include a mean and a variance; and generating the 3D patient model as the output of a decoder in the neural network-based variational autoencoder based on the samples and the user input; performing registration of the 3D patient model with an average patient shape including one or more reference checkpoints; and mapping the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model via the patient model estimator based on the registration of the 3D patient model with the average patient shape. The patient model estimator may include a neural network-based diffusion model, and generating the 3D patient model having the one or more anatomical checkpoints may further include: generating the 3D patient model via the neural network-based diffusion model based on a noisy average patient shape and the user input; performing registration of the 3D patient model with an average patient shape including one or more reference checkpoints; and mapping the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model via the patient model estimator based on the registration of the 3D patient model with the average patient shape.

[0013] In one general aspect, this disclosure relates to a non-transient machine-readable medium that may include a plurality of machine-executable instructions, which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations including: receiving user input from a user input device, the user input including patient health information or preoperative images; generating a 3D patient model as the output of a patient model estimator, the 3D patient model having one or more anatomical checkpoints from the user input; receiving one or more imaging frames from an external imaging system to track a medical procedure; detecting the progress of the medical procedure by a controller based on the proximity of an intravascular device to one of the one or more anatomical checkpoints; and generating action suggestions by the controller based on the proximity.

[0014] In some aspects, the implementation may include one or more of the following features. In the non-transient machine-readable medium, the one or more processors are also caused to perform operations that may include: registering the one or more imaging frames with the 3D patient model; and outputting a screen display to a display, the screen display including: the one or more imaging frames, a notification corresponding to the action suggestion, and anatomical checkpoints. The imaging frames may include X-ray frames, and the action suggestion may include at least one of: changing navigation speed, X-ray dose, or X-ray collimation setting. The one or more processors are also caused to perform operations that may include: generating an updated 3D patient model by the patient model estimator based on the user input and at least one of the one or more imaging frames. The one or more processors are also caused to perform operations that may include: automatically controlling the external imaging system or the intravascular device by the controller based on the action suggestion.

[0015] In some aspects, implementations may include one or more of the following features. Generating the 3D patient model may further include: generating the 3D patient model via the patient model estimator based on one or more shape features, wherein the shape features include an average patient shape and one or more shape variation patterns; and mapping one or more reference checkpoints on the average patient shape to the one or more anatomical checkpoints on the 3D patient model via the patient model estimator. The patient model estimator may include a neural network-based variational autoencoder, and generating the 3D patient model having the one or more anatomical checkpoints may further include: generating samples via the patient model estimator based on a latent shape distribution parameterized by one or more shape features, wherein the shape features include a mean and a variance; and generating the 3D patient model as the output of a decoder in the neural network-based variational autoencoder based on the samples and the user input; performing registration of the 3D patient model with an average patient shape including one or more reference checkpoints; and mapping the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model via the patient model estimator based on the registration of the 3D patient model with the average patient shape. The patient model estimator may include a neural network-based diffusion model, and generating the 3D patient model having the one or more anatomical checkpoints may further include: generating the 3D patient model via the neural network-based diffusion model based on a noisy average patient shape and the user input; performing registration of the 3D patient model with an average patient shape including one or more reference checkpoints; and mapping the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model via the patient model estimator based on the registration of the 3D patient model with the average patient shape.

[0016] This synopsis is provided to present, in a simplified form, the selection of concepts further described below in the detailed description. This synopsis is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. A broader presentation of the features, details, utility, and advantages of the automated measurement point detection system as defined in claims is provided in the following written description of various aspects of this disclosure and illustrated in the accompanying drawings. Attached Figure Description

[0017] The illustrated aspects of this disclosure will be described with reference to the accompanying drawings, in which: Figure 1 This is a schematic diagram of a networked system for automatic checkpoint identification based on various aspects of this disclosure.

[0018] Figure 2 This is a schematic diagram of the processor circuitry according to various aspects of this disclosure.

[0019] Figure 3 This is a schematic diagram of the configuration of deep learning networks based on various aspects of this disclosure.

[0020] Figure 4 The diagram illustrates a portion of the human vascular system.

[0021] Figure 5 This is a schematic diagram of at least a portion of an automated checkpoint marking system based on various aspects of this disclosure.

[0022] Figure 6 This is a schematic diagram of at least a portion of a training system for a patient model estimator based on various aspects of this disclosure.

[0023] Figure 7 This is a second schematic diagram of a training system for an untrained patient model estimator based on various aspects of this disclosure.

[0024] Figure 8 This is a graph of the patient model estimator based on various aspects of this disclosure.

[0025] Figure 9 This is a schematic diagram of a variational autoencoder based on various aspects of this disclosure.

[0026] Figure 10 A simplified example control dictionary is described based on various aspects of this disclosure.

[0027] Figure 11 The operation of the motion controller during intravascular procedures according to various aspects of this disclosure is described.

[0028] Figure 12 This is a schematic flowchart of a method for automatic checkpoint identification based on various aspects of this disclosure.

[0029] Figure 13 This is an example display for displaying the output of a motion controller, based on various aspects of this disclosure. Detailed Implementation

[0030] According to at least one aspect of this disclosure, an automated checkpoint identification system is provided that automatically generates a patient body model with identified anatomical checkpoints. This can allow for notification to the user (including medical professionals and other medical personnel) when a checkpoint is reached, for example, during an intravascular procedure or other procedure. Furthermore, the system provides an automated manner to notify the user of approaching a checkpoint and suggests adjustments to the settings of various systems utilized during the procedure, such as intravascular devices and X-ray imaging support systems. In some cases, the controller directly controls system settings rather than providing suggestions for user action.

[0031] These descriptions are provided for illustrative purposes only and should not be construed as limiting the scope of the automated checkpoint identification system. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.

[0032] For the purpose of facilitating an understanding of the principles of this disclosure, reference will now be made to aspects illustrated in the accompanying drawings, and these aspects will be described using specific language. However, it should be understood that this is not intended to limit the scope of this disclosure. Any changes and additional modifications to the devices, systems, and methods described in this disclosure, as well as any additional applications of the principles of this disclosure, are fully contemplated and included in this disclosure, as would normally occur to those skilled in the art to which this disclosure pertains. In particular, it is fully contemplated that features, components, and / or steps described with respect to one aspect may be combined with features, components, and / or steps described with respect to other aspects of this disclosure. However, for the sake of brevity, multiple iterations of these combinations will not be described separately. As used herein, access may include querying, retrieving, sorting, etc.

[0033] The systems and methods disclosed herein offer numerous benefits. During intravascular procedures, patients may be exposed to a considerable dose of radiation (e.g., X-rays) from the imaging support system. Therefore, the systems and methods disclosed herein improve control over system setup that allows for efficient procedures while also reducing radiation exposure. Furthermore, they reduce the cognitive requirements of medical device-supported imaging systems, thereby allowing for improved attention to other aspects of the patient and the procedure.

[0034] As used herein, registration may include establishing a spatial relationship / correspondence between the generated patient model and the mean patient model such that the position in the generated patient model matches the corresponding position in the mean patient model (and vice versa), or establishing a spatial relationship / correspondence between the patient model and (one or more) X-ray images such that the position in the patient model matches the corresponding position in (one or more) X-ray images (and vice versa).

[0035] Figure 1 This is a schematic diagram of a networked system 100 for automated checkpoint identification, according to various aspects of this disclosure. The networked system 100 can, for example, be used to receive patient health information from a user. The patient health information can be transmitted to different components of the networked system 100 to generate a patient model and register the patient model with images from a medical device console 140. The networked system 100 can provide notifications and / or suggested actions to the user based on real-time analysis of intravascular procedures and external imaging registered with the patient's body model.

[0036] Networked system 100 is used to generate notifications for users, suggest actions to users, or automatically control external imaging system settings and / or medical device settings during the process. Networked system 100 may include a network / cloud computer 110, a medical device console 140, a medical device 150, a user computer 160, a medical imaging console 170, and a medical imaging device 180. As depicted, each of the network / cloud computer 110, medical device console 140, user computer 160, and medical imaging console 170 can communicate with each other. As described herein, communication between different components can be achieved through any number of connections (e.g., wired and / or wireless).

[0037] The network / cloud computer 110 may include a processor 112, an input device 114, a display 116, a communication interface 117, and a memory 118.

[0038] Processor 112 may include a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), controller, field-programmable gate array (FPGA) device, another hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor 112 may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). Processor 112 is configured to process instructions stored in memory 118. Processor 112 is connected to communication interface 117.

[0039] Input device 114 allows users to make selections or provide instructions to network / cloud computer 110. Input device may be a mouse, touch screen, touchpad, etc.

[0040] Display 116 is coupled to processor 112. Display 116 may be a monitor or any suitable display. Display 116 is configured to display the output of patient model estimator 120, patient information 122, information in training database 124, preoperative images 126, network performance, or any other system diagnostic information.

[0041] Communication interface 117 is coupled to processor 112. Communication interface 117 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. Communication interface 117 may include hardware and / or software components implementing a specific communication protocol suitable for transmitting signals to other devices in networking system 100. Communication interface 117 may be referred to as a communication device or communication interface module.

[0042] Memory 118 is coupled to processor 112. Memory 118 can be any suitable storage device, such as cache memory (e.g., cache memory of processor 112), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory device, hard disk drive, solid-state drive, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0043] Memory 118 may be configured to store subject information, measurement results, data or subject-related documents, medical history, history of procedures performed, anatomical or biological characteristics, traits or medical conditions associated with the subject, computer-readable instructions (e.g., code, software or other applications), and any other suitable information or data. Subject information may include measurements, data, documents, other forms of medical history (e.g., but not limited to X-ray or ultrasound images), X-ray or ultrasound videos, and / or any imaging information related to the subject's anatomy. Subject information may include parameters related to the imaging process (e.g., anatomical field of view, device orientation, and / or the subject's position during the imaging process). Memory 118 may also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to the implementation of image recognition algorithms, image quantization algorithms, and / or process guidance algorithms (including those described herein) for detecting / segmenting anatomical structures.

[0044] In some cases, memory 118 includes data and stored instructions for the module, including patient model estimator 120. Patient model estimator 120 can generate a 3D model of the patient using patient health information and other patient-related images. For example, using the patient's age, height, and weight, patient model estimator 120 can generate a tessellation surface for using as a model of the patient's body shape.

[0045] The medical device console 140 may include a processor 142, an input device 144, a display 146, a communication interface 147, and a memory 148.

[0046] Processor 142 may include a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), controller, field-programmable gate array (FPGA) device, another hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor 142 may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). Processor 142 is configured to process instructions stored in memory 148. Processor 142 is connected to communication interface 147.

[0047] Input device 144 allows the user to make selections or provide instructions to the medical device control console 140. The input device may be a mouse, touchscreen, touchpad, etc. Using input device 144, the user can control device settings (e.g., robot speed) through the robot control of the medical device 150. For example, the robot control can control the navigation speed of a catheter through a vascular system.

[0048] Display 146 is coupled to processor 142. Display 146 may be a monitor or any suitable display. Display 146 is configured to display any output generated within network system 100, including patient models, intraoperative images with anatomical checkpoints, medical imaging device settings, or any other settings associated with the process.

[0049] Communication interface 147 is coupled to processor 142. Communication interface 147 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. Communication interface 147 may include hardware and / or software components implementing a specific communication protocol suitable for transmitting signals to other devices in networking system 100. Communication interface 147 may be referred to as a communication device or communication interface module.

[0050] Memory 148 is coupled to processor 142. Memory 148 can be any suitable storage device, such as cache memory (e.g., cache memory of processor 142), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory device, hard disk drive, solid-state drive, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0051] Memory 148 may be configured to store subject information, measurement results, data or subject-related documents, medical history, history of procedures performed, anatomical or biological characteristics, traits or medical conditions associated with the subject, computer-readable instructions (e.g., code, software or other applications), and any other suitable information or data. Subject information may include measurements, data, documents, other forms of medical history (e.g., but not limited to X-ray or ultrasound images), X-ray or ultrasound videos, and / or any imaging information related to the subject's anatomy. Subject information may include parameters related to the imaging process (e.g., anatomical field of view, device orientation, and / or the subject's position during the imaging process). Memory 148 may also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to the implementation of image recognition algorithms for detecting anatomical structures, segmenting anatomical structures (e.g., defining the outline of anatomical structures), image quantization algorithms, and / or process guidance algorithms (including those described herein).

[0052] In some cases, memory 148 includes data and instructions for storage in a module that includes a patient model estimator 120 and an action controller 520. For example... Figure 1 As depicted, the patient model estimator 120 resides on the network / cloud computer 110; however, it may also be stored on the medical device console 140, user computer 160, and medical imaging console 170. During automatic checkpoint identification, data and / or images can be sent from the medical imaging console 170 to the user computer 160, network / cloud computer 110, and / or medical device console 140. The patient model estimator 120 and motion controller 520 can generate patient models, checkpoints, and / or motion suggestions for the received data, which can then be displayed on a display device or sent back to one or more of the computers and devices in the networked system 100 (e.g., sent to the medical imaging console 170 to change the imaging dose or frame rate).

[0053] The patient model estimator 120 and action controller 520 can be any statistical model, machine learning model, neural network-based model, or rule-based programming model (e.g., as per [reference to...]). Figure 3 and Figure 9 (As described and depicted). For example, the patient model estimator 120 and the action controller 520 may implement a convolutional neural network (CNN), a variational autoencoder (VAE), a decision tree model, a support vector machine (SVM), a generative image-to-image transformer, a generative image-to-text transformer, or other machine learning models.

[0054] Medical device 150 can be used in procedures or imaging processes, including intravascular procedures (e.g., stent implantation). Medical device 150 can be an X-ray guided device (e.g., guidewires and catheters), a modified device (e.g., guidewires or catheters with embedded optical fibers in the case of shape sensing (e.g., Philips Fiber OpticRealShape or FORS)), etc. The medical device can communicate with medical device console 140 (e.g., via communication interface 147). When medical device 150 is an imaging device, medical images generated by medical device 150 can be displayed on display 146. Medical images can also be transmitted to any of computers, networks / clouds 110, 160, and 170.

[0055] User computer 160 may include processor 162, input device 164, display 166, communication interface 167, and memory 168.

[0056] Processor 162 may include a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), controller, field-programmable gate array (FPGA) device, another hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor 162 may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). Processor 162 is configured to process instructions stored in memory 168. Processor 162 is connected to communication interface 167.

[0057] Input device 164 allows the user to make selections or provide instructions to the user computer 160. Input devices may include mice, touchscreens, touchpads, etc.

[0058] Display 166 is coupled to processor 162. Display 166 may be a monitor or any suitable display. Display 166 is configured to display any output generated within network system 100, including patient models, intraoperative images with anatomical checkpoints, medical imaging device settings, or any other settings associated with the process.

[0059] Communication interface 167 is coupled to processor 162. Communication interface 167 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. Communication interface 167 may include hardware and / or software components implementing a specific communication protocol suitable for transmitting signals to other devices in networking system 100. Communication interface 167 may be referred to as a communication device or communication interface module.

[0060] Memory 168 is coupled to processor 162. Memory 168 can be any suitable storage device, such as cache memory (e.g., cache memory of processor 162), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory device, hard disk drive, solid-state drive, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0061] Memory 168 may be configured to store subject information, measurement results, data or subject-related documents, medical history, history of procedures performed, anatomical or biological characteristics, traits or medical conditions associated with the subject, computer-readable instructions (e.g., code, software or other applications), and any other suitable information or data. Subject information may include measurements, data, documents, other forms of medical history (e.g., but not limited to X-ray or ultrasound images), X-ray or ultrasound videos, X-ray data in the case of fiber optic or any number of guidewire / catheter-based procedures / techniques, and / or any imaging information related to the subject's anatomy. Subject information may include parameters related to the imaging procedure (e.g., anatomical field of view, device orientation, and / or subject position during the imaging procedure). Memory 168 may also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to the implementation of image recognition algorithms for detecting anatomical structures, segmenting anatomical structures, image quantization algorithms, and / or procedure guidance algorithms (including those described herein).

[0062] The medical imaging console 170 may include a processor 172, an input device 174, a display 176, a communication interface 177, and a memory 178.

[0063] Processor 172 may include a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), controller, field-programmable gate array (FPGA) device, another hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor 172 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Processor 172 is configured to process instructions stored in memory 178. Processor 172 is connected to communication interface 177.

[0064] Input device 174 allows the user to make selections or provide commands to medical imaging console 170. The input device can be a mouse, touchscreen, touchpad, etc. For example, the user can input different imaging settings to control X-ray dose, collimation settings, frame rate, etc.

[0065] Display 176 is coupled to processor 172. Display 176 may be a monitor or any suitable display. Display 176 is configured to display any output generated within network system 100, including patient models, intraoperative images with anatomical checkpoints, medical imaging device settings, or any other settings associated with the process.

[0066] Communication interface 177 is coupled to processor 172. Communication interface 177 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. Communication interface 177 may include hardware and / or software components implementing a specific communication protocol suitable for transmitting signals to other devices in networking system 100. Communication interface 177 may be referred to as a communication device or communication interface module.

[0067] Memory 178 is coupled to processor 172. Memory 178 can be any suitable storage device, such as cache memory (e.g., cache memory of processor 172), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory device, hard disk drive, solid-state drive, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0068] Memory 178 may be configured to store subject information, measurement results, data or subject-related documents, medical history, history of procedures performed, anatomical or biological characteristics, traits or medical conditions associated with the subject, computer-readable instructions (e.g., code, software or other applications), and any other suitable information or data. Subject information may include measurements, data, documents, other forms of medical history (e.g., but not limited to X-ray or ultrasound images), X-ray or ultrasound videos, and / or any imaging information related to the subject's anatomy. Subject information may include parameters related to the imaging process (e.g., anatomical field of view, device orientation, and / or the subject's position during the imaging process). Memory 178 may also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to the implementation of image recognition algorithms for detecting anatomical structures, segmenting anatomical structures, image quantization algorithms, and / or process guidance algorithms (including those described herein).

[0069] Medical imaging device 180 can generate medical images (e.g., X-ray images). The medical imaging device (e.g., via communication interface 177) communicates with medical imaging console 170. Medical images generated from data from medical imaging device 180 can be displayed on display 176. Medical images 185 can also be transmitted to a network / cloud, console, or any of computers 110, 140, or 160. In some aspects, the medical imaging device may include a fixed single-plane / dual-plane C-arm X-ray imaging system, a mobile C-arm X-ray imaging system, etc. In some cases, the C-arm can be tracked and used with intraoperative images to update checkpoints and patient models.

[0070] In some aspects, aspects of this disclosure can be implemented using medical images of a patient obtained using any suitable medical imaging device 180 and / or modality. Examples of medical images and medical imaging devices include X-ray images (angiographic images, fluoroscopic images, images with or without contrast) obtained by medical imaging devices such as X-ray imaging devices, computed tomography (CT) images obtained by CT imaging devices, positron emission tomography (PET-CT) images obtained by PET-CT imaging devices, magnetic resonance imaging (MRI) images obtained by MRI imaging devices, single-photon emission computed tomography (SPECT) images obtained by SPECT imaging devices, optical coherence tomography (OCT) images obtained by OCT imaging devices, and ultrasound images obtained by ultrasound imaging devices. The medical imaging device 180 can acquire medical images when positioned outside the subject's body, spaced apart from the subject's body, adjacent to the subject's body, in contact with the subject's body, and / or inside the subject's body.

[0071] The training database 124 can be stored on a network / cloud computer 110 or any other network or cloud computing environment. In some respects, the system described herein can be provided as a cloud service. The training dataset includes historical information, which may include preoperative images and / or measurements, intraoperative images and / or measurements, a baseline ground truth patient model with user-annotated checkpoints, patient health information (e.g., electronic medical record (EMR), age, sex, weight, etc.), interventional physician details, and tools, resources, and methods used for intervention.

[0072] It should be noted that the examples above are provided for illustrative purposes and are not intended to be limiting. Other devices and / or device configurations may be used to perform the operations described herein.

[0073] Figure 2This is a schematic diagram of processor circuitry 250 according to various aspects of this disclosure. Processor circuitry 250 may be implemented in a network / cloud computer 110, medical device console 140, user computer 160, medical imaging console 170, or other devices or workstations (e.g., third-party workstations, network routers, etc.), or on a cloud processor or other remote processing unit, to implement the method as needed. As shown, processor circuitry 250 may include processor 260, memory 264, and communication interface 268. These components may communicate directly or indirectly with each other, for example, via one or more buses.

[0074] Processor 260 may include any combination of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), ASIC, controller, or general-purpose computing device, reduced instruction set computing (RISC) device, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other related logic devices, including mechanical and quantum computers. Processor 260 may also include another hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor 260 may also be implemented as a combination of computing devices (e.g., a combination of DSP and microprocessors, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration).

[0075] Memory 264 may include cache memory (e.g., the cache memory of processor 260), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory devices, hard disk drives, other forms of volatile and non-volatile memory, or combinations of different types of memory. In one aspect, memory 264 includes a non-transient computer-readable medium. Memory 264 may store instructions 266. Instructions 266 may include instructions that, when executed by processor 260, cause processor 260 to perform the operations described herein. Instructions 266 may also be referred to as code. The terms “instruction” and “code” should be interpreted broadly to include any type of computer-readable statement(s). For example, the terms “instruction” and “code” may refer to one or more programs, routines, subroutines, functions, flows, etc., and “instruction” and “code” may include a single computer-readable statement or a number of computer-readable statements.

[0076] Communication interface 268 may include any electronic circuitry and / or logic circuitry to facilitate direct or indirect data communication between processor circuitry 250 and other processors or devices. In this regard, communication interface 268 may be an input / output (I / O) device. In some cases, communication interface 268 facilitates direct or indirect communication between various components of processor circuitry 250 and / or the computer of network system 100. Communication interface 268 can communicate within processor circuitry 250 via various methods or protocols. Serial communication protocols may include, but are not limited to, the US Serial Protocol Interface (US SPI), internal integrated circuits (I... 2 C) Recommended standards RS-232, RS-485, Controller Area Network (CAN), Ethernet, ARINC 429, MODBUS, MIL-STD-1553, or any other suitable method or protocol. Parallel protocols include, but are not limited to, Industry Standard Architecture (ISA), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI), IEEE 488, IEEE 1284, and other suitable protocols. Where appropriate, serial and parallel communication may be bridged by a Universal Asynchronous Receiver Transmitter (UART), a Universal Synchronous Receiver Transmitter (USART), or other suitable subsystems.

[0077] External communication (including, but not limited to, software updates, firmware updates, model sharing between the processor and the central server, or output generated by the model selection and prediction system described herein) can be achieved using any suitable wireless or wired communication technology (e.g., cable interfaces such as Universal Serial Bus (USB), Micro USB, Lightning or FireWire interfaces, Bluetooth, Wi-Fi, ZigBee, Li-Fi) or cellular data connections (e.g., 2G / GSM (Global System for Mobile Communications), 3G / UMTS (Universal Mobile Telecommunications System), 4G, LTE, WiMax, or 5G)). For example, Bluetooth Low Energy (BLE) radios can be used to establish connections to cloud services, for data transfer, and for receiving software patches. The controller can be configured to communicate with remote servers or local devices (e.g., laptops, tablets, or handheld devices), or may include a display capable of showing state variables and other information. Information can also be transmitted on physical media (e.g., USB flash drives or memory sticks).

[0078] Figure 3This is a schematic diagram of a deep learning network configuration according to various aspects of this disclosure. Configuration 300 may be implemented by a deep learning network. Configuration 300 includes a deep learning network 310, which may include one or more CNNs 312. CNN 312 is an example of a type of predictive model and / or machine learning model used in the patient model estimator 120. For the sake of simplicity in illustration and discussion, Figure 3 The diagram illustrates a CNN 312. However, any suitable number of CNN 312s may be included (e.g., approximately two, three, or more). Configuration 300 can be trained to identify various anatomical structures (organs, tissues, bones) and / or other features (natural and / or artificial) within a patient's anatomy. Configuration 300 can also be trained to segment human anatomy, diagnose medical conditions, or any number of other diagnostic or medical tasks.

[0079] CNN 312 may include a set 320 of N convolutional layers, followed by a set 330 of K fully connected layers, where N and K can be any positive integers. The convolutional layers 320 are shown as 320(1) to 320(N). The fully connected layers 330 are shown as 330(1) to 330(K). Each convolutional layer 320 may include a set 322 of filters configured to extract features (e.g., X-ray images or other data) from the input 302. The values ​​N and K, as well as the size of the filters 322, may vary depending on the purpose of the CNN. In some cases, the convolutional layers 320(1) to 320(N) and the fully connected layers 330(1) to 330(K-1) may be spread with modified nonlinearity (ReLU) or leaky ReLU or other activation functions and / or batch normalization layers. The fully connected layer 330 can gradually reduce the high-dimensional output to a lower dimension or the dimension of the predicted result 340 (e.g., the position of object detection landmarks, or the position and dimension of object detection boxes, or the number of categories in the classification output). The fully connected layer 330 can also be referred to as a classifier. In some respects, the fully convolutional layer 320 can also be referred to as a representation or encoding or feature.

[0080] When the prediction result 340 takes the form of a classification output, it can indicate a confidence score (e.g., probability) for each category 342 based on the input image 302. Categories 342 are designated as 342a, 342b, ..., 342c. For example, when the CNN 312 is trained for the classification of current anatomical features, category 342 can indicate a first anatomical feature category 342a, a second anatomical feature category 342b, a third anatomical feature category 342c, a fourth anatomical feature category 342d, or any other suitable category. A category 342 indicating a high confidence score indicates that the input image 302 or a segment or pixel of image 302 is likely to include an anatomical object / feature of category 342. Conversely, a category 342 indicating a low confidence score indicates that the input image 302 or a segment or pixel of image 302 is unlikely to include an anatomical object / feature of category 342.

[0081] CNN 312 can also output a feature vector 350 at the output of the last convolutional layer 320(N), although any layer in a CNN is a feature vector. The feature vector 350 can encode or represent some representation of an object detected from the input medical image 302 or other data. For example, the feature vector 350 can encode a representation of a patient model or a region associated with an anatomical examination point identified in image 302, which could be a patient medical image (e.g., an X-ray frame). The feature vector 350 can also encode some representation of pixels in a medical image associated with the location of anatomical structures (e.g., a relevant subset of the vascular system as described herein) associated with the anatomical examination point. These representations can be decoded using a reverse CNN, where fully connected layers extend low-dimensional representations to higher dimensions, and transposed convolutional layers extend the feature layer to the size of the original input, where pixel-wise outputs (e.g., binary segmentation maps, multi-class segmentation maps) can be generated.

[0082] Deep learning networks 310 can be implemented or include any suitable type of learning network. For example, in some aspects, and as regarding Figure 3 As described, the deep learning network 310 may include a convolutional neural network 312. Additionally, the deep learning network 310 may additionally or alternatively be or include any suitable network or unit that is a multi-class classification network, an encoder-decoder type network, a fully connected deep learning network, or that recognizes features within an image.

[0083] In some respects, when the deep learning network 310 includes a fully connected neural network, the fully connected neural network can transform data unrelated to medical images, or it can transform data derived from images (e.g., detected objects) generated by another network, program, or human annotator. Data from detected objects can be cascaded into layers (e.g., as additional channels). For simple numerical features such as age, weight, and other data, these can be cascaded into fully connected layers. For example, a fully connected neural network can transform information about a patient (e.g., age, weight, or other low-dimensional data).

[0084] In some aspects, when the deep learning network 310 includes an encoder-decoder network, the network may include two components. One component may be a constraint component or an encoder, where a large image (e.g., image 302) may be convolved by several convolutional layers 320 such that the size of image 302 changes relative to the depth of the network layers. For example, CNN 312 may be an encoder. Image 302 may then be represented in a low-dimensional space or a flattened space. From this flattened space, an additional component or decoder may extend the flattened space to the original size of image 302. For example, the inverse of CNN 312 may be a decoder. In some aspects, the encoder-decoder network may reconstruct the input image 302. In some aspects, the encoder-decoder network may segment image 302 into patches. In some aspects of this disclosure, the deep learning network 310 may include a multi-class classification network. In this aspect, the multi-class classification network may include an encoder path. For example, image 302 may be a high-dimensional image. Image 302 may then be processed with convolutional layers 320 such that its size is reduced. The resulting low-dimensional representation of image 302 may be used to generate Figure 3 The feature vector 350 is shown in the figure. The low-dimensional representation of image 302 can also be used by fully connected layer 330 to regress and output one or more categories 342. In some respects, fully connected layer 330 can process the output of convolutional layer 320. Fully connected layer 330 can also be referred to as task layer or regression layer, etc.

[0085] Any suitable combination or variation of the described deep learning network 310 is fully conceived. For example, the deep learning network may include fully convolutional networks or layers, or fully connected networks or layers, or a combination of both. Furthermore, the deep learning network may include multi-class classification networks, encoder-decoder networks, or any combination of networks. The process of training the deep learning network 310 involves tuning its parameters, or more specifically, adjusting the weights and biases that control the operations of the activation functions in the neural network. In supervised learning, the training process automatically adjusts the weights and biases so that, when presented with the input data, the neural network accurately provides the corresponding expected output data. To this end, the value of the loss function, or error, is calculated based on the difference between the predicted output data and the expected output data. The value of the loss function can be calculated using functions such as negative log-likelihood loss, mean squared error, Huber loss, or cross-entropy loss. During training, the value of the loss function is typically minimized. Various methods are known for solving the loss minimization problem (e.g., gradient descent, quasi-Newton methods, etc.). Various algorithms have been developed to implement these methods and their variations, including but not limited to stochastic gradient descent (SGD), batch gradient descent, mini-batch gradient descent, Gauss-Newton, Levenberg-Marquardt, Momentum, Adam, Nadam, Adagrad, Adadelta, RMSProp, and the Adamex "optimizer." These algorithms compute the derivative of the loss function with respect to the model parameters using a chain rule. This process is called backpropagation because the derivative is computed starting from the last layer or output layer and moving towards the first layer or input layer. These derivatives inform the algorithm how the model parameters must be adjusted to minimize the error function. The training process is performed iteratively by adjusting the weights and biases in each iteration. Training terminates when the error or difference between the predicted output data and the expected output data is within acceptable limits on the training data or some validation data. Subsequently, a neural network can be deployed, and the trained neural network uses the trained values ​​of its parameters to predict new input data. If the training process is successful, the trained neural network accurately predicts the expected output data based on the new input data.

[0086] Figure 4The diagram illustrates a portion of the human vascular system 400. This portion of the vascular system 400 may be diseased, requiring any number of different medical interventions. In some cases, plaque buildup may restrict flow within this portion of the vascular system 400. In such cases, an endovascular procedure may be necessary to restore flow. An endovascular procedure may begin by inserting a catheter into the vascular system at an entry site 402 near the groin. The catheter travels from the entry site 402 through the vascular system until it reaches the treatment area (i.e., the diseased tissue). Various anatomical checkpoints may be traversed along the path to the treatment site. For example, traveling from the entry site 402, the catheter may pass through the iliac checkpoint 404, the renal checkpoint 406, the aortic arch checkpoint 405, the neck checkpoint 410, and the head checkpoint 412. These checkpoints are merely exemplary; depending on the procedure and the location of the diseased tissue, fewer / more checkpoints may be present.

[0087] like Figure 4 As depicted, anatomical checkpoints can mark areas where more care must be taken when advancing the catheter. For example, branching vascular systems can increase the probability of the catheter deviating from the planned path through the vascular system. Proximity to a particular anatomical checkpoint can inform how various parameters associated with the procedure should be changed. For example, external X-ray imaging is often used for guidance purposes. Near a checkpoint, higher quality X-ray imaging may be required, which can be achieved through a higher radiation dose or a higher frame rate. Near such a checkpoint, alternative views of the anatomy may be needed, which can be achieved by repositioning the X-ray imaging arm. Other parameter settings (e.g., collimation settings) (i.e., changing the shape of the X-ray beam) and robot speeds used to mechanically move the catheter through the vascular system may also be changed. This disclosure also envisions other parameter settings that can be configured, monitored, and changed during intravascular procedures.

[0088] Figure 5This is a schematic diagram of at least a portion of an automated checkpoint identification system 500 according to various aspects of this disclosure. The automated checkpoint identification system 500 generates a patient model with checkpoints 512 and determines action decisions and / or (one or more) recommendations 525. User input (e.g., patient health information 504 and / or image-derived measurements 506) and selections can take the form of preoperative medical images (i.e., images taken prior to the planned procedure), patient information, medical records, procedure type, and the output can be a patient model with checkpoints 512 at each checkpoint, along with action decisions and / or (one or more) recommendations 525. The user can be any of many medical professionals. In some cases, the user's selections can cause the system 500 to automatically collect additional information based on selections from various databases (e.g., based on patient identifiers). The automated checkpoint identification system 500 may include a patient model estimator 120 and an action controller 520. Although depicted as two separate boxes, the patient model estimator 120 and the action controller 520 may comprise a single module.

[0089] The automated checkpoint identification system 500 can receive patient measurement results 502 from a medical device console 140, a user computer 160, or a medical imaging console 170. Patient measurement results 502 may include patient health information 504 and / or image-derived measurement results 506. In some cases, patient health information 504 can be entered by the user using a selection interface, allowing physicians to identify patients by patient identifiers, enabling retrieval of patient health information from a database, or allowing physicians to directly input patient health information. Patient health information may include patient height, weight, BMI, age, etc. In some cases, image-derived measurement results 506 can be derived from any number of imaging modalities (e.g., ultrasound applications, camera-based video, X-ray video, and 3D volumetric images (e.g., computed tomography (CT) scans, 3D rotational angiography (3DRA) scans, magnetic resonance imaging (MRI) scans, optical coherence tomography (OCT) scans, or intravenous ultrasound (IVUS) pull-back sequences)). Imaging modalities that can measure anatomical structures to scale may be particularly useful (e.g., CT, MRI, etc.). Preoperative images from these modalities may include medical image reports with measurements derived from the already determined images. Alternatively, preoperative images may be processed by convolutional neural networks or other image processing systems to derive measurements from the images. For example, convolutional neural networks can detect objects and / or segmented objects with known shapes or measurements in preoperative images, which can be used to scale anatomical measurements.

[0090] The patient model estimator 120 may receive patient measurements 502. The patient model estimator 120 may include shape features 508. The patient model estimator 120 may include several different structures, including linear / polynomial regression models (e.g., principal component analysis) or nonlinear regression models (e.g., neural network-based variational autoencoders, neural network-based diffusion models). Shape features 508 may include learned shape features, such as components describing the mean shape (P̃) or the mean (μ) of the shape on the learned representation, and the shape variation pattern (D) or the variance (σ) of the shape on the learned representation. Shape features may vary depending on the type of patient model estimator. In some aspects, the patient model estimator 120 may also receive intraoperative images, measurements, and / or imaging settings 522 (e.g., X-ray images, measurements derived from X-ray images, and settings of imaging equipment used for X-ray imaging (e.g., medical imaging console 170 and associated equipment 180)). As X-ray images are acquired during the procedure, new measurements of the patient may become available (e.g., if a 3DRA of the cerebral vascular system is acquired, the size of the patient's head may also become available). These new measurements can be used to update the estimated shape of the patient, ensuring that the new shape matches the previously available measurements. This allows the automated checkpoint identification system to keep the estimated patient shape and checkpoints updated as more information (e.g., more measurements) becomes available. This also applies to other imaging modalities (e.g., CT in a CT-guided procedure).

[0091] like Figure 8 As depicted, the patient model estimator 120 can receive patient health information 804 and measurement results 806 derived from preoperative images and / or images taken during the procedure (804 and 806 are related to...). Figure 5 Similar to 504 and 506 depicted in [the text]. For example, the patient model estimator 120 may receive a measurement 806 giving the patient's relative hip width relative to a known workbench width. Based on these inputs, the patient model estimator 120 may generate a 3D patient model 810 with multiple labeled example anatomical checkpoints: entry site 812, iliac 814, kidney 816, aortic arch 818, neck 820, and head 822. Although identified on the patient model 810, the anatomical checkpoints are associated with medical process-related checkpoints in the actual vascular system / anatomical structure (e.g., as per [the text]). Figure 4 (as described).

[0092] In some respects, the patient model estimator 120 includes a linear / multinomial regression model. By using the regression model, the patient model estimator 120 generates an estimate of the current patient shape based on the measurement results and learned shape features.

[0093] For example, patient measurement results Including principal component analysis coefficients Together they can be used to estimate regression coefficients via a linear regression model. :

[0094] The patient's shape can be calculated based on the patient's measurement information using the following formula. Estimate :

[0095] in This represents the average 3D shape calculated over all shapes. It corresponds to A subset of the eigenvectors with the highest eigenvalues ​​(representing the variation patterns in the 3D shape dataset; also known as shape variation patterns), and These represent PCA coefficients (or pattern weights describing how each 3D shape deviates from the average shape). In some respects, these features and parameters are shape features 508. This is an example of how a 3D shape can be represented in a parameter space learned using PCA. The shape of a new patient (and checkpoints / landmarks) can be estimated within this parameter space. It can be estimated using the following formula via linear or polynomial regression:

[0096] PCA coefficients for new patients (without shape information) can be estimated, for example, via gradient descent. This can be performed... The first estimate (e.g., with a value close to zero) A dimensional vector makes the estimation result approximate the average shape. Then, based on known patient measurements Compared with the patient shape estimated by using equation (2) Extracted estimated measurement results The error between the estimates is used to iteratively improve the results. As this is repeated, the estimated patient shape... Accuracy has been improved, and therefore, as described in this paper, the estimation of checkpoints from patient shape has also been improved. It can also be used to directly estimate the patient's shape by using the first equation of equation (2), bypassing the need for... The estimate, in which the measurement results Used only for iteratively improving the estimate And therefore also used for patient shape However, it is not used directly as on the right side of equation (2). The estimate.

[0097] In some aspects, the patient model estimator 120 may include one or more nonlinear regression models or neural networks suitable for generating a patient model representing the shape of a patient (e.g., a set of vertices representing the shape of a patient (and triangles, squares, and other polygons, or combinations thereof, possibly associated with tessellations including a 3D patient model)). In some aspects, the patient model estimator 120 includes a neural network-based variational autoencoder (VAE), such as... Figure 9 As stated above.

[0098] Figure 9 This is a schematic diagram of a variational autoencoder 900. Another example of learning a 3D shape representation in parameter space is through the use of a neural network architecture (e.g., a variational autoencoder 900). During training, the VAE 900 can encode a patient model 902 using an encoder 904. The encoder 904 encodes the patient model into a probability distribution 906, which can be sampled on a latent representation 908. The latent representation 908 can then be decoded using a decoder 910 to generate a reconstructed patient model 912. Such a model can be trained by updating the encoder and decoder to reproduce the original patient model 902 as a predicted output 912. During training, the VAE 900 learns the mean μ and variance σ, i.e., learns the distribution 906 (on the latent representation describing the training data), from which a latent representation of the input shape 908 is sampled. The input shape 912 is then decoded to reconstruct it. In some respects, the mean μ and variance σ can be shape features 508. Patient measurements can be combined with the learned features via a cascade of the sampled latent representation or with a layer in an earlier layer (e.g., a linear layer) of the decoder 910. Once the distribution 906 has been learned and the decoder 910 has been trained, it is used during inference to generate the patient model. In other words, patient health information and other measurements are input, the distribution 906 is sampled, and the decoder guided by the input generates the patient model.

[0099] By using variational autoencoder components μ and σ, it is possible to sample from the distribution (described by μ and σ) in the latent representation space. And the cascaded measurement results Decode together, to decode the shape, so that it matches the patient's measurement results. Matching. Patient measurement results. This includes, for example, preoperative patient measurements 502 and / or intraoperative patient measurements 522. The initial z can describe a shape close to the mean μ, since the patient shape is unknown. This can be achieved by using known measurements. Estimated measurements of patient shape from estimation or decoding The error between (e.g., height) is fine-tuned in the decoder 910 via gradient descent (as described in this paper) to iteratively improve the estimated shape. A relevant loss function (e.g., mean squared error (MSE) or mean absolute error (MAE)) can be used. The error can be backpropagated through the decoder layers to update... This allows the updated version to be decoded. To estimate a more accurate patient shape (i.e., and (The error between them is smaller). (Refer to...) Figure 9 The variational autoencoder architecture is further described. See also... Figure 3 The neural network architecture is described. The VAE used during inference can be used as a patient model estimator 120.

[0100] In some aspects, the patient model estimator 120 includes a neural network-based diffusion model. The conditional diffusion model can be used to estimate the 3D patient shape based on measurements, where the generated output is based on known patient measurements. As a condition. Since the estimated patient shape should be constrained to a range of feasible human body shapes, shape estimation can be derived from the average patient shape. To begin, in the diffusion model, noise is added to the initial estimate (e.g., average shape), and then that initial estimate is denoised. This is done by applying patient-specific measurements. Conditioned for denoising, and by comparing the estimated patient shape with the baseline true patient shape during training using reconstruction losses (e.g., MSE, MAE, etc.), a diffusion model can be trained to correlate measurements with shape characteristics (e.g., height versus head-to-toe distance of the patient shape). The trained diffusion model can receive a noisy initial shape (e.g., a noisy average patient shape). ) and patient measurement results Conditional diffusion models can be used to regulate information. The process involves guiding the diffusion sampling process (i.e., estimating the noise to be removed) and iteratively estimating the patient shape, such that known measurements from the estimated patient shape are obtained. and estimated measurement results Minimize the error between them. Known patient measurements can include and / or be derived from patient health information, preoperative images / measurements, and / or intraoperative images / measurements. All of the above models (PCA, VAE, diffusion, etc.) can be modified to directly estimate checkpoints rather than patient shape.

[0101] Furthermore, the patient model estimator 120 may include a checkpoint generator 510 to generate checkpoints (e.g., anatomical checkpoints as described herein) relative to the patient body model. If PCA is used, then known reference checkpoints from the mean shape can be transferred to the estimated shape using equation (2), or if VAE or a diffusion model is used, then known reference checkpoints from the mean shape can be transferred to the estimated shape via registration with the mean shape. The user can identify a first checkpoint (e.g., Figure 4 The "entry point" (402) can be used to estimate the location of all other checkpoints relative to that checkpoint. System data, including imaging systems (e.g., C-arm imaging systems or robotic systems), can be used to infer whether subsequent checkpoints have been reached. For example, if the C-arm or patient table has been moved a distance from the "entry point" checkpoint... This distance matches the distance between the checkpoint "entry point" and "iliac" (see...). Figure 4 and Figure 8 If the device moves in the X-ray image (e.g., using image processing or tracking equipment, including electromagnetic or EM tracking, shape sensing (e.g., Philips Fiber Optics RealSense or FORS)) or by tracking a robot (e.g., if the robot has moved the device a certain distance)...), then the system can infer that it has reached the "iliac" checkpoint. This is used to infer checkpoints. In some cases, once X-ray images of the expected checkpoint are acquired, the images can be analyzed to assess whether the checkpoint is in the expected location. For example, once an X-ray image of the aortic arch is acquired, the now-known distance the C-arm travels between the entry point and the aortic arch can be compared to the predicted distance. If the error exceeds a certain threshold, then, in addition to the previous measurements, these measurements can be used to refine the estimated shape of the patient and all subsequent checkpoints.

[0102] In some respects, depth sensors (e.g., infrared or IR sensors, hyperspectral cameras), RGB cameras, or other sensors in the procedure room can also be used to estimate patient measurements and for 3D shape estimation. However, patients are typically covered by a surgical drape, so specific features (e.g., the patient's head or other uncovered features) can be used to estimate some measurements. Alternatively, a model can be trained to estimate patient measurements (or directly estimate patient shape) through the surgical drape.

[0103] In some respects, the first checkpoint can also be identified using image processing techniques. For example, if no checkpoint is identified, any acquired X-ray image can be classified as one of several entry points (e.g., left femur, right femur, left radial, right radial, etc.). Other checkpoints can then be estimated relative to this checkpoint.

[0104] The action controller 520 can receive a patient model with anatomical checkpoints generated by the patient model estimator 120, as well as intraoperative images, measurements, and / or image settings 522. The action controller 520 may include a dictionary of system control decisions to suggest or trigger tasks or actions relevant to the current process at each checkpoint. (Described below) Figure 10 An example of controlling the dictionary is described in the text.

[0105] Figure 10 A simplified example control dictionary 1000 is depicted according to various aspects of this disclosure. The control dictionary includes conditions 1005 and associated actions 1010. In some aspects, conditions may include distance thresholds. For example, a first condition may be formulated as "within 2 cm of the iliac checkpoint," and when this condition is met, a suggestion or control signal is generated to reduce (e.g., X-ray) dose, or relatedly, reduce the frame rate of X-ray imaging. In some aspects, the action controller 520 is capable of comparing checkpoints generated using a patient model with the progress of the medical device traveling through the vascular system by first registering the patient body model with X-rays (or other intraoperative imaging being employed). Once registered, the distance between the location of the portion of the medical device traversing the vascular system and the location of the checkpoint can be calculated.

[0106] The following are examples of conditions that must be met for the associated recommendations / actions. During a neurointerventional procedure, at the "iliac" checkpoint (e.g., 814), the dose / frame rate of X-ray image acquisition is reduced so that the image quality is sufficient to visualize the device, but anatomical features may not be very clear, as the images are only needed to ensure the device is oriented correctly, not for diagnosing anatomical features or for treatment delivery. During a neurointerventional procedure, at the "iliac" checkpoint (e.g., 814), the vertical collimator diaphragm is moved inward to block anatomical structures far from the vascular system (i.e., anatomical structures that do not need to be visualized for successful navigation). During a neurointerventional procedure, at the "iliac" checkpoint (e.g., 814), the robot navigation speed is increased because navigation through these large vessels is relatively easy. During an endovascular aneurysm repair (EVAR) procedure, at the "iliac" checkpoint (e.g., 814), the robot navigation speed is reduced to slowly approach the "kidney" checkpoint so that the renal artery can be inserted. During neurointerventional procedures, at the “kidney” checkpoint (e.g., 816), the robot navigation speed (relative to the “iliac” checkpoint) is maintained as long as the device is moving in a straight line (i.e., the device has not accidentally inserted into the renal artery). At the “aortic arch” checkpoint (e.g., 818) during neurointerventional procedures, the robot navigation speed is reduced to carefully insert the appropriate branch (e.g., the vertebral artery or carotid artery) from the aortic arch. At the “head” checkpoint (e.g., 822) during neurointerventional procedures, the dose of X-ray image acquisition is increased because this is a critical area for treatment delivery and high image quality is required to administer optimal treatment. At the “head” checkpoint (e.g., 822) during neurointerventional procedures, the collimation / digital zoom settings are changed so that the imaging field of view is focused around the treatment target area. These are merely a few examples; this disclosure also envisions control dictionaries for other procedures and other imaging devices.

[0107] Return to Figure 5 The motion controller 520 selects or generates one or more motion decisions and one or more suggestions 525. When the motion controller 520 is configured to directly control and reconfigure medical imaging equipment and medical devices (e.g., such as...), Figure 1 When the motion controller is described as being at 140, 150, 170, and / or 180 degrees, the motion decision may include instructions for changing parameter settings of various systems utilized during the process (e.g., X-ray imaging support systems or robot speed controlling the progress of catheters through vascular systems). Even if the motion controller is configured to control systems associated with the process, the user can still be notified of the motion taken via a screen display, which may have the opportunity to override and change it.

[0108] Screen displays can be generated for display on any number of devices (e.g., monitors 116, 146, 166, or 176). The screen display may include intraoperative images 522 (e.g., X-rays) and action decisions and / or suggestions 525, as described herein. Figure 13 An example screen display is provided.

[0109] The automated checkpoint identification system 500 can also estimate confidence metrics. Confidence can be related to data quality (e.g., the number of measurements provided, where fewer measurements are associated with lower confidence, image quality, where low-dose images are associated with lower confidence, etc.). Confidence can also be calculated using dropout or other techniques that can be used in neural network implementations. Dropout ignores the outputs from different (randomly selected) nodes in the neural network layers during different training epochs. During inference, data flows through the trained network multiple times, where again, different nodes are ignored during different inference runs that produce different outputs. If the network has confidence, the differences in outputs are small. However, if the network has no confidence, the differences in outputs are large.

[0110] Figure 6 This is a schematic diagram of at least a portion of a training system 600 for a patient model estimator 120 according to various aspects of this disclosure. The training system 600 trains the patient model estimator 120 to accurately predict patient models and checkpoints based on the context provided at the time of inference. The patient model estimator 120 is trained by providing a known patient model 627 (e.g., a 3D shape model consisting of vertices and triangles) with user-annotated checkpoints 629 and associated patient health information 621, preoperative images and / or measurements 623, and intraoperative images and / or measurements 625. For example, preoperative images may be CT or MRI, and intraoperative images may be X-ray or ultrasound scans. The known patient model 627 can be obtained from preoperative images, such as volumetric images obtained from CT or MRI scans. Measurements associated with both preoperative and intraoperative images may include various anatomical measurements discernible within the field of view of the images (e.g., such as...). Figure 8 (The ratio of the depicted hip to the table width). This information can be recorded from earlier processes. Not all types of information are available for every patient. The training system 600 includes a patient model simulator or estimator 120 and a model objective / function 650.

[0111] Training data 610 can be stored in training database 124 of network / cloud computer 110. Training data 602 can include one or more of the following for each patient: patient model 627 with user-annotated checkpoints 629, patient health information 621, preoperative images / measurements 623, and intraoperative images / measurements 625. Patient model 627 and user-annotated checkpoints 629 represent the baseline ground truth that the patient model estimator 120 attempts to predict / reproduce during the operation of training system 600. User-annotated checkpoints can alternatively be annotated only on one patient model in the database or on the mean shape or mean patient model. Assuming all patient models are registered, these checkpoints can be transferred between all patient models.

[0112] The patient model estimator 120 receives patient health information (e.g., patient EMR, age, sex, weight, height, etc.), preoperative images and / or measurements 623, and / or intraoperative images and / or measurements 625, and generates a predicted 3D patient model and one or more predicted anatomical checkpoints. The predicted patient model and checkpoints 640 may be as described herein and specifically referenced. Figure 5 Generation. The predicted model and checkpoint 640 can be compared to the baseline true patient model 627 and the user-annotated checkpoint 629 using model objective / function 650. Model objective / function 650 may include an objective / function that penalizes the predicted patient model and checkpoint 640 to a greater or lesser degree, wherein the greater the penalty, the further the predicted patient model is from the baseline patient model 627, and the further the predicted checkpoint is from the user-annotated checkpoint 629; the closer they are, the smaller the penalty. In some aspects, model objective / function 650 may be mean squared error or mean absolute error.

[0113] The comparison results from the model objective / function 650 can update the parameters 660 of the patient model estimator 120. In some cases, the update can be accomplished by using the gradient of the objective function and backpropagation to update the parameters of the patient model estimator 120.

[0114] Similarly, the motion controller 520 can be trained to learn a control dictionary. Training data can be stored in a training database 124 of the network / cloud computer 110. The training data for the motion controller can include imaging data (e.g., X-ray images showing device movement, collimation settings, X-ray frame rates, etc.), robot data (e.g., the speed at which the device is manipulated, etc.), and information used from any other device. During training, the motion controller 520 can (e.g., by parsing relevant information such as X-ray frame rates, collimation settings, robot speeds, etc., and / or by segmenting devices from images using conventional image processing methods or machine learning / deep learning methods to infer the device type or device speed on the X-ray frame, etc.) receive a large amount of patient data and extract relevant information. The extracted information is then (using any method including simple averaging or more complex modeling) summarized to construct a dictionary of system control decisions or actions (e.g., with...). Figure 10 (Similarly), to suggest or trigger these system control decisions or actions at each checkpoint of a given process.

[0115] Figure 7 This is a second schematic diagram of a training system 600 for an untrained patient model estimator 730, based on various aspects of this disclosure. Figure 6 As described, after training is complete, the untrained patient model estimator 730 with parameter A is transformed into a trained patient model estimator 740 with parameter B. Parameters A and B differ between the trained and untrained estimators because, during training, the parameters in the patient model estimator are updated based on comparisons between the baseline ground truth labels and predictions (e.g., ...). Figure 6 (660 in the middle).

[0116] Figure 11 The operation of an action controller during an intravascular procedure is depicted according to various aspects of this disclosure. Time series 1100 is an example time series with three example times 1102, 1104, and 1106 selected and associated during different stages of the intravascular procedure. At each time slice, an image is generated by a supporting imaging system (e.g., a medical imaging console 170 and device 180), which may be an X-ray system.

[0117] At the first time point 1102, intraoperative image 1110 (e.g., X-ray image) depicts the endovascular device 1112 (e.g., 150) traversing the vascular system from its entry site. Intraoperative image 1110 is received by motion controller 520 along with an associated 3D patient model and checkpoint 1114. In some respects, motion controller 520 may need to co-register intraoperative image 1110 and 3D patient model 1114. As described herein, motion controller generates notifications / alarms and suggested actions 1116. At the first time point 1102, the endovascular device 1112 is approaching the iliac checkpoint, reflected in an alarm from motion controller 520: “Alarm: Within 4 cm of iliac checkpoint.” Motion controller 520 may also generate suggested actions: “Suggestion: Reduce X-ray dose.”

[0118] At a second time point 1104, intraoperative image 1120 (e.g., an X-ray image) depicts the intravascular device 1112 (e.g., 150) traversing the vascular system from its entry site. As depicted in image 1120, the procedure has progressed to a new stage, where the device 1112 has traveled further through the vascular system compared to the first time point 1102. Intraoperative image 1120 is received by motion controller 520 along with an associated 3D patient model and checkpoint 1124. The 3D patient model and checkpoint 1124 may differ from earlier models and checkpoints (e.g., 1114) because patient model estimator 120 generates an updated model based on new measurements, as described herein. In some aspects, motion controller 520 may only need to receive the 3D patient model and checkpoint 1124 once. In other aspects, motion controller 520 may need to co-register intraoperative image 1120 and 3D patient model 1124. As described herein, motion controller generates notifications / alerts and suggested actions 1126. At the second time point 1104, the intravascular device 1112 is approaching the kidney checkpoint, reflected in an alarm from the motion controller 520: "Alarm: Within 2 cm of the kidney checkpoint." The motion controller 520 can also generate a suggested action: "Suggestion: Maintain navigation speed."

[0119] At a third time point 1106, intraoperative image 1130 (e.g., X-ray image) depicts the endovascular device 1112 (e.g., 150) traversing the vascular system from its entry site. As depicted in image 1130, the procedure has progressed to a new stage, where the device 1112 has traveled further through the vascular system compared to the first time point 1102 and the second time point 1104. Intraoperative image 1130 is received by motion controller 520 along with an associated 3D patient model and checkpoint 1134. The 3D patient model and checkpoint 1134 may differ from earlier models and checkpoints (e.g., 1114, 1124) because patient model estimator 120 generates an updated model based on new measurements, as described herein. In some aspects, motion controller 520 may only need to receive the 3D patient model and checkpoint 1134 once. In other aspects, motion controller 520 may need to co-register intraoperative image 1130 and 3D patient model 1134. As described herein, the motion controller generates notifications / alarms and suggested actions 1136. At a third time 1106, the intravascular device 1112 is near the aortic arch checkpoint, reflected in an alarm from the motion controller 520: "Alarm: At the aortic arch checkpoint." The motion controller 520 may also generate suggested actions: "Suggestion: Reduce navigation speed."

[0120] The actions indicated by the alerts and recommendations are merely illustrative, and many other actions may be indicated depending on the procedure, medical equipment, and stage of the procedure.

[0121] Figure 12 This is a schematic flowchart of a method 1200 for automatic checkpoint identification based on various aspects of this disclosure. It should be understood that the steps of method 1200 can be... Figure 12 The different sequences of execution shown may provide additional steps before, during, and after the steps, and / or may otherwise replace or eliminate some of the described steps. One or more steps of method 1200 may be performed by one or more devices and / or systems described herein (e.g., components and / or processor circuitry 250 of networking system 100).

[0122] In step 1202, method 1200 includes receiving user input from a user input device, including patient health information or preoperative images. For example, a doctor may input patient health information 504 into user computer 160 via input device 164. In some aspects, preoperative images of the patient stored locally on the user computer may be uploaded, or relevant images may be optionally stored elsewhere (e.g., on network / cloud computer 110).

[0123] In step 1204, method 1200 includes generating a 3D patient model with one or more anatomical examination points from user input as the output of a patient model estimator. For example, patient model estimator 120 generates a patient model and anatomical examination points 512 based on patient measurements 502. In some aspects, patient model estimator 120 can generate a patient model and examination points based on intraoperative images and / or measurements 522.

[0124] In step 1206, method 1200 includes receiving one or more imaging frames from an external imaging system to track a medical procedure. For example, the medical imaging console 170 and medical imaging device 180 may be an external X-ray imaging system with a C-arm whose movement can be tracked and recorded. The external X-ray imaging system may generate X-ray frames and transmit them to other devices in the networked system 100.

[0125] In step 1208, method 1200 includes detecting the progress of a medical procedure by a controller based on the proximity of an intravascular device to one or more anatomical examination points. For example, motion controller 520 may receive a patient model and examination point 512, as well as intraoperative images from, for example, external X-ray imaging systems 170, 180, and compare the progress of the intravascular device (e.g., 1112) through the patient's vascular system, such as... Figure 11 As stated above.

[0126] In step 1210, method 1200 includes generating action suggestions by the controller based on proximity. For example, the action controller 520 action dictionary (e.g., 1000) may be searched to determine whether condition 1005 has been met, and if so, the associated action 1010 is selected / output to the met condition 1005.

[0127] In step 1212, method 1200 includes a display screen showing one or more imaging frames, notifications corresponding to action suggestions, and anatomical checkpoints. For example, display 1300 depicts imaging frame 1310, alarm 1340, action suggestion 1345, and anatomical checkpoint 1315.

[0128] Figure 13This is an example display 1300 for displaying the output of the action controller 520 according to various aspects of this disclosure. The example display 1300 may be a user interface provided to a medical personnel via a display at a medical device console 140, a user computer 160, or a medical imaging console 170. The display 1300 may include: a live intraoperative imaging frame 1310 (e.g., an X-ray), an anatomical checkpoint 1315, settings 1332-1334-1336 associated with the medical imaging console 170 and device 180 (e.g., an X-ray), notifications / alarms 1340, and suggested actions 1345. The intraoperative imaging frame 1310 may depict an intravascular device 1112 (e.g., similar to 150 in the networked system 100) near the iliac checkpoint 1315. The iliac checkpoint 1315 is marked on the imaging frame 1310. Settings A, B, and C 1332, 1334, and 1336 can be various imaging settings for external imaging systems (e.g., X-rays) and / or intravascular devices. For example, setting A 1332 can be the X-ray dose, setting B 1334 can be the collimation setting, and setting C 1336 can be the robot speed of the intravascular device, where the robot speed controls how fast the intravascular device navigates the vascular system. In some aspects, the settings can be changed by a user using an input device on a relevant device in the networked system 100.

[0129] In some respects, the action controller can directly control medical imaging equipment and / or medical device 150 (e.g., X-ray system and / or intravascular device) based on suggested actions. In such cases, the actions taken by the controller are provided on display 1300 as notifications, rather than as suggested actions performed by the user.

[0130] Furthermore, the techniques disclosed herein are also applicable to other medical imaging modalities obtained from medical imaging devices in which 3D data is available (e.g., other 3D ultrasound applications, camera-based video with post-processed 3D reconstruction, X-ray images from multiple views, fiber optic shape sensing, and 3D volumetric images (e.g., computed tomography (CT) scans, magnetic resonance imaging (MRI) scans)). The techniques described herein can be used in a variety of environments, including emergency departments, intensive care units, inpatient wards, and out-of-hospital environments.

[0131] Therefore, the logical operations constituting the aspects of the technology described herein are referred to differently as operations, steps, objects, layers, elements, components, algorithms, or modules. Furthermore, it should be understood that unless expressly specified in the claims, or a particular order is inherently necessary to the language of the claims, these logical operations may occur, be performed, or be arranged in any order.

[0132] All directional expressions (e.g., up, down, inside, outside, upward, downward, left, right, sideways, front, back, top, bottom, above, below, vertical, horizontal, clockwise, counterclockwise, near, and far) are used for illustrative purposes only to aid the reader's understanding of the claimed subject matter and do not create limitation. Connectivity expressions (e.g., attachment, coupling, connection, combination, or "communicating with") should be interpreted broadly and, unless otherwise stated, may include intermediate members between sets of elements and relative movement between elements. Thus, connectivity expressions do not necessarily mean that two elements are directly connected and in a fixed relationship with each other. The term "or" should be interpreted as meaning "and / or" rather than "exclusively or". The word "comprising" does not exclude other elements or steps, and the quantifiers "a" or "an" do not exclude a plurality. Unless otherwise indicated in the claims, the values ​​stated should be interpreted as illustrative only and should not be considered restrictive.

[0133] The foregoing specification, examples, and data provide a complete description of the structure and use of exemplary aspects of the automated checkpoint identification system as described herein. Although various aspects of the claimed subject matter have been described above with a degree of specificity or by reference to one or more individual aspects, those skilled in the art can make many changes to the disclosed aspects without departing from the spirit or scope of the claimed subject matter.

[0134] Other aspects are also envisioned. It is intended that everything contained in the foregoing description and shown in the accompanying drawings be interpreted as illustrative of specific aspects only, and not as limiting. Changes in detail or structure may be made without departing from the essential elements of the subject matter as defined in the following claims.

Claims

1. A computer-implemented method, comprising: Receive user input from a user input device, the user input including patient health information or preoperative images; A 3D patient model is generated as the output of the patient model estimator, the 3D patient model having one or more anatomical examination points from the user input; Receive one or more imaging frames from an external imaging system to track medical procedures; The progress of the medical procedure is detected by the controller based on the proximity of the intravascular device to one of the one or more anatomical examination points; as well as The controller generates action suggestions based on the proximity.

2. The method according to claim 1, further comprising: Register the one or more imaging frames with the 3D patient model; as well as Output a screen display to the display, the screen display including: the one or more imaging frames, a notification corresponding to the action suggestion, and anatomical checkpoints.

3. The method according to claim 2, in, The imaging frames include X-ray frames, and The action recommendations include at least one of the following: changing navigation speed, X-ray dose, or X-ray collimation settings.

4. The method according to claim 1, further comprising: The patient model estimator generates an updated 3D patient model based on the user input and at least one of the one or more imaging frames.

5. The method according to claim 1, further comprising: The controller automatically controls the external imaging system or the intravascular device based on the action recommendations.

6. The method according to claim 1, wherein, Generating the 3D patient model also includes: The 3D patient model is generated via the patient model estimator based on one or more shape features, wherein the shape features include an average patient shape and one or more shape variation patterns; and One or more reference checkpoints on the average patient shape are mapped to one or more anatomical checkpoints on the 3D patient model via the patient model estimator.

7. The method according to claim 1, wherein, The patient model estimator includes a neural network-based variational autoencoder, and wherein generating the 3D patient model having the one or more anatomical examination points further includes: Samples are generated via the patient model estimator based on a latent shape distribution parameterized by one or more shape features, wherein the shape features include mean and variance; and The 3D patient model is generated based on the sample and the user input as the output of the decoder in the neural network-based variational autoencoder; Perform registration of the 3D patient model with the average patient shape including one or more reference checkpoints; and The patient model estimator maps the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model with the average patient shape.

8. The method according to claim 1, wherein, The patient model estimator includes a neural network-based diffusion model, and wherein generating the 3D patient model having the one or more anatomical examination points further includes: A 3D patient model is generated using the neural network-based diffusion model based on the noisy average patient shape and the user input. Perform registration of the 3D patient model with the average patient shape including one or more reference checkpoints; and The patient model estimator maps the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model with the average patient shape.

9. A system comprising: The processor circuitry is configured as follows: Receive user input from a user input device, the user input including patient health information or preoperative images; A 3D patient model is generated as the output of the patient model estimator, the 3D patient model having one or more anatomical examination points from the user input; Receive one or more imaging frames from an external imaging system to track medical procedures; The progress of the medical procedure is detected by the controller based on the proximity of the intravascular device to one of the one or more anatomical examination points; as well as The controller generates action suggestions based on the proximity.

10. The system according to claim 9, wherein, The processor circuit is also configured to: Register the one or more imaging frames with the 3D patient model; as well as Output a screen display to the display, the screen display including: the one or more imaging frames, a notification corresponding to the action suggestion, and anatomical checkpoints.

11. The system according to claim 10, in, The imaging frames include X-ray frames, and The action recommendations include at least one of the following: changing navigation speed, X-ray dose, or X-ray collimation settings.

12. The system according to claim 9, wherein, The processor circuit is also configured to: The patient model estimator generates an updated 3D patient model based on the user input and at least one of the one or more imaging frames.

13. The system according to claim 9, wherein, The processor circuit is also configured to: The controller automatically controls the external imaging system or the intravascular device based on the action recommendations.

14. The system according to claim 9, wherein, Generating the 3D patient model also includes: The 3D patient model is generated via the patient model estimator based on one or more shape features, wherein the shape features include an average patient shape and one or more shape variation patterns; and One or more reference checkpoints on the average patient shape are mapped to one or more anatomical checkpoints on the 3D patient model via the patient model estimator.

15. The system according to claim 9, wherein, The patient model estimator includes a neural network-based variational autoencoder, and wherein generating the 3D patient model having the one or more anatomical examination points further includes: Samples are generated via the patient model estimator based on a latent shape distribution parameterized by one or more shape features, wherein the shape features include mean and variance; and The 3D patient model is generated based on the sample and the user input as the output of the decoder in the neural network-based variational autoencoder; Perform registration of the 3D patient model with the average patient shape including one or more reference checkpoints; and The patient model estimator maps the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model with the average patient shape.

16. The system according to claim 9, wherein, The patient model estimator includes a neural network-based diffusion model, and wherein generating the 3D patient model having the one or more anatomical examination points further includes: A 3D patient model is generated using the neural network-based diffusion model based on the noisy average patient shape and the user input. Perform registration of the 3D patient model with the average patient shape including one or more reference checkpoints; and The patient model estimator maps the one or more reference checkpoints to the one or more anatomical checkpoints on the 3D patient model based on the registration of the 3D patient model with the average patient shape.

17. A non-transient machine-readable medium comprising a plurality of machine-executable instructions, the plurality of machine-executable instructions being adapted, when executed by one or more processors, to cause the one or more processors to perform operations, the operations including: Receive user input from a user input device, the user input including patient health information or preoperative images; A 3D patient model is generated as the output of the patient model estimator, the 3D patient model having one or more anatomical examination points from the user input; Receive one or more imaging frames from an external imaging system to track medical procedures; The progress of the medical procedure is detected by the controller based on the proximity of the intravascular device to one of the one or more anatomical examination points; as well as The controller generates action suggestions based on the proximity.

18. The non-transient machine-readable medium according to claim 17, wherein, It also causes the one or more processors to perform operations including the following: Register the one or more imaging frames with the 3D patient model; as well as Output a screen display to the display, the screen display including: the one or more imaging frames, a notification corresponding to the action suggestion, and anatomical checkpoints.

19. The non-transient machine-readable medium according to claim 18, in, The imaging frames include X-ray frames, and The action recommendations include at least one of the following: changing navigation speed, X-ray dose, or X-ray collimation settings.

20. The non-transient machine-readable medium according to claim 17, wherein, It also causes the one or more processors to perform operations including the following: The controller automatically controls the external imaging system or the intravascular device based on the action recommendations.