Systems and methods for automated robotic intravascular medical device manipulation

Abstract

One or more example embodiments provide a method comprising: obtaining, by a trained system, at least one movement command for an interventional device of a robotic surgical system to a location within a patient; determining, by the trained system, at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the interventional device; and applying the actuator command to the robotic surgical system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority under 35 USC 119(e) to U.S. Provisional Patent Application No. 63 / 485,582, filed on February 17, 2023, entitled “AI-based Agents for Automated Robotic Endovascular Guidewire Manipulation,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] One or more example embodiments are directed to systems and methods for automated robotic intravascular medical device manipulation. Background Art

[0004] Intravascular guidewire manipulation is used in minimally invasive clinical applications. Percutaneous coronary intervention (PCI) is used to open narrowed coronary arteries and restore arterial blood flow to heart tissue, mechanical thrombectomy techniques are used in acute ischemic stroke (AIS) to remove blood clots from cerebral veins, and transjugular intrahepatic portosystemic shunt (TIPS) is used for portal hypertension, using a special needle and wire placed across the portal vein that passes through the liver.

[0005] These procedures typically use 3D vessel geometry from 3D CTA (Computed Tomography Angiography) images. Summary of the Invention

[0006] The independent claims set out the scope of protection sought by various example embodiments.Example embodiments and / or features described in this specification that do not fall under the scope of the independent claims, if any, should be construed as examples to aid in understanding the various embodiments.

[0007] Independent of the grammatical term usage, individuals with masculine and feminine genders are included in the term.

[0008] Unfortunately, conventional X-ray targets and associated components are not well-suited to the high currents and / or transient dose rates characteristic of UHDR radiotherapy. For example, conventional X-ray targets may degrade and / or be destroyed if exposed to the high beam currents and / or transient dose rates characteristic of UHDR radiotherapy.

[0009] According to at least one example embodiment, a method includes: obtaining, by a trained system, at least one movement command for a robotic surgical system to position an interventional device within a patient's body; determining, by the trained system, at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the interventional device; and applying the actuator command to the robotic surgical system.

[0010] According to at least one example embodiment, the interventional device is a guidewire.

[0011] According to at least one example embodiment, the at least one movement command is a desired position of the guidewire tip within the patient's body.

[0012] According to at least one example embodiment, the method further comprises obtaining an observation position of the guidewire from the robotic surgical system; and determining an estimated current position of the guidewire based on the observation position.

[0013] According to at least one example embodiment, the observation position of the interventional device corresponds to the observation position of the proximal portion of the guidewire.

[0014] According to at least one example embodiment, the estimated current position corresponds to an estimated position of the guidewire tip.

[0015] According to at least one example embodiment, the determining step determines a series of actuator commands, each actuator command in the series of actuator commands corresponding to a state of the guidewire.

[0016] According to at least one example embodiment, the method further comprises generating image data showing an estimated current position of the guidewire within the patient's blood vessel.

[0017] According to at least one example embodiment, the estimated position of the guidewire is overlaid on the three-dimensional vessel segmentation.

[0018] According to at least one example embodiment, the actuator command is at least one of a speed command or a torque command of the actuator.

[0019] According to at least one example embodiment, a trained system includes processing circuitry configured to cause the trained system to obtain at least one movement command for a position of an interventional device of a robotic surgical system into a patient's body, determine at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the interventional device, and apply the actuator command to the robotic surgical system.

[0020] According to at least one example embodiment, the interventional device is a guidewire.

[0021] According to at least one example embodiment, the at least one movement command is a desired position of the guidewire tip within the patient's body.

[0022] According to at least one example embodiment, the processing circuit is configured to cause the trained system to obtain an observed position of a guidewire from the robotic surgical system; and determine an estimated current position of the guidewire based on the observed position.

[0023] According to at least one example embodiment, the observation position of the interventional device corresponds to the observation position of the proximal portion of the guidewire.

[0024] According to at least one example embodiment, the estimated current position corresponds to an estimated position of the guidewire tip.

[0025] According to at least one example embodiment, the processing circuit is configured to cause the trained system to determine a series of actuator commands, each actuator command in the series of actuator commands corresponding to a state of the guidewire.

[0026] According to at least one example embodiment, the processing circuitry is configured to cause the trained system to generate image data showing an estimated current position of a guidewire within a patient's blood vessel.

[0027] According to at least one example embodiment, the estimated position of the guidewire is overlaid on the three-dimensional vessel segmentation.

[0028] According to at least one example embodiment, the actuator command is at least one of a speed command or a torque command of the actuator.

[0029] According to at least one example embodiment, a trained system includes means for obtaining at least one movement command for a position of an interventional device of a robotic surgical system into a patient's body; means for determining at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the interventional device; and means for applying the actuator command to the robotic surgical system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Example embodiments will become more fully understood from the following detailed description given herein and the accompanying drawings, in which like elements are designated by like reference numerals, which are given by way of illustration only and thus do not limit the present disclosure.

[0031] Figure 1 is a perspective view of a robotic procedure system according to one or more example embodiments;

[0032] Figure 2 is a block diagram of a robotics system according to one or more example embodiments;

[0033] Figure 3 is a block diagram of a robotic programming system depicting various actuation mechanisms according to one or more example embodiments;

[0034] Figure 4A A predictive control system according to one or more example embodiments is shown;

[0035] Figure 4B shows a flow chart for generating a physics-based simulation of a guidewire according to one or more example embodiments;

[0036] Figure 4C A flow chart illustrating generating a new mesh having a desired vessel-like structure to restrict movement of a portion of a guidewire according to one or more example embodiments is shown;

[0037] Figure 4D shows a level set map and a gradient level set map projected onto an inner cavity surface according to one or more example embodiments;

[0038] Figures 5A-5D shows a flow chart for generating a physics-based simulation of a guidewire using various images according to one or more example embodiments;

[0039] Figure 6A shows a training flow chart according to one or more example embodiments;

[0040] Figure 6B 3D blood vessel geometry with centerline is shown;

[0041] Figure 7A A flowchart illustrating an implementation of an AI predictive control system according to one or more example embodiments is shown;

[0042] Figure 7B According to one or more example embodiments Figure 7A The operation method of the AI ​​predictive control system;

[0043] Figure 8 According to one or more example embodiments Figure 7A Forward proxy shown;

[0044] Figure 9 According to one or more example embodiments Figure 7A The reverse proxy shown; and

[0045] Figure 10 An example converter model for implementing an AI predictive control system according to one or more example embodiments is shown.

[0046] It should be noted that these drawings are intended to illustrate the general characteristics of methods, structures, and / or materials used in certain example embodiments and to supplement the written description provided below. However, these drawings are not drawn to scale and may not precisely reflect the precise structure or performance characteristics of any given embodiment, and should not be interpreted as defining or limiting the range of values ​​or properties encompassed by the example embodiments. The use of similar or identical reference numerals in the various figures is intended to indicate the presence of similar or identical elements or features. DETAILED DESCRIPTION

[0047] Various example embodiments will now be described more fully with reference to the accompanying drawings, in which some example embodiments are shown.

[0048] Detailed illustrative embodiments are disclosed herein. However, the specific structural and functional details disclosed herein are merely representative for the purpose of describing the exemplary embodiments. However, the exemplary embodiments may be embodied in many alternative forms and should not be construed as being limited to the embodiments set forth herein.

[0049] It should be understood that there is no intention to limit the exemplary embodiments to the specific forms disclosed. On the contrary, the exemplary embodiments will cover all modifications, equivalents and alternatives that fall within the scope of this disclosure. Throughout the description of the drawings, the same numbers refer to the same elements.

[0050] As discussed herein, the terms "one or more" and "at least one" can be used interchangeably.

[0051] It should be understood that various example embodiments may be used in combination.

[0052] Minimally invasive clinical applications typically use 3D vessel geometry from 3D CTA (computed tomography angiography) images. During these procedures, clinicians manipulate wires through catheters and across blockages. Clinicians use X-ray fluoroscopy only intermittently to visualize and guide catheters, guidewires, and other devices (e.g., angioplasty balloons and stents).

[0053] Various types of robotic endovascular systems are being developed (e.g., Corindus TM TechNIQ TM ) to provide effective positional control of the device, helping clinicians mitigate treatment risks. The primary motions clinicians can use to control the movement and direction of the wire include rotation and pushing / retracting the proximal end of the wire away from the insertion point on the patient's body.

[0054] As used herein, the distal direction is the direction toward the patient, and the proximal direction is the direction away from the patient. The terms "upper" and "upper" refer to the general direction away from the direction of gravity, and the terms "bottom," "lower," and "lower" refer to the general direction of gravity.

[0055] Although the following example embodiments are described with reference to a control guidewire, it should be noted that other intravascular devices may also be used.

[0056] refer to Figure 1 , a robotic procedure system 10 is shown. The robotic procedure system 10 can be used to perform catheter-based medical procedures (e.g., percutaneous interventional procedures). Percutaneous interventional procedures may include diagnostic catheterization procedures, during which one or more catheters are used to help diagnose a patient's condition. For example, during one embodiment of a catheter-based diagnostic procedure, a contrast agent is injected into one or more coronary arteries via a catheter, and images of the patient's heart are taken. Percutaneous interventional procedures may also include catheter-based therapeutic procedures (e.g., balloon angioplasty, stent placement, treatment of peripheral vascular disease, etc.), during which a catheter is used to treat a condition. However, it should be noted that those skilled in the art will recognize that certain specific percutaneous interventional devices or components (e.g., type of guidewire, type of catheter, etc.) will be selected based on the type of procedure to be performed. The robotic procedure system 10 is capable of performing any number of catheter-based medical procedures with only minor adjustments to accommodate the specific percutaneous device to be used in the procedure. Specifically, although the embodiments of the robotic procedure system 10 described herein are primarily explained with respect to the diagnosis and / or treatment of coronary artery disease, the robotic procedure system 10 can be used to diagnose and / or treat any type of disease or condition that can be diagnosed and / or treated via a catheter-based procedure. For example, the robotic procedure system 10 can be used to treat hypertension by utilizing a radiofrequency emitting catheter to inactivate certain nerves that weaken the kidneys to control high blood pressure.

[0057] The robotic procedure system 10 includes a laboratory unit 11 and a workstation 14. The robotic procedure system 10 includes a robotic catheter system, shown as a bedside system 12, located within the laboratory unit 11 near a patient 21. Typically, the bedside system 12 can be equipped with appropriate percutaneous devices (e.g., guidewires, guide catheters, working catheters, catheter balloons, stents, diagnostic catheters, etc.) or other components (e.g., contrast agents, medications, etc.) to allow a user to perform a catheter-based medical procedure. The robotic catheter system (e.g., bedside system 12) can be any system configured to allow a user to perform a catheter-based medical procedure via a bedside system by operating various controllers (e.g., controllers located at the workstation 14). The bedside system 12 can include any number and / or combination of components to provide the functionality described herein to the bedside system 12.

[0058] Various embodiments of the bedside system 12 are described in detail in PCT International Application No. PCT / US2021 / 070042 filed on November 14, 2021, PCT International Application No. PCT / US2020 / 041964 filed on July 7, 2020, and PCT International Application No. PCT / US2009 / 042720 filed on May 4, 2009, and the entire contents of each of these applications are incorporated herein by reference.

[0059] In one embodiment, the bedside system 12 can be equipped to perform catheter-based diagnostic procedures. In this embodiment, the bedside system 12 can be equipped with one or more catheters for delivering contrast media to the coronary arteries. In one embodiment, the bedside system 12 can be equipped with a first catheter, a second catheter, and a third catheter, wherein the first catheter is configured to deliver contrast media to the coronary arteries on the left side of the heart, the second catheter is configured to deliver contrast media to the coronary arteries on the right side of the heart, and the third catheter is configured to deliver contrast media into the ventricles.

[0060] In another embodiment, the bedside system 12 can be equipped to perform catheter-based treatment procedures. In this embodiment, the bedside system 12 can be equipped with a guide catheter, a guidewire, and a working catheter (e.g., a balloon catheter, a stent delivery catheter, an ablation catheter, a microcatheter, etc.). In one embodiment, the bedside system 12 can be equipped with a working catheter that includes a second lumen through which a guidewire is passed during the procedure. In another embodiment, the bedside system 12 can be equipped with an over-the-wire working catheter that includes a central lumen through which a guidewire is passed during the procedure. In another embodiment, the bedside system 12 can be equipped with an intravascular ultrasound (IVUS) catheter. In another embodiment, any percutaneous device of the bedside system 12 can be equipped with a position sensor that indicates the location of the component within the body.

[0061] The bedside system 12 communicates with the workstation 14, allowing signals input by the user and / or generated by the control system of the workstation 14 to be transmitted to the bedside system 12 to control various functions of the bedside system 12. The bedside system 12 can also provide feedback signals (e.g., operating conditions, warning signals, error codes, etc.) to the workstation 14. The bedside system 12 can be connected to the workstation 14 via a communication link 38, which can be a wireless connection, a cable connector, or any other means capable of allowing communication between the workstation 14 and the bedside system 12.

[0062] The workstation 14 includes a user interface 30. The user interface 30 includes a controller 16. The controller 16 allows a user to control the bedside system 12 to perform a catheter-based medical procedure. For example, the controller 16 can be configured to cause the bedside system 12 to perform various tasks (e.g., advancing, retracting, or rotating a guidewire, advancing, retracting, or rotating a working catheter, advancing, retracting, or rotating a guide catheter, inflating or deflating a balloon located on a catheter, positioning and / or deploying a stent, injecting a contrast agent into a catheter, injecting a drug into a catheter, or performing any other function that can be performed as part of a catheter-based medical procedure, etc.) using various percutaneous devices that the bedside system 12 can be equipped with. In some embodiments, one or more of the percutaneous interventional devices can be steerable, and the controller 16 can be configured to allow the user to steer one or more steerable percutaneous devices. In one such embodiment, the bedside system 12 can be equipped with a steerable guide catheter, and the controller 16 can also be configured to allow a user located at the remote workstation 14 to control the curvature of the distal end of the steerable guide catheter. A further embodiment of a catheter system 10 including a steerable guide catheter is disclosed in U.S. Patent No. 9,452,277, the entire contents of which are incorporated herein by reference. Another example embodiment of a control system is shown in USSN 17 / 812,733, filed July 15, 2022, which is incorporated herein by reference.

[0063] In one embodiment, controller 16 comprises touch screen 18, special guide catheter controller 29, special guide wire controller 23 and special working catheter controller 25.In this embodiment, guide wire controller 23 is the joystick configured to advance, retract or rotate guide wire, working catheter controller 25 is the joystick configured to advance, retract or rotate working catheter, and guide catheter controller 29 is the joystick configured to advance, retract or rotate guide catheter.In addition, touch screen 18 can display one or more icons (such as icons 162,164 and 166), and these icons control the movement of one or more percutaneous devices via bedside system 12, or receive various inputs from the user, as discussed below.Controller 16 can also comprise sacculus or stent controller, and it is configured to inflate or deflate sacculus and / or stent.Each controller can comprise one or more buttons, joystick, touch screen etc., and it may be that the specific component that controls this controller is dedicated to is desired. As discussed in more detail below, the robotic programming system 10 includes a percutaneous device movement algorithm module or movement instruction module 114 that instructs the bedside system 12 how to respond to user manipulation of the controller 16 to cause the percutaneous device to move in a particular manner.

[0064] The controller 16 may include an emergency stop button 31 and a multiplier button 33. When the emergency stop button 31 is pressed, a relay is triggered to cut off power to the bedside system 12. The multiplier button 33 acts to increase or decrease the speed at which the associated components move in response to manipulation of the guide catheter controller 29, the guidewire controller 23, and the working catheter controller 25. For example, if operation of the guidewire controller 23 advances the guidewire at a rate of 1 mm / second, pressing the multiplier button 33 may cause operation of the guidewire controller 23 to advance the guidewire at a rate of 2 mm / second. The multiplier button 33 may be a switch that allows the multiplier effect to be turned on and off. In another embodiment, the user must press and hold the multiplier button 33 down to increase the speed of the components during operation of the controller 16.

[0065] The user interface 30 may include a first monitor 26 and a second monitor 28. The first monitor 26 and the second monitor 28 may be configured to display information or patient-specific data to a user at the workstation 14. For example, the first monitor 26 and the second monitor 28 may be configured to display image data (e.g., x-ray images, MRI images, CT images, ultrasound images, etc.), hemodynamic data (e.g., blood pressure, heart rate, etc.), and patient record information (e.g., medical history, age, weight, etc.). In addition, the first monitor 26 and the second monitor 28 may be configured to display procedure-specific information (e.g., duration of the procedure, catheter or guidewire position, volume of drug or contrast agent delivered, etc.). In one embodiment, a user may use a user input device or controller (e.g., a mouse) to interact with or select various icons or information displayed on the monitors 26 and 28. The monitors 26 and 28 may be configured to display information regarding the position and / or curvature of the distal end of the steerable guide catheter. In addition, the monitors 26 and 28 may be configured to display information to provide functionality associated with the various modules of the controller 40 discussed below. In another embodiment, the user interface 30 includes a single screen that is large enough to display one or more of the display components and / or touch screen components discussed herein.

[0066] The robotic procedure system 10 also includes an imaging system 32 located within the laboratory unit 11. The imaging system 32 can be any medical imaging system that can be used in conjunction with catheter-based medical procedures (e.g., non-digital x-ray, digital x-ray, CT, MRI, ultrasound, etc.). In the exemplary embodiment, the imaging system 32 is a digital x-ray imaging device that communicates with the workstation 14. Figure 1 As shown, the imaging system 32 may include a C-arm that allows the imaging system 32 to be partially or completely rotated around the patient 21 to obtain images at different angular positions relative to the patient 21 (e.g., sagittal view, caudal view, craniocaudal view, etc.).

[0067] The imaging system 32 is configured to capture x-ray images of appropriate areas of the patient 21 during a particular procedure. For example, the imaging system 32 can be configured to capture one or more x-ray images of the heart to diagnose a cardiac condition. The imaging system 32 can also be configured to capture one or more x-ray images (e.g., real-time images) during a catheter-based medical procedure to assist a user of the workstation 14 in correctly positioning a guidewire, a guide catheter, a working catheter, a stent, etc. during the procedure. The one or more images can be displayed on the first monitor 26 and / or the second monitor 28.

[0068] In addition, the user of workstation 14 may be able to control the angular position of imaging system 32 relative to the patient to obtain various views of the patient's heart and display them on first monitor 26 and / or second monitor 28. Displaying different views at different parts of the procedure can help the user of workstation 14 appropriately move and position the percutaneous device within the 3D geometry of the patient's heart. For example, displaying the correct view during the procedure can allow the user to view the patient's vasculature from the correct angle to ensure that the distal end of the steerable guide catheter is bent in the correct manner, thereby ensuring that the catheter moves as expected. In addition, displaying different views at different parts of the procedure can help the user select the appropriate instruction set for the movement instruction module 114 discussed below. In an exemplary embodiment, imaging system 32 can be any 3D imaging modality of the past, present, or future, such as an X-ray-based computed tomography (CT) imaging device, a magnetic resonance imaging device, a 3D ultrasound imaging device, etc. In this embodiment, the image of the patient's heart displayed during the procedure can be a 3D image. In addition, controller 16 can also be configured to allow the user at workstation 14 to control various functions of imaging system 32 (e.g., image capture, magnification, collimation, C-arm positioning, etc.).

[0069] refer to Figure 2 , a block diagram of a robot programming system 10 is shown according to an exemplary embodiment. The robot programming system 10 may include a control system, shown as a controller 40. Although a controller 40 is described, it should be understood that the exemplary embodiments are not limited thereto. For example, processing or control circuitry may be used, such as, but not limited to, one or more processors, one or more central processing units (CPUs), one or more controllers, one or more arithmetic logic units (ALUs), one or more digital signal processors (DSPs), one or more microcomputers, one or more field programmable gate arrays (FPGAs), one or more systems on a chip (SOCs), one or more programmable logic units (PLUs), one or more microprocessors, one or more application specific integrated circuits (ASICs), or one or more other devices capable of responding to and executing instructions in a defined manner.

[0070] like Figure 2 As shown in FIG, a controller 40 may be part of the workstation 14. The controller 40 communicates with one or more bedside systems 12, the controller 16, the monitors 26 and 28, the imaging system 32, and patient sensors 35 (e.g., an electrocardiogram ("ECG") device, an electroencephalogram ("EEG") device, a blood pressure monitor, a temperature monitor, a heart rate monitor, a respiratory monitor, etc.). In addition, the controller 40 may communicate with a hospital data management system or hospital network 34, one or more additional output devices 36 (e.g., a printer, a disk drive, a CD / DVD burner, etc.), and a hospital inventory management system 37.

[0071] Communication between the various components of the robotic procedure system 10 can be achieved via a communication link 38. The communication link 38 can be a dedicated wired or wireless connection. The communication link 38 can also represent communication over a network. The robotic procedure system 10 can be connected or configured to include any other systems and / or devices not explicitly shown. For example, the robotic procedure system 10 can include an IVUS system, an image processing engine, a data storage and archiving system, an automated balloon and / or stent inflation system, a contrast agent and / or drug injection system, a drug tracking and / or recording system, a user log, an encryption system, a system for restricting access to or use of the robotic procedure system 10, past, present, or future robotic catheter systems, and the like. Further embodiments of the robotic procedure system 10 including an inflation and / or contrast agent injection system are disclosed in U.S. Patent No. 9,545,497, issued on January 17, 2017, which is incorporated herein by reference in its entirety.

[0072] refer to Figure 3, a block diagram of an embodiment of a robotic programming system 10 is shown, according to an exemplary embodiment. The robotic programming system 10 can include various actuation mechanisms that move an associated percutaneous device in response to user manipulation of the controller 16. In the illustrated embodiment, the robotic programming system 10 includes a guidewire actuation mechanism 50, a working catheter actuation mechanism 52, and a guide catheter actuation mechanism 54. In other embodiments, the robotic programming system 10 can include an actuation mechanism for inflating an angioplasty or stent delivery balloon and an actuation mechanism for delivering a contrast agent. In the illustrated embodiment, the guidewire actuation mechanism 50 and the working catheter actuation mechanism 52 are combined within a box 56, which is coupled to a base of the bedside system 12. Additional embodiments of the bedside system 12 and box 56 are described in detail in PCT International Application No. PCT / US2021 / 070042 filed on November 14, 2021, PCT International Application No. PCT / US2020 / 041964 filed on July 7, 2020, PCT International Application No. PCT / US2020 / 041923 filed on July 14, 2020, and U.S. Patent No. 8,480,618 issued on July 9, 2013, the entire contents of each of which are incorporated herein by reference. Additional embodiments of the robot program system 10 are described in detail in PCT International Application No. PCT / US2021 / 070042 filed on November 14, 2021, PCT International Application No. PCT / US2020 / 041964 filed on July 7, 2020, PCT International Application No. PCT / US2020 / 041923 filed on July 14, 2020, U.S. Patent No. 8,790,297 issued on July 29, 2014, and U.S. Patent No. 9,545,497 issued on January 17, 2017, the entire contents of each of which are incorporated herein by reference.

[0073] The guidewire actuation mechanism 50 is coupled to the guidewire 58 so that the guidewire actuation mechanism 50 can advance, retract and rotate the guidewire 58. The working catheter actuation mechanism 52 is coupled to the working catheter 60 so that the working catheter actuation mechanism 52 can advance, retract and rotate the working catheter 60. The connector 62 couples the guide catheter 64 to the guide catheter actuation mechanism 54 so that the guide catheter actuation mechanism 54 can advance, retract and rotate the guide catheter 64. In various embodiments, the guidewire actuation mechanism 50, the working catheter actuation mechanism 52 and the guide catheter actuation mechanism can each include an engagement structure (e.g., a pair or more pairs of pinch wheels) that is suitable for engaging a corresponding percutaneous device so that the actuation mechanism can apply axial and / or rotational movement to the percutaneous device.

[0074] The Y-connector 66 is coupled to the guide catheter actuation mechanism 54 via a connector 68. In various embodiments, the connector 68 can be a separate component from both the Y-connector 66 and the guide catheter actuation mechanism 54. In other embodiments, the connector 68 can be part of (e.g., integral with) the Y-connector 66 or part of the actuation mechanism 54. In the embodiment shown, the Y-connector 66 is also connected to the cassette 56.

[0075] In one embodiment, the Y-shaped connector 66 includes a first branch, a second branch, and a third branch. The first branch of the Y-shaped connector is connected to the inner cavity of the guide catheter 64 or is in communication with it. The second branch deviates from the longitudinal axis of the guide catheter 64. The second branch provides a port for injecting fluid (e.g., contrast agent, drug, etc.) into the inner cavity of the guide catheter 64. The third branch of the Y-shaped connector 66 is coupled to the box 56 and receives both the guide wire 58 and the working catheter 60. Therefore, through this arrangement, the guide wire 58 and the working catheter 60 are inserted into the inner cavity of the guide catheter 64 through the Y-shaped connector 66.

[0076] Guidewire actuation mechanism 50 comprises rotary actuator 70 and propulsion / retraction actuator 72.Rotation actuator 70 is configured to rotate guidewire 58 around its longitudinal axis.Guidewire actuation mechanism 50 can also comprise one or more sensors 71, for measuring the speed, torque, position and other parameters of guidewire / actuator.Propel / retraction actuator 72 is configured to advance and / or retract guidewire 58 (that is, advance and / or retract along the longitudinal axis of guidewire) in patient 21 body.Working conduit actuation mechanism 52 comprises rotary actuator 74 and propulsion / retraction actuator 76.Rotation actuator 74 is configured to rotate working conduit 60 around its longitudinal axis.Working conduit 60 can also comprise one or more sensors 71, for measuring the speed, torque, position and other parameters of guidewire / actuator.Propel / retraction actuator 76 is configured to advance and / or retract working conduit 60 (that is, advance and / or retract along the longitudinal axis of working conduit) in patient 21 body. The guide catheter actuating mechanism 54 includes a rotary actuator 78, an advancement / retraction actuator 80 and a bending actuator 82. The guide catheter actuating mechanism 54 can also include one or more sensors 71 for measuring the speed, torque, position and other parameters of the guide wire / actuator. The rotary actuator 78 is configured to rotate the guide catheter 64 around its longitudinal axis. The advancement / retraction actuator 80 is configured to advance and / or retract the guide catheter 64 (i.e., advance and / or retract along the longitudinal axis of the guide catheter) in the patient 21 body. In certain embodiments, the guide catheter 64 may include one or more bending control elements that allow the user to bend the distal end of the guide catheter 64. In such an embodiment, the bending actuator 82 bends the distal end of the guide catheter 64 in response to the user's manipulation of the controller 16.

[0077] like Figure 3 , a controller 16 and a controller 40 located at the workstation 14 are communicatively coupled to various portions of the bedside system 12 to allow a user and / or control system to control the movement of the guidewire 58, the working catheter 60, and the guide catheter 64, as well as any other percutaneous devices equipped with the bedside system 12. In the illustrated embodiment, the controller 16 and the controller 40 are coupled to the guide catheter actuation mechanism 54 to allow the user to move the guide catheter 64. Additionally, the controller 16 and the controller 40 are coupled to the cassette 56 to allow the user to control the guidewire 58 via the guidewire actuation mechanism 50 and the working catheter 60 via the working catheter actuation mechanism 52. Control signals 116 generated by the control devices and controllers at the workstation 14 are transmitted to the bedside system 12 to control the movement of the percutaneous devices discussed herein.

[0078] Even with these robotic devices, clinicians rely on limited indirect observations from X-ray fluoroscopy (i.e., intermittent updates due to radiation limitations and a 2D partial view of the device) to passively control the guidewire / catheter. The observed device state from X-ray fluoroscopy shows a limited view and is unclear for estimating accurate state information, resulting in increased total execution time and not long-term predictions for guiding human users.

[0079] Modeling and controlling guidewire manipulation in coronary vessels remains challenging because the complex interaction between the guidewire motion with different physical properties (i.e., loading, coating) and the vessel geometry with lumen conditions results in a highly nonlinear system.

[0080] Routing blood vessels is a challenging problem. In particular, there are multiple devices (e.g., microcatheters, guidewires, catheters, etc.) used for nerve catheterization, which interact with each other in the blood vessels. Controlling each device by twisting, pushing, and pulling it back until it reaches the desired position in the blood vessel is highly dependent on the clinician's experience. During this period, X-ray images are repeatedly acquired to visualize the desired catheter path and verify when the catheter enters the desired blood vessel, exposing the clinician to excessive radiation.

[0081] Current robotic endovascular procedures have the following limitations: decision making and safety assessments for robotic control are missing, which previously relied heavily on user experience.

[0082] Example embodiments provide (1) a scalable learning pipeline for training AI predictive control systems, providing safety features for long-term control; and (2) long-term integrated predictive control of multiple devices for robotic neural catheterization.

[0083] Therefore, predictive control according to one or more example embodiments reduces clinician fatigue and reduces the total execution time proportional to the X-ray exposure time.

[0084] Figure 4A A predictive control system according to one or more example embodiments is shown.

[0085] The system includes an offline phase 405 and an online implementation phase 450 .

[0086] The offline phase 405 can be considered as pre-processing and training, while the online phase 450 can be considered as the real-time processing phase.

[0087] As shown, offline phase 405 includes data lake 410 , pre-processing module 415 , engine simulator 420 , and AI predictive control system 460 .

[0088] Data lake 410 utilizes available CTA images and physical parameters of the devices. In some exemplary embodiments, data lake 410 may store physical parameters such as tip length, shape, and material used for various types of guidewires and other devices. Furthermore, data lake 410 may store patient data. For example, data lake 410 may store two-plane 3D CTA brain and / or heart images.

[0089] The pre-processing module 415 is configured to perform 3D vessel segmentation and extract centerlines with cross-sectional information (e.g., Figure 6B For example, the type and image of the image can be adjusted so that the simulator 420 can determine a set of movements of the guide wire, such as the reference image. Figure 4B as described (e.g., 3D vessel segmentation, centerline determination, and distance map generation).

[0090] The pre-processing module 415, the engine simulator 420, the AI ​​predictive control system 460, and the image processing module 480 and their functions may be implemented by the controller 40. When the controller 40 is a processor, the processor may execute instructions stored in the memory 205 to perform the functions of the pre-processing module 415, the engine simulator 420, the AI ​​predictive control system 460, and the image processing module 480.

[0091] Figure 4B shows a flow chart for generating a physics-based simulation of a guidewire, and Figures 5A-5D An image illustrating a physics-based simulation for generating a guidewire according to one or more example embodiments.

[0092] In some example embodiments, at 415a, the pre-processing module 415 uses the patient's 3D CTA images (e.g., Figure 5A ) and generate a corresponding 3D segmentation of the entire coronary vessel (e.g., in Figure 5B In some example embodiments, segmentation can be performed using a 3D U-Net, as described in Tsang, “Review: 3D U-Net–Volumetric Segmentation (Medical Image Segmentation)”, Towards Data Science, April 2, 2019, the entire contents of which are incorporated herein by reference. At 415b, the pre-processing module 415 adds the insertion support of the guidewire (e.g., introducer / sheath) as a boundary tube, which is refined by a Poisson reconstruction filter to smooth irregularities in the reconstructed surface, making it more accurate and visually coherent within the entire 3D segmented scene.

[0093] Then, at 415c, the pre-processing module 415 calculates a 3D level set distance map containing the minimum distance to the vessel at each 3D point to define the boundary conditions (e.g., Figure 5C In some example embodiments, the distance map may be generated according to Guo et al., “DeepCenterline: a Multi-task full Convolutional Network forcentline Extraction,” arXiv:1903:10481v1, Mar. 25, 2019, which is incorporated herein by reference in its entirety.

[0094] The pre-processing module 415 is further configured to generate a stenosis grade for angiography (i.e., for fully automated coronary artery evaluation (FACE)), which can be used as the variable vessel lumen and friction information for the engine simulator 420. For example, the pre-processing module 415 can generate a stenosis grade as described in Avram et al., “CathAI: fully automated coronaryangiography interpretation and stenosis estimation,” npj Digital Medicine 6, Article No. 142 (2023), the entire contents of which are incorporated herein by reference.

[0095] The engine simulator 420 uses information from the pre-processing module 415 (e.g., 3D vessel geometry, centerline with cross-sectional information, stenosis classification with friction information, device properties) to create a physics-based simulation environment. The engine simulator 420 is configured to simulate the movement of the guidewire in the vessel geometry in real time and can generate synthetic x-ray images based on the simulated guidewire movement.

[0096] Return Reference Figure 4B In an example embodiment, at 420a, the engine simulator defines the initial position and orientation of the guidewire. The guidewire can be simulated using a Cosserat-rod model. The engine simulator 420 applies a desired force to the proximal segment of the guidewire (at 420a) using the Cosserat-rod model, which interacts with the defined boundary conditions by solving partial differential equations. The Cosserat-rod model allows for all possible degrees of freedom of deformation: bending, twisting, stretching, and shearing. For example, Figure 5D A guide wire 505 simulated by a Cosserat-rod is shown. Figure 5D As shown in FIG, Cosserat-rod simulates the guide wire 505 into n segments 5051-505 n .

[0097] The guidewire is modeled as a deformable curve with an additional deformable vector to characterize its orientation. The orientation follows a system of coupled partial differential equations to maintain both linear and angular momentum. In some example embodiments, a neural ordinary differential equation (ODE) method is used to efficiently solve the partial differential equations to achieve real-time simulation because the time integration using neural ODEs is more accurate than using traditional time integrators. The inventors have found that neural ODEs help solve the momentum conservation equation from Cosseratrod theory for guidewire insertion.

[0098] Figure 4C A flow chart is shown for generating a new mesh having a desired vessel-like structure to restrict movement of a portion of a guidewire according to one or more example embodiments.

[0099] At S490, the simulator 420 obtains the 3D segmentation of the patient's vascular structure (e.g., from pre-processing). The simulator 420 uses boundary conditions to keep the guidewire within the blood vessel while limiting the base of the guidewire when the guidewire is inserted. Therefore, new boundary conditions are developed. At S492, the simulator 420 adds a vascular structure of desired width to the entry point of the large blood vessel. Then, at S494, the simulator 420 cuts the mesh and finally applies a Poisson surface reconstruction filter to merge them into a waterproof surface, as shown in S496.

[0100] To confine the catheter within the generated watertight surface, at each time step of the simulation, the simulator 420 calculates the wall response of the blood vessel applied to the rod (guidewire) as it approaches the lumen wall, i.e.

[0101] ε=rd>0 (1)

[0102] Where d is the distance from the centerline of the guidewire to the closest point of the lumen, and r is the radius of the guidewire. The wall response force is then given by:

[0103]

[0104] where k w is the wall stiffness, v w is the dissipation coefficient, F ⊥ is the sum of the forces of all the rods against the wall, k w ε is the elastic term in the normal velocity, and is its dissipative term, where v is the applied velocity, and is the normal velocity perpendicular to the vessel wall.

[0105] Stiffness k w and dissipation coefficient v w Controls the effect of the wall boundary condition.

[0106] To define this boundary condition, for each discrete point of the rod in the cavity, the minimum distance to the cavity surface and the normal direction can be calculated. However, this operation is very time-consuming and highly dependent on the number of points (size) of the cavity.

[0107] In some example embodiments, the simulator 420 uses novel boundary conditions where an impermeable surface is used to compute a 3D image on a bounding box containing the lumen mesh, which contains a level set map defined at each voxel, as in Figure 4D Seen in.

[0108] exist Figure 4D The level set map shown on the left of is computed as signed distances, so its gradient is orthogonal to the level set contours. Signed distances are computed exactly in the neighborhood of the surface and can be extrapolated to greater distances for computational efficiency. Given any 3D point within the lumen of a large vessel, the simulator 420 retrieves an approximation of its shortest distance to the lumen surface simply by evaluating the 3D image of the level set map at that point, which is computationally efficient. As mentioned above, the normal direction is also computed. Due to the definition of the level set map, this becomes straightforward by evaluating the gradient of the image at the desired point, as Figure 4D as shown on the right side of .

[0109] Furthermore, only one boundary condition is required, which avoids having to iterate over different boundary conditions defined at each point on the luminal surface. This allows the simulation to be computationally efficient. That is, for an endoluminal mesh with 17,000 points, the application of this new boundary condition means an acceleration factor of 3,000, allowing the simulation to proceed in real time. In real life, blood vessels are constantly moving, making the guidance of catheters more complex. In an example embodiment, the simulator 420 uses a level set method to move the vessel boundaries and therefore has a continuously updated distance map that follows the movement of the vessel, which can be accomplished using a boundary node method for cell motility as described in U.S. Patent No. 10,282,638, the entire contents of which are incorporated herein by reference.

[0110] This provides a scalable simulation of the device state inside the blood vessel (e.g., Figure 5D More specifically, the engine simulator 420 includes a set of device motions that can be applied to each created environment. The simulated device can be composed of multiple segment points (for example, a guidewire is defined as 300 segment points to be modeled using the Cosserat-rod model). Each segment state can be defined as a 3D position, velocity, and applied force. The accumulated information of the segment states indicates the device state.

[0111] The engine simulator 420 uses the device motion and associated environment as training data to create various states of the bedside system 12 .

[0112] In addition, the simulator 420 can incorporate the motion of the guidewire into a digital reconstruction radiography (DRR) algorithm to create a synthetic X-ray image in real time. It should be understood that during the online phase, the AI ​​predictive control system 460 and / or the image processing module 480 can also generate a synthetic image in the same or similar manner.

[0113] The motion of the guidewire at each output time step can be incorporated into the 3D CTA information by locating the voxels closest to the guidewire point and assigning them maximum intensity. Simulator 420 can generate a rendered image with the guidewire point assigned the maximum intensity. Once this information is incorporated, simulator 420 uses a DDR algorithm to simulate the C-arm and project this information onto a 2D image to generate a synthetic X-ray image. Simulator 420 can use the DDR algorithm described in U.S. Patent No. 10,282,638.

[0114] Finally, the output image is optimized using neural style transfer, e.g., by taking a set of ground truth images which can be from fluoroscopy or angiography images of the patient, which defines a style reference image by blending it with the content image from DRR.

[0115] The pre-processing module 415 and the engine simulator 420 provide an extensible environment and associated various device controls in simulation, which allows for a reduction in the amount of CTA data and / or images of the patient during actual device control.

[0116] The online phase 450 includes a trained AI predictive control system 460, a bedside system 12, and an image processing module 480. The AI ​​predictive control system 460 and the image processing module 480 may be executed by the controller 40 or other hardware, such as a server in the cloud.

[0117] The engine simulator 420 includes a set of device motions that can be applied to each created environment.

[0118] Figure 6A A flow chart for training the AI ​​predictive control system 460 is shown. As mentioned above, various states of the bedside system 12 using device motion and the associated environment are used as training data. The AI ​​predictive control system 460 can be trained offline based on data collected from the pre-processing module 415 and the physics engine simulator 420. Patient-specific data can also be applied to the engine simulator 420 to validate and strengthen the AI ​​predictive control system 460. The AI ​​predictive control system 460 can be implemented as a recursive neural network, a long short-term memory (LSTM) network, a transformer architecture, a neural ordinary differential equation (ODE), other known deep learning architectures, combinations thereof, or sub-combinations thereof. In some example embodiments, the AI ​​predictive control system 460 can be implemented by the controller 40. In other example embodiments, the AI ​​predictive control can be implemented by a processing circuit located away from the bedside system 12 (e.g., in the cloud).

[0119] like Figure 6A As shown, the user and / or simulator inputs motion commands π to the robotic surgery system 12 and the AI ​​predictive control system 460 within time t. t More specifically, the motion command π t is a command for the guidewire actuator 50 to advance / retract (translation motion) and rotate the guidewire, and therefore, may also be referred to as a motor command or actuator command. In some example embodiments, the motion command π t At least one of a speed or torque command of the guidewire may be included.

[0120] The simulator 420 maintains a true internal state x(t), which can be obtained from executing the motion command π tThe 3D tip position of the guidewire at time t is calculated. More specifically, the AI ​​predictive control system 460 is configured to predict a long-term state estimate based on a trajectory, which is a history of continuous states. In some example embodiments, the AI ​​predictive control system 460 uses a recurrent neural network (e.g., RNN, LSTM, transformer, N-ODE, etc.) to model the long-term state estimator.

[0121] The data generated from the simulator 420 is used to train the AI ​​predictive control system 460. In some example embodiments, the input data from the simulator 420 can be the 3D tip position, centerline, vessel friction, device properties, and distance traveled at time t, as well as the ground truth of the tip position (i.e., the true internal state x(t)). The AI ​​predictive control system 460 is trained based on the motion instructions π t , based on our desired tip motion, we determine the motion command π*.

[0122] The simulator 420 can determine the observed measured 3d position y(t) of the guidewire tip. The simulator 420 knows the tip position / applied torque (motion), which creates a mapping function between 3d tip position and motion.

[0123] In some example embodiments, the observed measured position y(t) may also be determined by the image processing module 480 using the CTA image captured at time t

[0124] The true internal state may guide realistic observations of the wire tip position, which may be observed using known sensors and / or models (eg, from simulator 420).

[0125] The AI ​​predictive control system 460 maintains an estimated internal state The estimated internal state can be obtained from executing the motion command π t The AI ​​predictive control system 460 outputs the estimated 3D position of the guidewire tip at the starting time t. Estimated 3D position of the guidewire tip in a blood vessel Can be displayed or overlaid on 3D vessel segmentation or 2D X-ray.

[0126] At 605, the training system determines the observed measured position y(t) and estimated 3D position of the guidewire tip The error / loss Δy between the two. The loss Δy is fed back to the AI ​​predictive control system 460. As shown, Figure 6A The AI ​​neural network training process between the ground truth (obtained from the simulator 420) and the AI ​​predictive control system 460 (ie, the target model) is shown. Thus, based on all input conditions, the AI ​​predictive control system 460 is trained to be close to the ground truth.

[0127] In some example embodiments, the AI ​​predictive control system 460 is trained using mean squared error (MSE) based on the error / loss Δy.

[0128] The AI ​​predictive control system 460 can be generated by applying a 3D past tip position sequence with the vessel centerline, physical parameters, and travel distance, and then creating a future estimated state sequence.

[0129] In some example embodiments, 100 CTA images of the right coronary artery are used as the simulator environment, and π prox 1000 randomly generated sequential controls were applied for 60 seconds, generating 120 sequence sample points for each set of sequential controls (one of the randomly generated sequential controls) in each segmented vessel (one of the 100 CTA images). 80% of the data was used as a training set, and 20% as a test set. We then presented a safe ratio and difference between the estimated forces and the ground truth in the test set.

[0130] Figure 6B A 3D blood vessel geometry 610 with a centerline 615 is shown.

[0131] Figure 7A A flow chart illustrating an implementation of an AI predictive control system according to one or more example embodiments is shown.

[0132] As shown, the AI ​​predictive control system 460 includes a reverse agent 705 and a forward agent 710. The AI ​​predictive control system 460 and the image processing module 480 can be executed by the controller 40 or other hardware, such as a server in the cloud. In some example embodiments, the reverse agent 705 and the forward agent 710 are trained as a dual transformer model, such as described in Vaswani et al., "Attention is all you need," Advance in Neural Information Processing Systems, Vol. 30, 2017, the entire contents of which are incorporated herein by reference.

[0133] More specifically, the reverse proxy 705 generates an estimate of the guidewire tip's state based on the command position CMD and the forward proxy 710. The difference between the two generates the guidewire motion command

[0134] In some example embodiments, the forward agent 710 and the reverse agent 705 (i.e., forward / reverse transformer models) are learned using 30 state sequences as input, 1 output (many-to-one), 12 heads, 4 encoder / decoder layers, 128 feedforward dimensions, and 0.1 dropout.

[0135] The reverse proxy 705 can generate movement commands for the guidewire (ie, control of the guidewire while attached to the robot, such as translation and rotation).

[0136] More specifically, x(t) is the 3D tip position at time t, q(t) is the motion state applied at time t, e is the vessel geometry (i.e., the lumen geometry from the level set distance map generated by the preprocessing module 415), and a is a discrete action at the proximal end (i.e., the behavior library).

[0137] The current position x of the tip of a given guidewire tip and the desired location to go, the AI ​​predictive control system 460 creates a motion sequence that minimizes the target cost (e.g., execution time, precise motion) under constraints (e.g., no collision with a blood vessel and no puncture).

[0138] Thus, the AI ​​predictive control system 460 uses reinforcement learning (RL) and includes a forward agent 710 for forward state estimation, which estimates the next state x(t+1) (e.g., the position of the guidewire tip), and a reverse agent 705 for reverse state estimation, which estimates the next action a(t+1) (e.g., at least one of torque or velocity).

[0139] At the guidewire's point of attachment to the bedside system 12 (eg, where the torquer is attached to the bedside system 12), the guidewire's proximal end position is

[0140] x prox ≡<Δ,φ>, (3)

[0141] where Δ is the translation and φ is the rotation.

[0142] At the distal end (tip) of the guidewire, the distal state of the guidewire x tip It can be defined by its position and orientation in the SE(3) transform, which is a 3D homogeneous transformation matrix consisting of a translation and a rotation.

[0143] The physical parameters p of the guidewire may include diameter, Poisson's ratio, Young's modulus, and number of elements. However, example embodiments are not limited thereto. Let e∈E denote the spatial environment of a blood vessel, including 3D blood vessel geometry, 3D blood vessel centerline, and cross-sectional markers.

[0144] Since the AI ​​predictive control system 460 does not have a direct measurement of the guidewire tip, the AI ​​predictive control system 460 simplifies the 3D vessel centerline by projecting the 3D vessel centerline onto a known 3D vessel centerline (e.g., from the pre-processing module 415).

[0145] therefore

[0146] x tip ≈x^ tip =<Δ tip ,γ tip > (4)

[0147] where Δ tip and γ tip represent the translation along the centerline and the distance from the centerline to the tip, respectively.

[0148] The forward proxy 710 implements the forward transfer function FF, and the reverse proxy 705 implements the reverse transfer function IF as follows:

[0149]

[0150] Given the parameter p and the spatial environment e of the blood vessel, control π prox ∈Π transitions from one state to another. Proximal state (e.g., position before exercise), The position after movement is known from measurements of the guidewire actuator mechanism 50 (e.g., from the motor encoder), and the proximal command and is also known from the measured values ​​of the actuator mechanism 50 .

[0151] Given the desired tip state x according to command CMD tip , and based on the current tip position (eg, observed tip position), the reverse proxy 705 generates a proximal command sequence for the surgical system 12 Allows the surgical system to steer the guidewire to the desired tip state It should be noted that the reverse proxy 705 also generates an estimated proximity location (rather than measuring the position ) to transmit the correct commands to the bedside system 12.

[0152] The reverse proxy 705 generates the proximal command sequence by minimizing the cost C as follows

[0153] Where t is the current time and n is the long-term time step. More specifically, the AI ​​control model 460 can be trained using mean squared error (MSE) so that the AI ​​control model 460 provides a sequence of actions that minimizes cost (eg, error).

[0154] Therefore, the AI ​​predictive control system 460 is configured to estimate the state information sequence (ie, and ).

[0155] The image processing module 480 is configured to provide intermittent observations and provide information on the estimated state and For example, the image processing module 480 (or the AI ​​predictive control system 460) can generate an attention-based force map for the guidewire (or another percutaneous device) for display on one of the monitors 26 and 28. The attention-based force map can be generated by the image processing module 480 (or the AI ​​predictive control system 460) according to Li et al., “Efficient Self-supervised Vision Transformers for Representation Learning,” ICLR 2022, arXiv:2106.09785v2, July 6, 2022, which is incorporated herein by reference in its entirety.

[0156] If the AI ​​predictive control system 460 estimates a violation of the constraint (e.g., the likelihood of the guidewire contacting the vessel wall exceeds a threshold), the image processing module 480 provides observations / measurements to the AI ​​predictive control system 460. For example, the AI ​​predictive control system 460 can calculate an uncertainty value based on the error between the centerline and the vessel wall, so that the AI ​​predictive control system 460 can determine whether the tip is in danger (i.e., too close to the wall in a vertical direction). If it is determined that the tip is close to the vessel wall, the AI ​​predictive control system 460 can issue a notification to the user and / or the image processing module 480 to perform an observation based on the X-ray.

[0157] In some example embodiments, the image processing module 480 uses a dynamic roadmap (DRM) and FACE stenosis grading with respect to 2D device tracking information.

[0158] The image processing module 480 registers the 2D tracking information to the 3D vascular lumen based on the registration with the original CTA image to generate the actual measurement value of the guidewire y(t). The measurement value y(t) can then be applied to the AI ​​predictive control system 460 (to generate Δy) and reduce possible errors.

[0159] In addition, the image processing module 480 generates an image 708 that can reflect the estimated position of the guidewire and / or the measured position of the guidewire, thereby allowing the user to monitor the program based on the uncertainty measurement of the AI ​​predictive control. The image 708 can also be used to make autonomous decisions. For example, given the parameters p of the wire and the spatial environment e of the blood vessel, the AI ​​predictive control system 460 can provide commands to steer the wire across the stenosis, perform all state transitions via control, and find a motion sequence that minimizes the cost C, which is defined as the completion time to cross the target area or the distance to the blood vessel wall. The AI ​​predictive control system 460 can calculate the motion sequence using equation (7)

[0160] Therefore, the AI ​​predictive control system 460 generates a motion command sequence from the long-term predictive control Furthermore, the cost function can be based on many factors, such as allowing the AI ​​predictive control system 460 to determine the motion command sequence in real time even without any observations. The optimal control of the long-term horizontal time will be applied to the robotic manipulator 12 to either autonomously control or have the user confirm the control to be executed. One advantage here is that if the safety boundary conditions are not met, the predictive control can request new observations from the user, allowing the user to actively supervise the system, or if it is an autonomous system mode, the integrated image modality will obtain new intermittent observations, as described above.

[0161] Figure 7B Shown Figure 7A Operation method of AI predictive control system. Figure 7B The method may be performed by the AI ​​predictive control system 460, and more specifically, by a processing circuit (eg, the controller 40) that executes the AI ​​predictive control system 460.

[0162] As reference Figure 7A As described above, at S720, the AI ​​predictive control system 460 obtains at least one movement command (e.g., CMD) for positioning an interventional device of the robotic surgery system into the patient's body. At S725, the AI ​​predictive control system 460 determines at least one actuator command (e.g., CMD) of the robotic surgery system associated with the at least one movement command based on the movement command and the estimated current position of the interventional device. ). At S730, the AI ​​predictive control system 460 applies the actuator commands to the robotic surgery system.

[0163] Additionally, steps S720, S725, and S730 may be based on the images of the patient obtained at S715. For example, if the safety boundary conditions are not met, the predictive control may request a new observation (e.g., a new actual X-ray image) from the user, allowing the user to actively supervise the system, or if in autonomous system mode, the integrated image modality will obtain a new intermittent observation.

[0164] Figure 8 According to one or more example embodiments Figure 7A Forward proxy shown.

[0165] The forward agent 710 includes a transformer 805 and an autoregressive time series transformer 810. The autoregressive time series transformer 810 includes a long-term prediction encoder 815 and a long-term prediction decoder 820. The long-term prediction encoder 815 receives the estimated tip position Proximal position Near-end status Estimated tip position Based on the input to the long-term prediction encoder 815, the long-term prediction decoder 820 generates an estimated tip position

[0166] The transformer 805 is configured to generate at least one attention heat map 830 of the image plane. The attention heat map may be generated based on at least one of the real image 802 generated by the imaging system 32 or the estimated image from the AI ​​predictive control system 460. Figure 8 In the illustrated embodiment, the forward agent 710 receives a first planar image set 802a and a second planar image set 802b. Each of the image sets 802a, 802b may include a DSA image and a fluorescence image. The transformer 805 generates attention heat maps 830a, 830b for the first and second image planes, respectively. The long-term prediction decoder 820 is configured to use the estimated tip position An estimated heat map 835a, 835b is generated for each image plane.

[0167] Figure 9 According to one or more example embodiments Figure 7A The reverse proxy shown.

[0168] The reverse proxy 705 includes an autoregressive time series transformer 910. The autoregressive time series transformer 910 includes a long-term prediction encoder 915 and a long-term prediction decoder 920. The long-term prediction encoder 915 receives p,e as input.

[0169] As mentioned, the AI ​​predictive control system provides an efficient way to dictate when to turn X-rays on / off.

[0170] In some example embodiments, the transformer model described herein may be the transformer described in Vaswani et al., “Attention Is All You Need,” 31st Conference on Neural Information Processing Systems (NIPS 2017), arXiv:1706.03762v7, the entire contents of which are incorporated herein by reference.

[0171] Figure 10 An example converter model for implementing an AI predictive control system according to one or more example embodiments is shown.

[0172] like Figure 10As shown, Transformer 1000 consists of an encoder 1010, a decoder 1002, a positional encoding 1003 with cascaded operations at encoder 1010 and decoder 1002, an input embedding 1004, and an output embedding 1005. The input x is embedded with the positional encoding. Encoder 1010 maps the input sequence of symbolic representations x to a representation sequence. A multi-head attention mechanism 1011 performs self-attention on the input. This multi-head attention mechanism 1011 encodes information about relevant context, allowing the model to focus on relevant context at different length scales. A feedforward network 1013 performs post-processing operations to generate a representation sequence z. Decoder 102 generates an output sequence y of symbols, one element at a time. Transformer 1000 uses L layers in encoder 1010 to perform sequential operations (e.g., 110_1, 110_2, ..., 110_L) and L layers for decoder 1002. The layers of the encoder 1010 are processed by addition and normalization components 1012, 1014 via skip feeds 1015, 1016 to perform residual connections, followed by layer normalization.

[0173] Although the terms first, second, etc. can be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of this disclosure, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0174] When an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or intervening elements may be present. In contrast, when an element is referred to as being "directly connected" or "directly coupled" to another element, no intervening elements are present. Other words used to describe the relationship between elements should be interpreted in a similar manner (e.g., "between" versus "directly between," "adjacent" versus "directly adjacent," etc.).

[0175] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms "comprises," "comprising," "includes," and / or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0176] It should also be noted that in some alternative implementations, the functions / acts mentioned may not occur in the order mentioned in the drawings. For example, depending on the functions / acts involved, two figures shown in succession may actually be executed substantially simultaneously, or may sometimes be executed in the reverse order.

[0177] Specific details are provided in the following description to provide a thorough understanding of the example embodiments. However, one of ordinary skill in the art will appreciate that the example embodiments may be practiced without these specific details. For example, systems may be shown in block diagrams so as not to obscure the example embodiments with unnecessary detail. In other cases, well-known processes, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the example embodiments.

[0178] As discussed herein, the illustrative embodiments are described with reference to symbolic representations of acts and operations (e.g., in the form of flowcharts, flow diagrams, data flow diagrams, structure diagrams, block diagrams, etc.), which may be implemented as program modules or functional processes, including routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types, and may be implemented using existing hardware, such existing hardware being, for example, processing or control circuitry such as, but not limited to, one or more processors, one or more central processing units (CPUs), one or more controllers, one or more arithmetic logic units (ALUs), one or more digital signal processors (DSPs), one or more microcomputers, one or more field programmable gate arrays (FPGAs), one or more systems on a chip (SOCs), one or more programmable logic units (PLUs), one or more microprocessors, one or more application specific integrated circuits (ASICs), or one or more other devices capable of responding to and executing instructions in a defined manner.

[0179] Although a flowchart may describe operations as a sequential process, many operations may be performed in parallel, concurrently, or simultaneously. Furthermore, the order of operations may be rearranged. A process may be terminated when its operations are completed, but may also have additional steps not included in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0180] As disclosed herein, the terms "memory," "storage medium," "processor-readable medium," "computer-readable storage medium," or "non-transitory computer-readable storage medium" may refer to one or more devices for storing data, including read-only memory (ROM), random-access memory (RAM), magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices, and / or other tangible machine-readable media for storing information. The term "computer-readable medium" may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instruction(s) and / or data.

[0181] In addition, the example embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments that perform the necessary tasks may be stored in a machine or computer readable medium, such as a computer readable storage medium. When implemented in software, one or more processors will perform the necessary tasks. For example, as described above, according to one or more example embodiments, at least one memory may include or store computer program code, and at least one memory and computer program code may be configured to, together with at least one processor, cause a network element or network device to perform the necessary tasks. In addition, the processor, memory, and example algorithms encoded as computer program code act as components for providing or causing the execution of the operations discussed herein.

[0182] As used herein, the terms "including" and / or "having" are defined as comprising (i.e., open language). The term "coupled," as used herein, is defined as connected, although not necessarily directly and not necessarily mechanically. Terms derived from the word "indicate" (e.g., "indicates" and "indicates") are intended to encompass all of the various techniques that can be used to convey or reference the indicated object / information. Some, but not all, examples of techniques that can be used to convey or reference the indicated object / information include: communication of the indicated object / information, communication of an identifier of the indicated object / information, communication of information used to generate the indicated object / information, communication of a portion or part of the indicated object / information, communication of a derivative of the indicated object / information, and communication of a symbol representing the indicated object / information.

[0183] The benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and any element(s) that may cause or lead to such benefits, advantages, or solutions, or cause such benefits, advantages, or solutions to become more significant, should not be construed as critical, required, or essential features or elements of any or all of the claims.

[0184] Non-limiting illustrative embodiments

[0185] The following is a list of non-limiting illustrative embodiments disclosed herein:

[0186] Illustrative Embodiment 1. A method includes obtaining, by a trained system, at least one movement command for a robotic surgical system for positioning an interventional device within a patient; determining, by the trained system, at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the interventional device; and applying the actuator command to the robotic surgical system.

[0187] Illustrative Example 2. The method according to Illustrative Example 1, wherein the interventional device is a guidewire.

[0188] Illustrative embodiment 3. The method of illustrative embodiment 2, wherein the at least one movement command is a desired position of the guidewire tip within the patient's body.

[0189] Illustrative Embodiment 4. The method of any of Illustrative Embodiments 2-3, further comprising obtaining an observation position of the guidewire from the robotic surgical system; and determining an estimated current position of the guidewire based on the observation position.

[0190] Illustrative Embodiment 5. The method of Illustrative Embodiment 4, wherein the observation position of the interventional device corresponds to the observation position of the proximal portion of the guidewire.

[0191] Illustrative embodiment 6. The method of illustrative embodiment 5, wherein the estimated current position corresponds to an estimated position of the guidewire tip.

[0192] Illustrative Embodiment 7. The method of any of Illustrative Embodiments 2-6, wherein the determining step determines a series of actuator commands, each actuator command in the series of actuator commands corresponding to a state of the guidewire.

[0193] Illustrative Embodiment 8. The method of any of Illustrative Embodiments 2-7, further comprising generating image data showing an estimated current position of the guidewire within the patient's blood vessel.

[0194] Illustrative Embodiment 9. The method of any of Illustrative Embodiments 2-8, wherein the estimated current position of the guidewire is overlaid on the three-dimensional vessel segmentation.

[0195] Illustrative embodiment 10. The method of any of illustrative embodiments 1-9, wherein the actuator command is at least one of a velocity command or a torque command to the actuator.

[0196] Illustrative Embodiment 11. A trained system includes processing circuitry configured to cause the trained system to obtain at least one movement command for a position of an interventional device of a robotic surgical system into a patient's body, determine at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the interventional device, and apply the actuator command to the robotic surgical system.

[0197] Illustrative Embodiment 12. The trained system according to Illustrative Embodiment 11, wherein the interventional device is a guidewire.

[0198] Illustrative Embodiment 13. The trained system of Illustrative Embodiment 12, wherein at least one movement command is a desired position of the guidewire tip within the patient.

[0199] Illustrative Embodiment 14. The trained system of any of Illustrative Embodiments 12-13, wherein the processing circuit is configured to cause the trained system to obtain an observation position of the guidewire from the robotic surgical system; and determine an estimated current position of the guidewire based on the observation position.

[0200] Illustrative Embodiment 15. The trained system of Illustrative Embodiment 14, wherein the observation position of the interventional device corresponds to the observation position of the proximal portion of the guidewire.

[0201] Illustrative Embodiment 16. The trained system of Illustrative Embodiment 15, wherein the estimated current position corresponds to an estimated position of the guidewire tip.

[0202] Illustrative Embodiment 17. The trained system of any of Illustrative Embodiments 12-16, wherein the processing circuit is configured to cause the trained system to determine a series of actuator commands, each actuator command in the series of actuator commands corresponding to a state of the guidewire.

[0203] Illustrative Embodiment 18. The trained system of any of Illustrative Embodiments 12-17, wherein the processing circuit is configured to cause the trained system to generate image data showing an estimated current position of the guidewire within a patient's blood vessel.

[0204] Illustrative Example 19. The trained system of any of Illustrative Examples 12-18, wherein the estimated position of the guidewire is overlaid on the three-dimensional blood vessel segmentation.

[0205] Illustrative Embodiment 20. The trained system of any of Illustrative Embodiments 11-19, wherein the actuator command is at least one of a velocity command or a torque command to the actuator.

[0206] Illustrative Embodiment 21. A trained system comprising means for obtaining at least one movement command for positioning an interventional device of a robotic surgical system within a patient's body; means for determining at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the interventional device; and means for applying the actuator command to the robotic surgical system.

[0207] Illustrative Embodiment 22. The trained system of Illustrative Embodiment 21, wherein the interventional device is a guidewire.

[0208] Illustrative Embodiment 23. The trained system of Illustrative Embodiment 22, wherein at least one movement command is a desired position of the guidewire tip within the patient.

[0209] Illustrative Embodiment 24. The trained system of any of Illustrative Embodiments 22-23, further comprising means for obtaining a viewing position of the guidewire from the robotic surgical system; and means for determining an estimated current position of the guidewire based on the viewing position.

[0210] Illustrative Embodiment 25. The trained system of Illustrative Embodiment 24, wherein the observation position of the interventional device corresponds to the observation position of the proximal portion of the guidewire.

[0211] Illustrative Embodiment 26. The trained system of Illustrative Embodiment 25, wherein the estimated current position corresponds to an estimated position of the guidewire tip.

[0212] Illustrative Embodiment 27. The trained system of any of Illustrative Embodiments 22-26, further comprising means for determining a series of actuator commands, each actuator command in the series of actuator commands corresponding to a state of the guidewire.

[0213] Illustrative Embodiment 28. The trained system of any of Illustrative Embodiments 22-27, further comprising means for generating image data showing an estimated current position of the guidewire within the patient's blood vessel.

[0214] Illustrative Embodiment 29. The trained system of any of Illustrative Embodiments 22-28, wherein the estimated position of the guidewire is overlaid on the three-dimensional blood vessel segmentation.

[0215] Illustrative Embodiment 30. The trained system of any of Illustrative Embodiments 21-29, wherein the actuator command is at least one of a velocity command or a torque command to the actuator.

Claims

1. A method comprising: obtaining, by the trained system, at least one movement command for an interventional device of the robotic surgical system to a position within the patient's body; determining, by the trained system, at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and the estimated current position of the interventional device; and Applying actuator commands to a robotic surgical system. The method of claim 1 , wherein the interventional device is a guidewire.

3. The method of claim 2, wherein the at least one movement command is a desired position of the guidewire tip within the patient's body.

4. The method according to claim 2, further comprising: Obtaining observational position of the guidewire from the robotic surgical system; and An estimated current position of the guidewire is determined based on the observed position. The method of claim 4 , wherein the observation position of the interventional device corresponds to the observation position of the proximal portion of the guidewire. The method of claim 5 , wherein the estimated current position corresponds to an estimated position of the guidewire tip.

7. The method of claim 2, wherein the determining step determines a series of actuator commands, each actuator command in the series of actuator commands corresponding to a state of the guidewire.

8. The method according to claim 2, further comprising: Image data is generated showing an estimated current position of the guidewire within the patient's blood vessel. The method of claim 8 , wherein the estimated current position of the guidewire is overlaid on a three-dimensional vessel segmentation. 10 . The method of claim 1 , wherein the actuator command is at least one of a velocity command or a torque command to the actuator.

11. A trained system comprising: processing circuitry configured to cause the trained system to obtaining at least one movement command for an interventional device of a robotic surgical system to a position within a patient's body, determining at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and the estimated current position of the interventional device, and Applying actuator commands to a robotic surgical system.

12. The trained system of claim 11, wherein the interventional device is a guidewire.

13. The trained system of claim 12, wherein the at least one movement command is a desired position of the guidewire tip within the patient's body.

14. The trained system of claim 12, wherein the processing circuit is configured to cause the trained system to: Obtaining viewing position of the guidewire from the robotic surgical system; and An estimated current position of the guidewire is determined based on the observed position.

15. The trained system of claim 14, wherein the observation position of the interventional device corresponds to the observation position of the proximal portion of the guidewire.

16. The trained system of claim 15, wherein the estimated current position corresponds to an estimated position of the guidewire tip.

17. The trained system of claim 12, wherein the processing circuit is configured to cause the trained system to: A series of actuator commands is determined, each actuator command in the series of actuator commands corresponding to a state of the guidewire.

18. The trained system of claim 12, wherein the processing circuit is configured to cause the trained system to: Image data is generated showing an estimated current position of the guidewire within the patient's blood vessel.

19. The trained system of claim 18, wherein the estimated current position of the guidewire is overlaid on a three-dimensional vessel segmentation.

20. The trained system of claim 11, wherein the actuator command is at least one of a velocity command or a torque command to the actuator.

21. A trained system comprising: means for obtaining at least one movement command for positioning an interventional device of a robotic surgical system into a patient's body; means for determining, based on the movement commands and the estimated current position of the interventional device, at least one actuator command for the robotic surgical system associated with the at least one movement command; and Apparatus for applying actuator commands to a robotic surgical system.

Images