Robotic tool control

Abstract

A system for controlling a robotic tool includes a memory storing instructions and a processor executing the instructions. When executed by the processor, the instructions cause the system to perform a process comprising monitoring sequential movement of tissue in a three-dimensional space. The process also includes: planning a position and a corresponding time when the tissue will be at the planned position in the three-dimensional space; identifying an identified position of the tissue in the three-dimensional space based on the planned position; and setting a trajectory of the robotic tool to encounter the tissue at the identified position at a planned time corresponding to the identified position.

Description

Background Art

[0001] Minimizing invasive surgical procedures on a beating heart can yield benefits such as reduced trauma and avoidance of extracorporeal circulation. Cardiac surgery is often performed in a confined and dynamic environment. Consequently, efforts to minimize cardiac surgery face challenges in navigating around the heart, visualizing the heart and its surroundings, and manipulating the heart and the tools used to perform cardiac surgery.

[0002] The motion associated with the heart and lungs is known as cardiopulmonary motion, and this motion is relatively predictable. Consequently, conventional practice in cardiac surgery focuses on motion compensation capabilities that allow robots to move in sync with tissue, freeing medical professionals (e.g., surgeons) from having to cognitively and manually process heartbeat motion. However, when tissue moves beyond one degree of freedom, conventional mechanical motion compensation capabilities may not be able to faithfully follow the tissue.

[0003] Another aspect of cardiac surgery is that the procedure can sometimes involve continuous manipulation of moving tissue. However, some procedures do not require this continuous manipulation of moving tissue. Examples of procedures that may not require continuous manipulation of moving tissue include mitral valve repair and patent foramen ovale closure. Summary of the Invention

[0004] According to one aspect of the present disclosure, a system for controlling a robotic tool includes: a memory storing instructions; and a processor executing the instructions. When executed by the processor, the instructions cause the controller to perform a process comprising: monitoring sequential movement of tissue in a three-dimensional space; planning positions and corresponding times when the tissue will be at the planned positions in the three-dimensional space; identifying an identified position of the tissue in the three-dimensional space based on the planned positions; and setting a trajectory of the robotic tool to encounter the tissue at the identified position at the planned time corresponding to the identified position.

[0005] According to another aspect of the present disclosure, a method for controlling a robotic tool includes monitoring sequential movement of tissue in a three-dimensional space. The method includes planning, by a tangible and non-transient processor, positions and corresponding times when the tissue will be at the planned positions in the three-dimensional space. The method also includes identifying an identified position of the tissue in the three-dimensional space based on the planned position and at least one characteristic of the robotic tool; setting a trajectory of the robotic tool to encounter the tissue at the identified position at a planned time corresponding to the identified position; and deploying the robotic tool or a medical treatment device attached to the robotic tool to encounter the tissue at the planned position along the trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The exemplary embodiments are best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that the various features are not necessarily drawn to scale. In fact, the dimensions may be arbitrarily increased or decreased for clarity of discussion. Where applicable and practical, like reference numerals refer to like elements.

[0007] Figure 1 A process for robotic tool control according to a representative embodiment is illustrated.

[0008] Figure 2A A system for robotic tool control according to a representative embodiment is illustrated.

[0009] Figure 2B A general purpose computer system including an instruction set for robotic tool control is illustrated according to a representative embodiment.

[0010] Figure 3 Another process for robotic tool control according to a representative embodiment is illustrated.

[0011] Figure 4 A heart is illustrated with tissue designated by an operator for robotic tool control, according to a representative embodiment.

[0012] Figure 5 FIGURE 1 illustrates a diagram of a robotic tool with a suggested trajectory for robotic tool control according to a representative embodiment. Figure 4 heart.

[0013] Figure 6 FIGURE 1 illustrates an actual trajectory of a robotic tool with use in robotic tool control according to a representative embodiment. Figure 4 heart.

[0014] Figure 7A Illustrated is a simplified version of tissue motion monitored in robotic tool control according to a representative embodiment.

[0015] Figure 7B Another simplified form of tissue motion monitored in robotic tool control is illustrated in accordance with a representative embodiment.

[0016] Figure 7C Another simplified form of tissue motion monitored in robotic tool control is illustrated in accordance with a representative embodiment.

[0017] Figure 8 Another process for robotic tool control according to a representative embodiment is illustrated.

[0018] Figure 9Data obtained from monitoring tissue motion for robotic tool control is illustrated according to a representative embodiment. DETAILED DESCRIPTION

[0019] This article describes a surgical system. The surgical system exploits the idea that for various procedures, a robot may only need to interact with the moving tissue of the heart for a fraction of a second (e.g., within a heartbeat). The robotic control strategy for the surgical system monitors tissue motion from the heartbeat and may monitor tissue motion from other sources (e.g., respiration). The robotic control strategy integrates patterns identified from tissue motion into a control loop so that the desired trajectory, as defined by the user or task, is matched based on the timing of tissue motion. The operator may not be able to see the matching of heartbeat-based tissue motion to the robotic control, so the operator only needs to specify the tissue target and the robot calculates the trajectory and timing required to run the deployment. Thus, the cognitive burden on the medical professional is reduced and operation is simplified.

[0020] In the following detailed description, representative embodiments of the disclosure details are set forth for the purpose of explanation rather than limitation, so as to provide a thorough understanding of the embodiments according to the present teachings. Descriptions of known systems, devices, materials, operating methods, and manufacturing methods may be omitted to avoid making the description of the representative embodiments unclear. Nevertheless, systems, devices, materials, and methods within the capabilities of those of ordinary skill in the art are within the scope of the present teachings and may be used according to the representative embodiments. It should be understood that the terms used herein are only for the purpose of describing specific embodiments and are not intended to be limiting. The defined terms are terms outside the technical and scientific meanings of the defined terms that are generally understood and accepted in the technical field of the present teachings.

[0021] It should be understood that although the terms first, second, third, etc. may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another element or component. Therefore, without departing from the teachings of the present invention, the first element or component discussed below may be referred to as the second element or component.

[0022] The terms used herein are for the purpose of describing specific embodiments only and are not intended to be limiting. As used in the specification and claims, the singular forms of the terms "a", "an" and "the" are intended to include both the singular and the plural forms, unless the context clearly indicates otherwise. In addition, when used in this specification, the terms "include" and / or "comprise" and / or similar terms specify the presence of the features, elements and / or parts, but do not exclude the presence or addition of one or more other features, elements, parts and / or their groups. The term "and / or" used herein includes any combination and all combinations of one or more of the associated listed items.

[0023] Unless otherwise specified, when an element or component is referred to as being "connected to," "coupled to," or "adjacent to" another element or component, it should be understood that the element or component can be directly connected or coupled to the other element or component, or that intervening elements or components may be present. In other words, these terms and similar terms cover situations where one or more intervening elements or components may be used to connect two elements or components. However, when an element or component is referred to as being "directly connected" to another element or component, this only covers situations where the two elements or components are connected to each other without any intervening or intervening elements or components.

[0024] In view of the foregoing, the present disclosure is intended to provide one or more of the advantages specifically noted below through one or more of its various aspects, embodiments, and / or specific features or subcomponents. For purposes of explanation and not limitation, example embodiments disclosing specific details are set forth to provide a thorough understanding of embodiments according to the present teachings. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein are still within the scope of the claims. In addition, descriptions of well-known apparatus and methods may be omitted so as not to obscure the description of the example embodiments. Such methods and apparatus are within the scope of the present disclosure.

[0025] As described herein, for some cardiac surgical procedures such as mitral valve repair, a robot is able to grasp the target tissue to reduce or stop tissue motion, thereby securing the tissue for further manipulation. By reducing or stopping tissue motion, even if only for a short while, the robot is able to reduce its reliance on the integrity of motion compensation. As an alternative to grasping the target tissue, the robot can simply deploy a device attached to the target tissue. Whether grasping the target tissue, deploying the device, or performing any other procedure that can be performed quickly, the robot only needs to adapt to the heartbeat motion within a single time interval. As a result, continuous motion compensation is technically redundant and requires the robot to include mechanisms with higher performance, precision, complexity, and cost. The robotic tool control described herein can be used alone without continuous motion compensation, or by a system that selectively uses robotic tool control for one or more procedures while also using continuous motion compensation for other procedures.

[0026] Figure 1 A process for robotic tool control according to a representative embodiment is illustrated. Figure 1 , the process begins at S110, at which point a 4-dimensional (4D) rendering of the heart is generated based on preoperative images. The term "4D" as used herein refers to a sequence of 3-dimensional (3D) renderings obtained over time, so time is the fourth dimension. Preoperative images are images of the patient before surgery, and may include images taken immediately before surgery and images taken at a location different from the surgical site long before surgery. Additionally, while it may be useful to render the heart at S110, it is not specifically required for the robotic tool control described herein to function properly.

[0027] At S120, the 4-dimensional rendering is registered to the patient model, intraoperative imaging, and the robot. The patient model is, for example, patient-specific information in a predetermined format. The 4-dimensional rendering can be part of the patient model or simply associated with the patient model. Intraoperative imaging is imaging of the patient during surgery, for example, by ultrasound or X-ray. The 4-dimensional rendering can be used for comparison with the intraoperative images to determine or confirm physiological aspects of the patient encountered during surgery. The robot is a medical robot that moves in 1, 2, or 3 positional dimensions and 1, 2, or 3 rotational dimensions to move robotic tools and / or commercial therapeutic devices in space as part of a surgery. An example of a commercial therapeutic device is a medical therapeutic device.

[0028] At S130, the robot is guided to a pre-deployment position near the target tissue. The specific tissue may not yet be identified, so the pre-deployment position can be a default pre-mapped position in the 3D surgical space. The 3D surgical space can be provided with an origin so that the robot and all information can be mapped relative to the same origin using predefined directional axes. The robot can be guided automatically or manually. Automatic guidance can be performed by a computer running a program based on intraoperative images obtained during surgery. Manual guidance can be performed by an operator using, for example, a joystick, a touch screen, a roller ball, or other forms of touch input.

[0029] At S140, the anatomical location to be targeted by the robot is set. The anatomical location can be a specific part of the tissue identified by pathology or other clinical characteristics, color or other visual characteristics, or by positioning relative to other anatomical locations. Other anatomical locations can be considered to be in the form of anatomical landmarks. The medical professional can also select the anatomical location from the image, and the surgical system can also automatically select the anatomical location from the image using anatomical landmarks, or automatically select the anatomical location by using the characteristics of the tissue to be targeted (e.g., force values ​​from a mechanical sensor or optical readings from an optical sensor). The anatomical location can be automatically set by a computer (e.g., a computer running a program based on intraoperative images), but when set automatically, the location should be confirmed by the medical professional. Alternatively, the anatomical location to be targeted by the robot can be set by the medical professional (e.g., by an operator using a joystick, touch screen, roller ball or other form of touch input).

[0030] As an alternative to using images, tissue motion can also be sensed through sensing that does not necessarily involve imaging. For example, tissue motion information can be obtained through range sensing, pressure sensing, electrical (e.g., capacitance) sensing, etc. Any form of sensing of tissue motion can be used in addition to or in place of the intraoperative images described herein.

[0031] At S150, tissue motion is interactively monitored and then matched to the motion model. Tissue motion can include various forms of movement, including heartbeat motion, respiratory motion, body motion caused by the patient moving muscles, and / or feedback motion caused by the patient moving in response to a previously performed movement by the robot. The motion can be movement of one or more parts of the organ in different directions, including XYZ motion and rotational motion. In addition, the motion can include multiple different components in any direction or rotation. For example, the entire heart may be moving at a measured pace, while at the same time, specific tissues of the heart move at a pace that deviates from the measured pace. Similarly, specific tissues of the heart may move in a manner similar to the parts of a circus ride, where the arms rotate about an axis, the arms move up and down while rotating about an axis, and the carriages at the ends of the arms rotate around different axes at the ends of each arm. Of course, the tissues of the heart do not include the full range of such similar motions, but the different degrees of motion may be similar. The tissue motion monitored at S150 may be monitored by a variety of mechanisms including electrocardiogram (ECG), live imaging, predetermined models synchronized with live patient physiological signals, or some hybrid thereof.The cardiac tissue may be marked and monitored in 3 dimensions.

[0032] In addition, multiple motion models can be pre-registered and stored in memory. For example, different tissue motions that have been identified as corresponding to different symptoms or diseases can be pre-registered as different motion models. The motion model can also correspond to the patient's age and gender, as well as the patient's demographic information and health characteristics. Alternatively, the motion model can be completely independent of the patient's demographic information and health characteristics and can specifically correspond only to the type and pattern of tissue motion observed.

[0033] At S155, the robot is driven along the trajectory to encounter the tissue at the identified location. The robot may be driven from the pre-deployed position reached at S130. The driving of the robot may occur in several ways / modes. That is, Figure 1 The intended result of the process in is that the robot's end effector is deployed as described below with respect to S170. When the robot encounters tissue, the deployment of the robot's end effector can take many forms, such as grasping the tissue, installing (e.g., attaching, attaching) a device, cutting the tissue, etc. The trajectory itself can reflect several considerations, including meeting the target cardiac tissue, avoiding collisions with other tissues and any other targets in the 3D surgical space, and considering speed and direction to minimize or completely avoid the possibility of injury to the patient if the trajectory is invalidated due to unexpected behavior of the patient.

[0034] The position at which the robot's end effector is deployed can be a single optimal position determined according to predetermined criteria. For example, the optimal position for the robotic tool to intercept cardiac tissue can be a position where the movement of the cardiac tissue is at a position extreme and / or a velocity minimum. Alternatively, the optimal position of the cardiac tissue can be a position where the patient has completed exhalation and is about to begin inhalation. Therefore, the optimal position can correspond to the physiological characteristics of the patient during surgery, and these physiological characteristics can be determined dynamically in real time. Alternatively, the optimal position can correspond to the characteristics of the tissue movement, and these motion characteristics can also be determined dynamically in real time. In addition, the optimal position can correspond to an optimal time, where the primary consideration is timing rather than position. For example, this may be correct for setting the position based on respiratory timing.

[0035] The position at which the robot's end effector is deployed can also be a preferred position from a set of multiple optimal positions, such as when the multiple positions meet a minimum threshold for identifying potential interception positions. The preferred position can be randomly determined from a set of feasible positions, can be specified by a medical professional from a set of feasible positions, or can be determined based on user settings of a medical professional (e.g., whether the surgeon is left-handed or right-handed).

[0036] In a first mode of telesurgery deployment, a medical professional may use one or more input devices to move the robot toward a target tissue, and the surgical system implicitly uses heartbeat timing signals to constrain true motion along the trajectory. The constraints may limit the speed and position of the movement, and even if heartbeat timing signals are used to constrain the motion, such use is performed in a form that is not visible or noticeable to the medical professional, such that the medical professional is unaware that the constraints have been imposed. In a second mode of autonomous deployment, the medical professional may specify the target tissue to the surgical system, and the surgical system automatically performs the movement. In the second mode, the system is able to select the optimal timing and trajectory for deploying the robot's end effector. Additionally, as Figure 5 As shown, the surgical system can present a simulated view of what the deployment will look like for confirmation by the medical professional.

[0037] At S160, it is determined whether the target state of tissue movement has been reached. The target state may be, for example, a specific position in 3D and / or the speed and direction or trajectory of the tissue's movement. If the tissue is not in the expected state, the process may be restarted. Figure 1 The determination can be made by comparing the intraoperative image with the 4D rendering in real time, or by marking the tissue with, for example, a dye, so that the medical professional can visually make the determination at S160. If the target position has not been reached (S160=No), the process returns to S150 and interactive monitoring of tissue movement is performed again.

[0038] If the target position is reached (S160=Yes), the end effector is deployed at S170. The end effector may be a robotic tool or a commercially available therapeutic device, such as a mitral valve clip or valve.

[0039] Figure 2A A system for robotic tool control according to a representative embodiment is illustrated. Figure 2A In the embodiment of the present invention, structure 204 can be a bed or table (e.g., an operating table). The 3-dimensional space around structure 204 can be pre-mapped as described herein. Imaging instrument 205 can be a video camera, ultrasound equipment, X-ray equipment, or other equipment capable of capturing images of human organs and tissues during medical surgery.

[0040] The imaging instrument 205 feeds information from the captured images to a computer 206. The computer 206 may store and run motion tracking software that tracks the movement of tissue in real time during surgery in order to plan a trajectory for the robotic tool 207 to encounter the tissue. The robot 202 receives instructions from the computer 206 and manipulates the arms 203A and 203B in three dimensions to move the robotic tool 207 to encounter the tissue at an identified location at an identified time. Manipulation of the arms 203A and 203B may also involve rotation. Manipulation of the arms 203A and 203B involves moving the robotic tool 207 along the trajectory to encounter the tissue while ensuring that the arms 203A and 203B and any other associated components do not collide with the patient or any instrument parts due to, for example, the trajectory of the robotic tool 207.

[0041] Although computer 206 is described as performing most or all logical operations for the embodiment of FIG2 , the robot 202 can directly perform all logical operations if the robot has appropriate mechanisms, such as a processor and memory, for performing the logical operations. In one embodiment, the robot 202 and computer 206 are part of a single integrated system. In another embodiment, the robot 202 and computer 206 can be provided as a single subsystem. The logical operations include motion tracking of a designated tissue, planning the future position and time of the designated tissue based on the motion tracking, and setting a trajectory for the robotic tool 207 based on the planned future position and time of the designated tissue.

[0042] The robot 202 described herein operates based on the concept that continuous motion compensation may not be required, but rather that temporary but rapid observations of tissue motion can be incorporated into the control strategy. Thus, the robot 202 controls its arms 203A and 203B based on observations of tissue motion confirmed by the trajectory of the robotic tool 207. The tissue to be monitored can be set by a medical professional, and the trajectory can be defined or confirmed by the medical professional for the task to be performed. The location and time of interception of moving tissue are determined based on the timing of tissue motion. The processing required to set the interception location and time is invisible to the operator, allowing the operator to simply specify the tissue target, and the computer 206 calculates the trajectory and timing required to execute the deployment. As a result, the cognitive burden on the medical professional is reduced and the procedure is simplified. The computer 206 does not need to track rapid and sudden tissue motion for an extended period of time and can perform the procedures described herein within a single heartbeat or a few heartbeats. Therefore, monitoring / tracking tissue motion to identify patterns caused by heartbeat and respiration can be considered similar to gating. As described herein, the tracked motion can be used to plan the location and corresponding timing of cardiac tissue and generate a trajectory for the robotic tool to intercept cardiac tissue.

[0043] Because the system uses heartbeat timing to time the deployment of the robot's end-effector, the surgical system must also determine where the tissue should be when deployment occurs so that the correct trajectory can be calculated. The surgical system may consider various parameters when making this determination, including when the tissue is moving slowest, fastest, or at its most extreme points. The calculation of the trajectory of the robotic tool 207 can take into account the mechanical / computational delays of the robot 202 and the likelihood of successful deployment in a given scenario.

[0044] In an embodiment, the robotic system may include an embedded sensing system that monitors tissue motion. The embedded sensing system may be or include, for example, a small ultrasound transducer, electrodes (cardiac mapping probe), or a mechanical sensor. Information from the sensing system can be used to generate a model of tissue motion prior to the procedure and to update the model in real time, and / or to combine the information with an external monitoring system. Figure 2A In the present invention, such a sensing system is generally represented by an imaging instrument 205. Information from the sensing system can be used to distinguish different sub-phases of tissue movement via aperiodic or nonlinear motion models. The robot 202 can ultimately control the arms 203A and 203B in real time based on the information from the sensing system, and the control can be performed via the computer 206 and / or the robot 202 using memory or statistical models to take action according to each intraoperative situation.

[0045] In an embodiment, the robotic system may include a controller based on the robot 202 and / or the computer 206. That is, a memory storing instructions and a processor executing the instructions may be components of the controller. Such a controller may be a subsystem of the robot 202 or the computer 206, or may be a distributed subsystem of both the robot 202 and / or the computer 206, or may even be another device such as a remote computer that partially executes the instructions. The instructions executed by the controller are used to perform some or all of the processes described herein for robotic tool control (including the processes described herein). Figure 1 and Figure 8 process).

[0046] Figure 2B A general purpose computer system including an instruction set for robotic tool control is illustrated according to a representative embodiment. Figure 2B 2 is an illustrative embodiment of a general-purpose computer system on which a method for robotic tool control, shown and labeled 200, can be implemented. The computer system 200 can include an instruction set that can be executed to cause the computer system 200 to perform any one or more of the methods or computer-based functions disclosed herein. The computer system 200 can operate as a standalone device or can be connected to other computer systems or peripheral devices, for example, using a network 201.

[0047] In a networked deployment, the computer system 200 can operate as a server, or as a client user computer in a server-client user network environment, or as a peer computer system in a peer-to-peer (or distributed) network environment. The computer system 200 can also be implemented as a variety of devices or incorporated into a variety of devices, such as controllers, fixed computers, mobile computers, personal computers (PCs), laptop computers, robots, medical robots, surgical robots, robotic systems, or any other machine or system capable of (in a sequential manner or otherwise) running an instruction set that specifies the actions to be performed by the machine. The computer system 200 can be incorporated into a device or incorporated into a device that is in turn in an integrated system that includes additional devices. In an embodiment, the computer system 200 can be implemented using electronic devices that provide voice, video, or data communications. In addition, although a single computer system 200 is illustrated, the term "system" should also be considered to include any collection of systems or subsystems that individually or jointly run one or more instruction sets to perform one or more computer functions.

[0048] like Figure 2BAs shown, computer system 200 includes processor 210. The processor for computer system 200 is tangible and non-transient. The term "non-transient" as used herein should not be interpreted as a permanent state characteristic, but rather as a state characteristic that will persist for a period of time. The term "non-transient" specifically excludes transient characteristics, such as carrier waves, signals, or other forms of characteristics that only exist briefly at any time and place. A processor is an article of manufacture and / or a machine component. The processor for computer system 200 is configured to execute software instructions to perform the functions described in the various embodiments herein. The processor for computer system 200 can be a general-purpose processor or can also be part of an application-specific integrated circuit (ASIC). The processor for computer system 200 can also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic unit. The processor for computer system 200 can also be a logic circuit, including a programmable gate array (PGA) such as a field programmable gate array (FPGA), or another type of circuit including discrete gates and / or transistor logic units. The processor for computer system 200 can be a central processing unit (CPU), a graphics processing unit (GPU), or both. In addition, any processor described herein can include multiple processors, parallel processors, or both. Multiple processors can be included in a single device or multiple devices or coupled to a single device or multiple devices.

[0049] In addition, computer system 200 includes a main memory 220 and a static memory 230 that can communicate with each other via bus 208. The memory described herein is a tangible storage medium capable of storing data and executable instructions, and is non-transient during the time the instructions are stored in the memory. The term "non-transient" as used herein should not be interpreted as a permanent state characteristic, but rather as a state characteristic that persists for a period of time. The term "non-transient" specifically disclaims transient characteristics, such as carrier waves, signals, or other forms of characteristics that exist only briefly at any time and in any place. The memory described herein is an article of manufacture and / or a machine component. The memory described herein is a computer-readable medium from which a computer can read data and executable instructions. The memory described herein can be random access memory (RAM), read-only memory (ROM), flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, magnetic tape, a compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a floppy disk, a Blu-ray disc, or any other form of storage medium known in the art. The memory may be volatile or non-volatile, secure and / or encrypted, unsecure and / or unencrypted.

[0050] As shown, the computer system 200 may also include a video display unit 250 (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT)). In addition, the computer system 200 may include an input device 260 (e.g., a keyboard / virtual keyboard or a touch-sensitive input screen or voice input with voice recognition) and a cursor control device 270 (e.g., a mouse or a touch-sensitive input screen or pad). The computer system 200 may also include a disk drive unit 280, a signal generating device 290 (e.g., a speaker or a remote control), and a network interface device 240.

[0051] In an embodiment, Figure 2B As shown, the disk drive unit 280 may include a computer-readable medium 282 in which one or more instruction sets 284 (e.g., software) may be embedded. The instruction set 284 may be read from the computer-readable medium 282. In addition, the instructions 284, when executed by the processor, may be used to perform one or more of the methods and processes described herein. In an embodiment, the instructions 284 may reside completely or at least partially in the main memory 220, the static memory 230, and / or the processor 210 during execution by the computer system 200.

[0052] In alternative embodiments, dedicated hardware implementations (e.g., application specific integrated circuits (ASICs), programmable logic arrays, and other hardware components) can be constructed to implement one or more of the methods described herein. One or more embodiments described herein may use two or more specific hardware modules or devices interconnected by control and data signals that can communicate between and through modules to implement functionality. Therefore, the present disclosure encompasses software implementations, firmware implementations, and hardware implementations. Nothing in this application should be interpreted as being implemented solely by software and not by hardware (e.g., a tangible, non-transient processor and / or memory).

[0053] According to various embodiments of the present disclosure, a hardware computer system running a software program can be used to implement the methods described herein. In addition, in exemplary, non-limiting embodiments, implementations can include distributed processing, component / object distributed processing, and parallel processing. A virtual computer system process can be constructed to implement one or more of the methods or functions described herein, and the processors described herein can be used to support a virtual processing environment.

[0054] The present disclosure contemplates a computer-readable medium 282 that includes instructions 284 or receives and executes instructions 284 in response to a propagated signal, enabling devices connected to the network 201 to communicate voice, video, or data over the network 201. Additionally, the instructions 284 may be sent or received over the network 201 via the network interface device 240.

[0055] Figure 3 Another process for robotic tool control according to a representative embodiment is illustrated. Figure 3 The process in FIG3 begins at S310, where a 4-dimensional rendering is generated by acquiring 3D ultrasound frames. The 4-dimensional rendering is also registered to the anatomy of the patient undergoing the medical procedure. In other words, at S310, a 4-dimensional rendering of the heart is generated based on preoperative images (e.g., a time series of 3-dimensional ultrasound images). The rendering is registered to the patient model, intraoperative imaging, and the robot. S310 is not a necessary component of the control system itself, but including S310 can increase the overall usability of the robotic system.

[0056] At S320, the robot is guided to a pre-deployment position. As described above, the robot can be guided to the pre-deployment position manually or by an automated process. When using the manual approach, the robot can be guided to the vicinity of a specific organ of the patient, based on the judgment of the medical professional controlling the robot. When using the automated process, the robot can be guided to a default position in a controlled area around the patient, for example, a position above the feet of an operating table supporting the patient. In other words, at S320, the medical professional and / or the robot 202 guides the robot to a rough pre-deployment position near the target site, for example, the mitral valve. This part of the navigation is simpler than the intracardiac stage, and therefore, precise coordination between the monitored heart movement and the trajectory of the robotic tool is not necessary.

[0057] At S330 , a simulation of the robot 202 being guided to its final position is generated visually or using a 3D printed model, and then displayed to the medical professional controlling the robot. In embodiments, multiple simulations can be generated and displayed, allowing the medical professional to select the simulation to use. The medical professional can also be provided with the option of selecting from an automated model or manually guiding the robotic tool within the constraints imposed by the surgical system. The simulation can show the trajectory of the robotic tool from a pre-deployed position to the target tissue. Images can be used to pre-map a 3D space encompassing the robot's pre-deployed position and the surgical field or a relevant portion of the surgical field, and images acquired during the surgical session can be incorporated to provide context for the patient's anatomy. The 3D ultrasound frames acquired at S310 are examples of images acquired during the surgical session, but other forms of imaging can also be used to provide context for a specific patient relative to the pre-mapped 3D space used for the procedure. In other words, at S330 , the medical professional indicates the desired anatomical location to be targeted by the robot via pre-operative and intraoperative imaging displays. Simulations of the robot's motion, along with predicted tissue motion, can optionally be displayed. Medical professionals can confirm or readjust goals as needed.

[0058] At S340, the system triggers the user to execute positioning at the next heartbeat. To explain, during surgery, the patient is closely monitored using medical instruments, and this monitoring includes precise monitoring of the heartbeat and other relevant medical conditions (e.g., respiratory function). The controlled robotic tool 207 is deployed based on this precise monitoring and according to the trajectory shown in the simulation guidance at S330. In other words, since the heart moves according to the heartbeat, respiratory function, and other conditions, the movement of the heart is planned based on this monitoring. Therefore, the planned location of the tissue and the corresponding time are determined, and the trajectory shown in the simulation guidance is specifically generated to ensure that the robotic tool or a device attached to the robotic tool encounters the tissue at the planned location at the planned time. Once confirmed, the robotic system monitors the live tissue movement via intraoperative imaging (e.g., 3D ultrasound) or other forms of sensing and matches this movement to a known motion model that predicts tissue movement. Once the tissue reaches the target location, the robot executes deployment of the end effector. This may involve grasping a leaflet or introducing the device into the tissue.

[0059] Figure 4 A heart is illustrated with tissue designated by an operator for robotic tool control, according to a representative embodiment. Figure 4 S320 is illustrated, wherein a trajectory is shown by which the robotic tool or treatment device is roughly guided to a position near a specific tissue of the target heart. Figure 4 、 Figure 5 and Figure 6 In the present invention, the heart is labeled with many different identifiers that correspond to parts of the heart that medical professionals will recognize. With respect to any such part of the heart mentioned herein, the identifier can be referenced to provide a generalized context for the discussion.

[0060] Figure 5 FIGURE 1 illustrates a diagram of a robotic tool with a suggested trajectory for robotic tool control according to a representative embodiment. Figure 4 heart. Figure 5 S330 is illustrated, in which a simulated guide to the final position of the specific tissue is generated and displayed to the user. Figure 5 The simulation results in are shown by the dashed curves of the suggested trajectories, and the encounter positions of specific tissues with the robotic tool or treatment device are shown by arrows.

[0061] Figure 6 FIGURE 1 illustrates an actual trajectory of a robotic tool with use in robotic tool control according to a representative embodiment. Figure 4 heart. Figure 6 S340 is illustrated, wherein actual deployment of the robotic tool or therapeutic device is performed at the next heartbeat based on a user trigger. The time of the user trigger is the time when the medical professional confirms the simulation guidance from S330. The next heartbeat is the heartbeat after the user trigger. Of course, based on confirmation via the user trigger, the system can wait or tolerate two or more heartbeats before deployment at S340. However, because the simulation results at S340 are precisely based on real-time monitoring of the heart and medical condition, the simulation results will generally be invalid after a long period of time. Therefore, the language of "next" heartbeat is intended to convey that the deployment of the robotic tool is performed quickly after the user confirms the trajectory simulated at S330 at S340.

[0062] The simulation at S330 can be updated quickly and in real time. For example, the simulation of the heart motion and the planned trajectory of the robotic tool or treatment device can be performed once for each heartbeat, every 1 second, or even faster (e.g., every 0.1 second). The repeated execution of the simulation can be reflected in the refreshed image on the monitor. The simulation results will generally not change significantly from one refresh to the next, but may change based on sudden influences (e.g., non-rhythmic movement of the heart, or even non-rhythmic movement of only part of the heart).

[0063] Figure 7A A simplified diagram of tissue motion monitored in robotic tool control according to a representative embodiment is shown. Figure 7A, the heart's motion is shown to be linear. Linear motion may still be reflected in changes in the values ​​of each of the X-direction (horizontal), Y-direction (vertical), and Z-direction (depth), as long as the linear movement is almost never along a fixed axis in the X-direction, Y-direction, or Z-direction.

[0064] Figure 7B Another simplified form of tissue motion monitored in robotic tool control is illustrated in accordance with a representative embodiment. Figure 7B In , the motion of the heart appears to be purely rotational. However, rotational motion can also vary in radius relative to the central axis (or origin) about which the rotation occurs. Figure 7C , angles are specified as theta (Θ) and phi (Φ), where Θ is the angle in the XY plane and Φ is the angle in the YZ plane.

[0065] Figure 7C Another simplified form of tissue motion monitored in robotic tool control is illustrated in accordance with a representative embodiment. Figure 7C In the , rhythmic movement is shown as Figure 7A The motion that occurs in the segment being traced back in at least one dimension of the XYZ scale in . That is, Figure 7C The movement in can be fully or partially retraced back and forth in the X direction, Y direction and / or Z direction. Figure 7C The motion in is rhythmic in its temporal component, such as by fully or partially repeating the pattern by hitting Y maxima and Y minima at regular, periodic intervals.

[0066] Given the complexity of the human body and multiple functional organs, the movement of cardiac tissue may reflect multiple rhythms that overlap in different directions at the same time. Upon close inspection, the results may appear visually confusing, but one or more rhythms can be detected using motion tracking software that can analyze changes in the position of cardiac tissue over time. In addition, in addition to (one or more) rhythmic movements, motion analysis software can also separate linear or rotational motion. Once separated, motion can be planned forward in time to identify the position of cardiac tissue at a corresponding time in the (near) future.

[0067] Figure 8 Another process for robotic tool control according to a representative embodiment is illustrated. Figure 8 In the present invention, the process starts at S805, where the tissue to be monitored is identified based on input from the operator. The operator can visually identify the tissue and then use an input device such as a mouse and cursor to specify the tissue.

[0068] At S810, the identified tissue is monitored sequentially over time. That is, the identified tissue and surrounding tissue are monitored, for example, by images, for processing by motion analysis software.

[0069] At S815, the sequential motions identified from the monitoring at S810 are matched to predetermined patterns. For example, a plurality of predetermined patterns may be stored in a memory and used to match the motions identified from the monitoring at S810. Each of the predetermined patterns may be associated with a different identifier, for example to specify when the pattern has been previously associated with a type of medical condition.

[0070] At S820, the position and timing of the cardiac tissue are planned based on the monitoring. The position and timing may be planned partially or completely using the predetermined pattern matched at S815.

[0071] At S830, an optimal position and time for intercepting tissue using a robotic tool or a therapeutic tool are identified. Alternatively, the identified position and time may be preferred but not optimal. That is, multiple potentially suitable positions and times may be identified, and a single position and time may be designated by a medical professional or according to a predetermined setting. Thus, the identified position may be systematically optimal among a set of potential positions, or may be preferred, for example, when no clear optimal position exists.

[0072] At S840, the tool trajectory is set to encounter the tissue at the optimal (or preferred) location at the optimal (or preferred) time. The trajectory can be a precise setting of the future tool position. The time for multiple positions can be set to the millisecond or even microsecond level, and the position of the robotic tool can be set to XYZ coordinates relative to the origin or two RΘΦ coordinates relative to the origin.

[0073] At S850, a determination is made as to whether the tissue is in the optimal position and time. This determination can be made by motion tracking software or sensors that sense range, pressure / touch, or electrical properties (e.g., capacitance). If the tissue is not in the optimal position at the expected time (S850=No), the tool is withdrawn and the process returns to S810 for repetition. If the tissue is in the optimal position at the expected time (S850=Yes), the procedure is performed at S860. The procedure can be inserting a needle, cutting tissue, applying a therapeutic tool such as a mitral clip or valve, or other similar medical procedure.

[0074] Figure 9 1 illustrates data obtained from monitoring tissue motion for robotic tool control according to a representative embodiment. Figure 9, twenty-five (25) times are shown in sequence from T1 to T25 in the first column. The X values ​​are shown as X1-X25 and are the X (axis) coordinate data values ​​of the tissue monitored at each time from T1 to T25. The Y values ​​are shown as Y1-Y25 and are the Y (axis) coordinate data values ​​of the tissue monitored at each time from T1 to T25. The Z values ​​are shown as Z1-Z25 and are the Z (axis) coordinate data values ​​of the tissue monitored at each time from T1 to T25. The R values ​​are shown as R1-R25 and are the R (radius) data values ​​of the tissue monitored at each time from T1 to T25. The θ values ​​are shown as θ1-θ25 and are the θ (angle in the XY plane) data values ​​of the tissue monitored at each time from T1 to T25. The Φ values ​​are shown as Φ1-Φ25 and are the Φ (angle in the ZY plane) data values ​​of the tissue monitored at each time from T1 to T25.

[0075] Figure 9 The data shown is thus positional data of a specific tissue monitored over time. This data is from the tissue being monitored to identify patterns such as linear motion, rotational motion, and rhythmic motion. These patterns are then used to plan the optimal or preferred position and time at which the robotic tool or treatment device can intercept the monitored tissue.

[0076] As described above, the surgical system allows medical professionals to robotically perform tasks on or in the presence of moving targets without having to cognitively consider motion in the decision-making process. The nature of the tasks and robotic motions handled by the surgical system described herein is discrete, such that continuous robotic manipulation of a moving target is not required, although other uses may still exist. Nevertheless, the surgical system described herein can still be used to perform robotic tasks that use or previously used motion compensation because such tasks can be transformed into a series of discrete robotic deployments rather than continuous manipulation. The motion compensation problem previously used has been replaced so that the system does not have to follow the motion of the tissue, but rather the system can pick out distinct situations for spot deployment and time the motion according to the motion model. Where applicable, the methods provided by the robotic motion control described herein may be easier to implement and execute than simple motion compensation.

[0077] While the robotic tool control has been described with reference to several exemplary embodiments, it is to be understood that the words which have been used are words of description and illustration, rather than words of limitation. Changes may be made within the scope of the claims, as presently stated and amended, without departing from the scope and spirit of the robotic tool control aspects. While the robotic tool control has been described with reference to particular means, materials, and embodiments, the robotic tool control is not intended to be limited to the details disclosed; rather, the robotic tool control extends to all functionally equivalent structures, methods, and uses, such as those within the scope of the claims.

[0078] As described above, even when tissue motion occurs in more than one degree of freedom, the surgical system can monitor the tissue and plan its position and timing in advance. Such monitoring can even be used to identify complex patterns that account for the actual motion of tissue in multiple degrees of freedom. Thus, even in situations where tissue motion is rapid or sudden, the surgical system can dynamically, precisely, and accurately deploy robotic tools at specific, pre-determined locations and times based on analysis of tissue motion to perform surgery.

[0079] As described above, cardiac procedures such as mitral valve repair and patent foramen ovale closure can be performed using the surgical system described herein, as long as it is not necessary to continuously track or manipulate moving tissue for an extended period of time. The surgical system described herein reduces the reliance on motion compensation integrity and is able to grasp the target tissue and secure it for further manipulation. Alternatively, the robot can deploy a device attached to the target tissue. The surgical system described herein provides a robot that needs to adapt to the motion of the heartbeat in a short, single time interval.

[0080] The description of the embodiments described herein is intended to provide a general understanding of the structure of various embodiments. These descriptions are not intended to be used as a complete description of all elements and characteristics of the present disclosure described herein. After reading this disclosure, many other embodiments may be apparent to those skilled in the art. Other embodiments may be utilized and derived from this disclosure so that structural and logical replacements and changes can be made without departing from the scope of this disclosure. In addition, these descriptions are merely representative and may not be drawn to scale. Certain proportions in the figures may be amplified, while other proportions may be minimized. Therefore, this disclosure and the accompanying drawings should be considered to be illustrative and not restrictive.

[0081] In this document, for convenience only, the term "invention" may be used to refer individually and / or collectively to one or more embodiments of the present disclosure, without any intention to limit the scope of this application to any particular invention or inventive concept. In addition, although specific embodiments have been illustrated and described herein, it should be understood that any subsequent arrangements designed to achieve the same or similar purposes can be replaced with respect to the specific embodiments shown. This disclosure is intended to cover any and all subsequent adjustments or variations of the various embodiments. After reading the specification, combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art.

[0082] The Abstract of the Disclosure is provided to comply with 37 CFR §1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the foregoing Detailed Description, various features may be grouped together or described in a single embodiment for the purpose of streamlining the disclosure. This disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as reflected in the claims, the inventive subject matter may be directed to fewer than all features of any one of the disclosed embodiments. Accordingly, the claims are incorporated into the Detailed Description, with each claim independently defining separately claimed subject matter.

[0083] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. As such, the subject matter disclosed above should be considered illustrative rather than restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the present disclosure. Therefore, to the maximum extent permitted by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the claims and their equivalents, and should not be limited by the foregoing detailed description.

Claims

1. A controller for controlling a robot tool, comprising: a memory for storing instructions; as well as a processor that executes said instructions, Wherein, when executed by the processor, the instructions cause the controller to perform the following process, the process comprising: monitoring sequential movement of tissue in three-dimensional space based on sensory signals of tissue movement; planning a position and planning a corresponding time when the tissue will be at the planned position in the three-dimensional space; identifying an identified position of the tissue in the three-dimensional space based on the planned position; and setting a trajectory of the robotic tool to encounter the tissue at the identified location at a planned time corresponding to the identified location; A control signal is sent to the robot tool (207) according to the set trajectory.

2. The controller according to claim 1, in, The tissue is a heart, and the sequential motion of the tissue is at least one of: motion of a heartbeat, motion of a breathing motion, and motion of a feedback motion caused by the robot moving in response to the sequential motion of the tissue.

3. The controller according to claim 1, wherein: The process also includes: determining the organization based on input from an operator, selecting the identified location and the planned time corresponding to the identified location from a set of feasible locations and planned times corresponding to the feasible locations, Wherein, the monitoring, the planning, the identifying and the setting are automatically performed based on the determination.

4. The controller according to claim 1, wherein: The process also includes: matching the sequential motion to a pattern in a set of predetermined patterns of sequential motion; Planning locations and corresponding times based on matching patterns; and The robotic tool is retracted when the tissue is not at the identified location at the planning time.

5. The controller according to claim 1, in, The sequential motion includes multiple rhythmic motions overlapping in different directions at the same time.

6. The controller according to claim 1, wherein: The process also includes: acquiring a sequence of images including said tissue to monitor said sequential movement of said tissue; Wherein, the image is a 3D image.

7. A system for controlling a robotic tool, comprising: a memory for storing instructions; as well as a processor that executes said instructions, When executed by the processor, the instructions cause the system to perform the following process, which includes: Monitor sequential movement of tissue in three-dimensional space; planning a position and planning a corresponding time when the tissue will be at the planned position in the three-dimensional space; identifying an identified position of the tissue in the three-dimensional space based on the planned position; and A trajectory of the robotic tool is set to encounter the tissue at the identified location at a planned time corresponding to the identified location.

8. The system according to claim 7, wherein: The process also includes: acquiring a sequence of images including the tissue to monitor the sequential movement of the tissue, and Wherein, the image is a 3D image.

9. A computer readable medium storing instructions which, when executed by a processor, cause the processor to perform a method for controlling a robotic tool, the method comprising: monitoring sequential movement of tissue in three-dimensional space based on sensory signals of tissue movement; planning, by a tangible and non-transient processor, positions and corresponding times when the tissue will be at the planned positions in the three-dimensional space; identifying an identified location of the tissue in the three-dimensional space based on the planned location and at least one characteristic of the robotic tool; setting a trajectory of the robotic tool to encounter the tissue at the identified location at a planned time corresponding to the identified location; A control signal is sent to the robot tool according to the set trajectory.

10. The computer-readable medium of claim 9, the method further comprising: acquiring a sequence of images including said tissue to monitor said sequential movement of said tissue, determining the tissue to be monitored based on the sequence of images, and The trajectory is simulated as a display to confirm or readjust the tissue to be monitored.

11. The computer-readable medium of claim 9, the method further comprising: The robotic tool is guided to a pre-deployment position and then moved along the trajectory to encounter the tissue at the identified position at the planned time starting from the pre-deployment position.

12. The computer-readable medium of claim 9, the method further comprising: matching the sequential motion to a known motion model from a predetermined set of known motion models, and The identified location is identified from a plurality of feasible locations where the robotic tool or a medical treatment device attached to the robotic tool will encounter the tissue.

13. The computer-readable medium of claim 9, the method further comprising: in a teleoperated deployment, moving the robotic tool toward the tissue based on input from an operator and automatically constraining the motion of the robotic tool based on the monitoring of the sequential motion of the tissue, or In autonomous deployment, the robotic tool is moved toward the tissue based on determining the tissue to be monitored, and deployment of the robotic tool is autonomously run by optimizing timing and trajectory.

14. The computer-readable medium of claim 9, the method further comprising: A characteristic is identified from a set of possible characteristics for an optimal location for identifying the identified location where the robotic tool or a medical treatment device attached to the robotic tool will encounter the tissue.

15. The computer readable medium of claim 9, in, The identified location of the tissue is identified based on at least one of range sensing, pressure sensing, force sensing, electrical sensing, optical sensing, or imaging.

Images