Endoluminal robotic (ELR) systems and methods

Abstract

The intracavitary robotic system and corresponding methods include subsystems for visualization, navigation, pressure sensing, platform compatibility, and user interface. The user interface can be implemented using one or more of the following: console, haptic feedback, image fusion, voice control, remote support, and multi-system control.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 17 / 396,759, filed August 8, 2021, which claims the benefit of U.S. Provisional Application No. 63 / 064,938, filed August 13, 2020. Technical Field

[0003] The technology generally relates to an intracavitary robotic (ELR) system, which includes subsystems for visualization, navigation, pressure sensing, platform compatibility, and user interface. These subsystems can be implemented by one or more of the following: console, haptic, image fusion, voice control, remote support, and multi-system control. Background Technology

[0004] Medical procedures such as endoscopy (e.g., bronchoscopy) involve accessing and visualizing the interior of a patient's network of lumens (e.g., the airway). During the procedure, an endoscope can be inserted into the patient. Another tool or instrument can be passed through the endoscope to reach target tissue. During such procedures, a physician and / or computer system navigates the medical tool or instrument through the patient's network of lumens to diagnose and treat target tissue. Medical robotic systems can be used to insert and / or manipulate endoscopes and tools or instruments. Robotically enabled medical systems can be used to perform a variety of medical procedures, including both minimally invasive procedures (such as laparoscopic surgery) and non-invasive procedures (such as endoscopic surgery). In endoscopic surgery, robotically enabled medical systems can be used to perform bronchoscopy, ureteroscopy, gastroenterology, etc. Summary of the Invention

[0005] The technology disclosed herein generally relates to intracavitary robotic systems and methods.

[0006] In one aspect, this disclosure provides an endovascular robotic system for performing suturing procedures. The endovascular robotic system includes at least one robotic arm and an endoscopic tool removably coupled to the robotic arm. The endoscopic tool includes an imaging device coupled to a distal end portion of the endoscopic tool. The endovascular robotic system also includes a needle actuator tool removably coupled to the at least one robotic arm and a grasping tool removably coupled to the at least one robotic arm. The endovascular robotic system further includes a processor and a memory storing instructions thereon, which, when executed by the processor, cause the processor to: receive an image from the imaging device; overlay a suture needle path onto the received image; and control the at least one robotic arm to operate the needle actuator tool to drive the suture needle based on the overlaid suture needle path.

[0007] The intracavitary robotic system may include force sensors coupled to a grasping tool. When executed by a processor, the instruction causes the processor to determine the force applied to the tissue by the grasping tool based on force measurement data output from the force sensors, and to generate an alarm in response to determining that the force applied to the tissue is greater than a predetermined force. The force sensor may include force sensors distributed in an array. When executed by a processor, the instruction causes the processor to: determine the tissue type based on processing the force measurement data output from the force sensors using a machine learning-based algorithm; receive grasping tool type information; and determine a predetermined force based on the tissue type and grasping tool type information. When executed by a processor, the instruction causes the processor to: determine the tissue type; acquire grasping tool type information; and set the maximum force applied by the grasping tool based on the tissue type and grasping tool type.

[0008] The suture needle path may include at least one of a needle entry mark or a needle exit mark. The intracavitary robotic system may include: an electromagnetic (EM) field generator configured to generate an EM field; and at least one EM sensor coupled to the suture needle. When executed by a processor, the instruction causes the processor to track the position of the suture needle based on the EM field sensed by the at least one EM sensor.

[0009] When executed by a processor, this instruction causes the processor to detect suture needle slippage and, in response to the detection of suture needle slippage, to control at least one robotic arm to manipulate an endoscopic tool and / or a needle actuator tool to take the detected slippage into account. Detecting slippage may include detecting movement of the suture needle relative to the needle actuator tool. When executed by a processor, this instruction causes the processor to adjust the suture needle path based on the location of the detected slippage and to overlay the adjusted suture needle path onto the received image. Controlling at least one robotic arm may include controlling at least one robotic arm to manipulate the endoscopic tool and the needle actuator tool based on the adjusted suture needle path overlaid on the received image.

[0010] The intracavitary robotic system may include a user controller. When executed by a processor, the instructions may cause the processor to provide tactile feedback to the user controller in response to the detection of needle slippage. The instructions may also cause the processor to generate vibrations in the user controller in response to the detection of needle slippage. The intracavitary robotic system may include a pressure sensor. When executed by a processor, the instructions may cause the processor to generate tissue tension data based on measurement data output from the pressure sensor, and to predict needle slippage based on that tissue tension data.

[0011] When executed by the processor, this instruction causes the processor to: determine the current position and orientation of the suture needle; determine that the suture needle is near the tissue; and, in response to determining that the suture needle is near the tissue, overlay a marker showing where the suture needle will exit the tissue onto the received image based on the current position and orientation of the suture needle. When executed by the processor, this instruction also causes the processor to overlay a marker showing the planned location where the suture needle will exit the tissue onto the received image.

[0012] When executed by the processor, this instruction causes the processor to display critical structures to one side or behind the suture location on the received image. When executed by the processor, this instruction causes the processor to display at least one of the suture needle entry position, orientation, or depth to avoid approaching critical structures. When executed by the processor, this instruction causes the processor to determine and display the amount of tissue resistance to the movement of the suture needle.

[0013] In another aspect, this disclosure provides an endoscopic robotic system. The endoscopic robotic system includes an endoscopic tool with an imaging device coupled to a distal end portion of the endoscopic tool. The endoscopic robotic system also includes a needle-driving tool. The endoscopic robotic system further includes a processor and a memory storing instructions thereon that, when executed by the processor, cause the processor to: receive an image from the imaging device; overlay a suture needle path onto the received image; and control the needle-driving tool to drive a suture needle based on the suture needle path overlaid on the received image.

[0014] In another aspect, this disclosure provides a method. The method includes: receiving an image from a camera mounted on an endoscopic tool; overlaying a suture needle path onto the received image; and controlling a robotic manipulator tool to drive a suture needle based on the suture needle path overlaid on the received image. The method further includes: detecting suture needle slippage; adjusting the suture needle path based on the detected slippage; overlaying the adjusted suture needle path onto the received image; and controlling the robotic manipulator tool based on the adjusted suture needle path overlaid on the received image. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating an example of an ELR system that includes a "shed" of medical tools.

[0016] Figure 2 This is a schematic diagram illustrating the 3D tubes in the surgeon's console.

[0017] Figure 3 It is a perspective view of a variable focal length camera, illustrating an example of a camera that can be incorporated into the ELR system of this disclosure.

[0018] Figure 4 It is a view that includes multiple images, showing the probe used to obtain the white light image and the white light image overlaid on the ultrasound image.

[0019] Figures 5A to 5C These are multiple images depicting near-infrared (NIR) imaging using indocyanine green (ICG) labeling.

[0020] Figure 6 It is a perspective view of the ELR subsystem used to hold an organization in place while manipulating that organization.

[0021] Figure 7 This is a block diagram illustrating a robotic surgical system.

[0022] Figure 8 It is used for drawing control Figure 7 System block diagram of the robotic surgical control system of the robotic surgical system.

[0023] Figure 9 Is with Figure 7 A perspective view of the distal portion of a catheter assembly used in a robotic surgical system.

[0024] Figure 10 This is a flowchart illustrating a method for performing intracavitary robotic suturing surgery. Detailed Implementation

[0025] For ELR systems, the placement, positioning, and / or structural support of equipment can be challenging. The operating table presents opportunities to facilitate ELR procedures through added functionality. The systems and methods of this disclosure can incorporate enhancements related to the operating table (OR), where the OR is part of a robot. For example, the robotic arm and instruments including tools can be fixed, and the boom support can be detached from the table. In various aspects, a C-arm can be integrated into the OR. In various aspects, the OR may include a swing arm for allowing various techniques to enter or leave the work area. This allows OR personnel to approach the patient as needed during the procedure. The robotic arm can be temporarily positioned to be unobstructed, rather than being disengaged to approach the patient. These features can provide additional benefits to surgeons, staff, central processing, and hospitals because they can save time, improve workflow, and make working in the OR easier.

[0026] In all aspects, such as Figure 1The illustration shows that the robotic arm and instruments can be stored under the table without obstructing the view. They can also be deployed when needed. If not in use, the robotic arm and instruments can be stored in this location to free up space in the OR (Operating Room). In various ways, the OR table may include an integrated autoclave or cleaning solution system. The robotic arm and / or instruments can be returned to the "shed" under the table after surgery and undergo an autoclaving process, such as a car wash-like process, or a cleaning solution spraying system to clean or disinfect the instruments and / or robotic arm.

[0027] In various respects, the OR table enables automated attachment of instruments within a sterile area. The OR table may include a separate “shed” containing the instruments, which can be located adjacent to the surgical area (e.g., below the table) where the robotic arm can retract to change instruments. This can be accomplished by storing certain instruments before the case or surgery and registering their positions in the “shed.” During the procedure, the surgeon can then select different instruments, and the robotic arm can automatically remove the old instruments and attach new, desired instruments.

[0028] The systems and methods disclosed herein can be combined with automated navigation. Careful manual navigation with instruments can be time-consuming and inherently risky to the patient if the surgeon does not perform the procedure correctly. This is particularly challenging when navigating instruments through tubular anatomy using an ELR system. Automated navigation features can allow for easier use of the ELR system, provide enhanced functionality for greater patient safety, or save time on OR (oral retraction).

[0029] The ELR system and method disclosed herein enable automatic catheter advance, braking, or retraction. The ELR system can store position information in memory for re-motion. This is accomplished by storing position and motion information in the system memory. This allows for an "undo" or retraction to a specific area stored in the system memory without manual control.

[0030] The ELR system can use computed tomography (CT) scans for navigation functionality. The ELR system can use CT scans of the patient's anatomy to automatically deliver tools, treatments, medications, etc., to specific areas of interest.

[0031] ELR systems can provide image stabilization, allowing them to lock onto the target. For example, ELR systems can compensate for tissue or instrument movement.

[0032] An ELR system can provide features related to the control of an instrument that is out of or moving out of sight. For example, when the instrument is out of or moving out of sight, the ELR system can lock the instrument in place, or if the instrument is moving, it can issue an alarm when the instrument is out of sight.

[0033] ELR systems may include improvements to the surgeon's console for non-virtual reality (VR) or non-augmented reality (AR) implementations. In various aspects, the surgeon's console may be located on a rotating component (e.g., a 360-degree rotating component) that rotates around the patient according to the surgical procedure or the desired position of the surgeon or other clinician. The ELR system may include head sensors that allow the screen to move automatically relative to the head sensor's position.

[0034] ELR systems and methods can combine one or more of a variety of control features. Control features may include mapping images (e.g., images of the endoscopic field of view) onto a physical three-dimensional (3D) tube in the surgeon's console to allow the surgeon to touch the console to control specific locations. Specifically, such as Figure 2 The illustration shows a surgeon's console that can be a 3D tube in which the surgeon stands. Instead of displaying images on a 2D screen, the images are mapped onto the interior of the 3D tube. This mapping can involve magnifying the image, making the surgeon feel as if they are physically located inside the tubular structure in the image. The inner surface of the 3D tube can be a touchscreen, allowing the surgeon to control the movement of a camera, tools, or virtual markers on the touchscreen. The surgeon can draw virtual boundaries or mark boundaries with energy.

[0035] Control features may include a mouse or touchpad for a more intuitive interface. These features may allow pinch and zoom gestures similar to those found in conventional laptops, allow left or right clicks for selection, and / or provide drop-down menu features. Control features may allow clinicians to move fiber optic components closer to the target to provide optical zoom functionality, or to crop and magnify the image to provide digital zoom functionality. ELR systems may include touchpads to control features of the ELR system. For example, a touchpad can be used for manual control.

[0036] The ELR system and method disclosed herein can use virtual reality (VR) and augmented reality (AR) systems to improve the surgeon's console for endoscopic robotic surgery. The ELR system may include a VR headset, with audio and visual functionality integrated into the VR headset along with a portable control station. The VR headset may include a microphone for more intuitive control and more visualization or control options. For example, a clinician wearing the VR headset can speak into it to control one or more features, functions, or subsystems of the ELR system.

[0037] The ELR system can provide voice-activated actions. These actions can include features or functionalities provided by Siri, Alexa, or Google Home. The ELR system can provide a channel for communicating with teammates. This channel can be specific to the person being communicated. The voice activation action can respond to keywords such as "start" and "stop." Furthermore, keywords can be specific to a particular device, such as "Alexa, moving camera."

[0038] ELR systems can provide motion detection via, for example, a bodysuit or control over specific body parts. Sensor interaction may exist; for example, a surgeon's finger ring needs to be in a specific position to allow access to or use of the ELR system's features. The speed of movement can add additional control, such as waving an iPhone to delete a message.

[0039] VR or AR systems can be used to guide the integration of specific images or to direct which images are displayed on a screen. VR or AR systems can monitor eye movements to detect safety or additional commands. VR or AR systems can monitor eye movements to automatically display options for a given area.

[0040] VR or AR systems can provide directional sound as alarms. They can provide tones from different locations to convey different alarms. For example, a high cannula torque alarm could be a specific tone coming from behind a clinician. VR or AR systems can also provide proximity marker alarms based on camera movement or instrument movement.

[0041] VR or AR systems can perform surgical monitoring. For example, a VR or AR system can monitor a surgeon's alertness, heart rate, fatigue level, blink rate, or posture, and alert the surgeon when necessary based on this monitoring. VR or AR systems can monitor and provide features or functions related to ergonomics or posture correction.

[0042] ELR visualization features may include scaling functionality to improve white light image quality and usability. Tubular images can be mapped at a desired magnification (e.g., 10X) to allow surgeons selective, fine anatomical control. The catheter can be moved axially closer to or further away from the lesion to correspondingly magnify the closer or wider field of view. ELR systems can use lensless camera systems or variable focal length camera systems, such as... Figure 3 The image shows a camera system that can use ultrasonic distorting lenses made of soft materials.

[0043] The ELR system and method disclosed herein can incorporate various tissue sensing or tactile features. In various respects, blood perfusion or tissue type can be determined based on oxygen saturation measurements. Optical or physical sensors can be used to detect motion (e.g., ureteral peristalsis, arterial pulsation).

[0044] ELR systems and methods can measure compressed or tensile tissue points. The movement of compressed or tensile tissue points reveals tissue elasticity. Tissue compression or tension can be measured using pressure sensors or strain gauges. The measured tissue compression or tension can be used to calculate suture pressure, dilation of stenosis in structures such as the cervix or intestine, tissue perfusion, or tensile load on tissue during grasping, pinning, or sealing. The measured tissue compression or tension can be used to calculate grasping force. When the grasping force is too high, the ELR system can transmit an alarm to the surgeon. The pressure sensors can be an array of pressure sensors arranged circumferentially. Alternatively, a single pressure sensor can be used. A pressure-measuring balloon can be used to expand tissue or lumen to sense tissue tension and / or tissue movement such as during peristalsis. When a threshold of tissue compression, tension, or movement is exceeded, the ELR system can issue a vibration alarm to the surgeon.

[0045] ELR systems and methods may include impedance monitoring of tissue (e.g., retroperitoneal structural, density, or fibrotic tissue scales). An ELR system may include a lasso that may include circumferentially spaced electrodes to measure tissue impedance along the wall, thereby understanding tissue properties such as tissue type, density, fibrotic nature, or the amount of surrounding tissue present.

[0046] Tissue sensing or tactile features of ELR systems and methods can be based on force or vibration. Tissue sensing features may include a gripper force sensor to detect tissue fragility. Vibration can alert the surgeon to the strength of their grip (i.e., soft versus hard in terms of strength). For example, vibration can alert the surgeon when their grip is too hard. Clinicians can select the tissue type, and the ELR system can set the maximum gripping strength to be used based on the selected tissue type and instrument type. The ELR system can determine the instrument type by reading a radio frequency identification (RFID) tag attached to the instrument. The ELR system can use machine learning-based algorithms to determine the tissue type. Alarms or warnings can be set based on the instrument type. Pressure sensors can be positioned along the length of the catheter or integrated into the instrument or camera. Pressure sensors can be used to monitor dilation, which can drive incremental dilation of the cervix or dilation of stenotic bowel stenosis. Pressure sensors can be used to determine the tensile load when pulled on tissue, for example, in end-to-end colorectal anastomosis (EEA) or ureteral reattachment. Pressure sensors can be used to measure suture pressure.

[0047] An ELR system can trigger a vibration alarm based on proximity to an object. Tools or instruments may include sensors for detecting proximity to a location in 3D space. The location in 3D space can be manually marked for future detection of proximity to that location. An ELR system can use distributed pressure sensors, such as a dense array of pressure sensors (e.g., 16x32). Distributed pressure sensors can be pressed against tissue to locate hard tissue or pulsating flow. Image processing or machine learning can be used to detect objects.

[0048] ELR systems can integrate different imaging modalities to provide useful real-time imaging to assist surgeons. ELR methods may include performing elastography using ultrasound (US) data, CT data, MRI data, or using intraluminal stiffness data probes to understand tissue properties along the wall. ELR systems and methods can, for example, display both white light and CT images by overlaying one image onto another. To improve the quality or usefulness of non-white light images, near-infrared (NIR) or ultrasound (US) images can be overlaid or displayed with white light images, such as... Figure 4 The illustration is shown in the image. This can be extended to preoperative or perioperative imaging, including, for example, CT, MRI, US, elastography, X-ray, laser coding, or hyperspectral imaging. The lumen or intraluminal balloon can be filled with saline solution, air, contrast agents, dyes, or any fluid suitable for enhancing the results of the imaging modality. Fluids, saline solutions, air, and plugs or balloons can be used to dilate tissue to improve intraluminal US visibility. Contrast materials or dyes (e.g., methylene blue, indocyanine green (ICG)) or gases (such as xenon) can be infused or injected into the intraluminal tissue or fluid.

[0049] ELR systems and methods can use laser-encoded or laser spectral imaging for white light imaging because it is smaller and provides high depth of field. An ELR system may include an X-ray probe or capsule. For example, an X-ray probe may include a source and receiver at the tip of the probe and can use reflectivity to perform imaging. An ELR system can use laser-driven ultrasound (US) without contact and without using photoacoustic effects. An ELR system can be configured to perform hyperspectral imaging. An ELR system can acquire color gradient information from white light imaging and convert this color gradient information into depth-sensing information in a manner similar to how a nurse examines a vein before making an intravenous (IV) connection.

[0050] Endocaval suturing can be challenging and may involve associated risks. The ELR system and method disclosed herein address these potential challenges and risks by projecting the needle path and alerting the surgeon via tactile feedback in the presence of needle slippage. The ELR system and method can project the needle path to assist the surgeon, for example, in manipulating the needle or performing suturing. The needle entry or exit point can be projected to guide the surgeon in adjustments made as tissue moves during the procedure (e.g., when grasping tissue). For example, during suturing, needle projection can guide the needle's position to the desired location. Electromagnetic (EM) tracking can be used to track the needle in real time in a manner similar to catheter navigation.

[0051] In all respects, the ELR system and method can account for needle slippage in real time and adjust the needle trajectory accordingly. The position and orientation of the needle in the actuator tool determine where the needle will pass through the patient's anatomy. Poorly aimed needles may fail to produce effective sutures and may pass through critical structures or trap objects that should not be trapped in the suture, potentially leading to future patient complications. The ELR method disclosed herein may include 3D imaging of the suture needle and actuator tool, which simplifies and / or provides additional methods for slippage detection. Optical detection of needle slippage can be performed using one or more imaging devices (e.g., white light cameras) and image processing software. The goal of the image processing software is to detect the independent movement of the suture needle or actuator tool. When no slippage occurs, the suture needle and actuator tool are expected to move as a unit. Additionally, the suture needle and actuator tool should move with the same angular motion. Using image processing software with object recognition, such as convolutional neural networks (CNNs), the actuator tool and suture needle can be detected in each frame of a video. Points on the tool and needle are selected for motion measurement. When no slippage occurs, all points should move in the same direction and at the same speed.

[0052] Providing an additional camera allows for increased detail and detection of slippage. Tissue tension data, obtainable through imaging, machine learning, or other suitable tissue sensing techniques, can be used for prediction, alerting, and / or compensation of needle slippage. The ELR system can provide tactile feedback, such as resistance, in response to detected needle slippage to guide the surgeon along the desired path while still allowing the surgeon to maintain control. The ELR system can provide vibration upon detecting needle slippage.

[0053] Visualization and tracking during endovascular surgery can be challenging. The ELR system and method of this disclosure utilize indocyanine green (ICG) for high-resolution imaging to improve visualization and tracking. According to aspects of this disclosure, markers such as ICG (e.g., Figures 5A to 5CNear-infrared (NIR) imaging of cancer detection markers (such as those labeled with Surgilab) or similar markers, combined with white light imaging, can be used to guide ELR procedures. Electromagnetic tracking can also be used to better guide endovascular procedures. The ELR system and method disclosed herein can use NIR imaging in conjunction with white light imaging during ELR to: track the direction of blood flow, combine laparoscopic and endovascular tracking of the ELR catheter, and perform chip-based NIR for cancer detection using markers such as Surgilab molecules (TR).

[0054] The ELR system and method disclosed herein can be combined with ELR catheter navigation including electromagnetic (EM) tracking and retraction. Catheter navigation may include 3D tracking of the device relative to a specific point (e.g., a tracker on the catheter). Electromagnetic sensing can be used to track the tip path of the ELR catheter or device. Electromagnetic sensing can be performed by a subsystem using a cover or sheet having transmitters at different locations and receivers or sensors positioned at the tip of the ELR catheter or device. Based on the signals received by the sensors, the tip of the ELR catheter or device can be tracked relative to the patient's bed.

[0055] Using information about the pathway and optionally preoperative or perioperative imaging with or without machine learning, the ELR system can predict and automatically advance the catheter forward or retract the ELR catheter or instrument backward. ELR control can resemble the accelerator pedal, brake pedal, or reverse gear in a typical car.

[0056] Anatomical sections can be marked with temporary metal clips for tracking functions, such as tracking catheters in the ureter. Specific distances or boundaries can be detected by appropriate sensors. The ELR system can issue an alert when the robotic arm or instrument is near a marker associated with a critical structure.

[0057] The ELR system and method disclosed herein can use transient images or videos to reconstruct 3D images or models of anatomical structures. Imaging from one or more imaging modalities can be used to reconstruct 3D anatomical structures and track ELR catheters or instruments to aid surgeons in visualizing real-time navigation. Virtual colonoscopy or bronchoscopy images can be captured, and computer vision can be used to reconstruct 3D models of anatomical structures (e.g., organs). The reconstructed 3D model can correspond to what the clinician sees in the camera. The reconstructed 3D model can be updated as the procedure progresses. In various respects, CT or fluorescence imaging can be used for 3D reconstruction to see where the ELR catheter, tool, or instrument is located and / or as overlay for real-time guidance.

[0058] ELR systems and methods can use multiple camera views stitched together to provide a wider variety of views.

[0059] The ELR system and method disclosed herein can combine the visualization features and functions of various types of cameras at various locations to improve the quality or usefulness of white light images. Cameras can be placed at the front tip, side, and / or back of the ELR catheter or device facing the ELR catheter or device to capture more views. Views can be reconstructed to obtain better field of view or to obtain a 3D map with depth perception. Cameras can be attached to the tip of the catheter or device. Cameras can be pop-out or located on the side of the catheter or device. Cameras can be implemented using a chip-on-tip, which can incorporate near-infrared (NIR) imaging capabilities; fiber optic components; or any other lens system suitable for placement on the tip of the catheter or device. Multiple lenses can be built into the ELR system using microelectromechanical systems (MEMS) technology, and these lenses can be turned on or off to obtain different fields of view.

[0060] An ELR system may include a capsule endoscope device, such as PillCam, which uses a front camera and a rear camera to view forward and backward, respectively. These two cameras can provide proximal and distal views of the pathology. This can be used for retrograde resection of the pathology. Alternatively, the front and rear cameras can be coupled to an endoscope or catheter to obtain images including both anterior and posterior views. Images including the views from both cameras can be obtained during preoperative endoscopic examination. Then, images including the views from both cameras can be obtained at a later time (e.g., during surgery), and these images can be compared with the images including the views from the preoperative endoscopic examination. In one aspect, images from two intraoperative camera views can be overlaid on two preoperative camera views and displayed to the user on a monitor.

[0061] In various aspects, images from the capsule endoscope device can be used to reconstruct 3D models. Landmarks can be identified, and frame-by-frame stitching can be performed. This stitching can include progressive views captured by optical sensors as the capsule endoscope device passes by. Figure 1 The images are spliced ​​together to form a continuous view of the anatomical structure.

[0062] During surgery, magnets can be placed outside the patient's body to fix a camera or LED along an anatomical wall (e.g., the abdominal wall). LEDs can have different wavelengths to provide distinguishable visibility.

[0063] A camera can provide an orthogonal side view with sutures. In various ways, two imaging devices (e.g., cameras) attached to the endoscope can map a wider field of view. For example, one imaging device can be attached to the side of the endoscope. A 2D view, a 3D view, or some other adaptation can be presented. ELR systems and methods can include those similar to Google Street View. Figure 1 Characteristics of the same.

[0064] An ELR system may include different conduits. These different conduits may include a conduit for white light and a conduit for the camera. The camera can be placed on each instrument in, for example, a multi-articulated system. The lamp's structure, time of flight, and stereoscopic view can be used for 3D mapping.

[0065] The ELR system can be combined with a microendoscopy. This can be used for biopsies. For example, all or part of the microendoscopy can be inserted into or otherwise integrated into a biopsy needle.

[0066] An ELR system may include multiple cameras that are split and connected to a common fiber optic channel of different sensors at the distal end portion.

[0067] In various applications, aspiration can be used to pull tissue or provide tissue tension. Aspiration can draw tissue into an ELR system to isolate the tissue or organ of interest. This feature can be used in conjunction with a microendoscopy. ELR systems may include aspiration channels that draw tissue into the tip before performing tasks (such as suturing) using other arms (in multi-arm articulated ELR systems for large-bore applications).

[0068] The ELR system and method disclosed herein can use, for example, pneumatic pressure or inflatable air to stabilize the camera, visibility, and / or instruments. Local pressure can facilitate the advancement of the catheter through the lumen. The ELR system may include a balloon catheter that uses a balloon to move or articulate the catheter. Proximal and distal balloons can be used to expand the tissue area. The ELR system and method can inflate between the proximal and distal balloons to open the lumen.

[0069] The distal tip of the catheter may not necessarily be used to provide stability. One or more instruments may emerge from the side of the catheter at a more proximal location, and stability can be provided in this configuration. Balloons proximal and / or distal to the catheter or instrument opening can be used to provide stability. ELR systems and methods can provide image stabilization functionality. Vibrations at the tip of the ELR catheter or instrument can be sensed by a piezoelectric sensor or by image motion detection. Image motion can be compensated for by adding a mechanical damper to the tip of the ELR catheter or instrument or by video processing such as smoothing. ELR systems can provide the ability to lock the camera or instrument in a desired location. This is especially applicable to catheters or instruments outside the field of view. If the camera or instrument is forced to move, the ELR system can alert the surgeon.

[0070] ELR systems may include a nitinol (or other shape-memory active material whose properties can be altered based on electromagnetic or thermal fields) framework with barbed fixation elements (e.g., barbed scaffold fixation elements) extending from the working channel to maintain tissue tension during tissue removal or manipulation. Figure 6The illustration is shown in the figure. An ELR system may include a rugged endoscope, which may be implemented by a pull-string system that holds the components together. The rugged endoscope may use a material that varies with different electric or thermal fields (e.g., a memory-active material that changes in the presence of a magnetic field, electric field, or thermal field). The rugged endoscope may use a rubber hose-like design that stiffens when filled with fluid (e.g., a saline solution). Once the endoscope reaches the target anatomical structure, the ELR channel can be filled, allowing the endoscope to stiffen for better stability. The endoscope can retract as the fluid exits the ELR channel. The ELR system and method may use magnets externally to stabilize and / or drive the distal tip of the ELR catheter or instrument. The ELR system may include a coiled cannula to open or advance along the lumen (similar to a water pipe toy, snake, or water wigglies) to coil and drive the catheter forward. This can also provide stability.

[0071] ELR systems and methods may include manometry or balloon catheters. A balloon catheter filled with saline solution can be used to obtain pressure mapping. Manometry can be used to detect peristalsis. When peristalsis is detected, the procedure can be paused until the wave has passed. Manometry can be used to detect stenosis or dilation. A balloon catheter can be used to open the lumen. Manometry can be used to check for suture defects. Manometry can be performed along the length of the endoscope. This can be used to aid navigation.

[0072] The ELR system and method disclosed herein can provide platform compatibility. The ELR system can be integrated with robot-assisted surgical (RAS) systems. The ELR system can use RAS system designs and components, in addition to single-port (multi-arm, multi-hinged) systems that lead to the location of the cannula holder.

[0073] In all aspects, the ELR system and approach can be used in combined endoscopic laparoscopic surgery (CELS) applications. The combined endoscopic robotic approach allows for the use of both endoscopic and general-purpose surgical robots. Using the ELR and RAS systems to track the position of all instruments enables critical surgical steps.

[0074] The combined ELR and RAS system allows switching between laparoscopic and endoscopic views. In other words, the combined ELR and RAS approach allows switching between ELR and RAS views. Views can be changed manually, for example, via a toggle switch at a general surgeon's console, or automatically, for example, by tracking stages of the procedure and changing the view based on those stages. For example, the combined system can switch between ELR and RAS views during suturing. In one implementation, when performing inside-to-outside suturing, the needle tip can be tracked and the system can automatically switch views based on the needle tip's position, for example, whether the needle tip is inside or outside. In all aspects, tools can be tracked to perform this functionality whenever ELR and RAS tools need to interact, for example, when performing full-thickness suturing or applying tension to tissue.

[0075] Coordination of instruments during CELS procedures can involve the following characteristics. During CELS, for example, the surgical team can coordinate the use of laparoscopic instruments used in visually-assisted minimally invasive surgery with endoscopic instruments used for polyp and cancer removal. This combination of instruments allows the surgical team to better identify the location of the surgical site while providing alternative options for continuing the procedure based on what has been learned during the examination of the surgical site. For example, the surgeon may recognize that a polyp cannot be removed via colonoscopy due to local anatomy, necessitating laparoscopy to complete the procedure.

[0076] This disclosure describes a method for real-time identification and relaying of the positional orientation between endoscopes and laparoscopic tools. Electronavigation is a method for tracking devices in the body and is routine practice in electronavigated bronchoscopy (ENB). While the patient lies on an electromagnetic field generator, the bronchoscope or biopsy tool may have an electromagnetic sensor attached to its distal tip. Detection of distortion in the field is correlated with the sensor's position in 3D space. By placing such sensors on each tool used in CELS procedures, the positional orientation of each tool can be tracked. Using reference sensors placed in known anatomical locations or using a calibration procedure based on known anatomy, the 3D position can be referenced and overlaid onto the patient's CT or fluoroscopic images to demonstrate its relationship to the anatomy.

[0077] An alternative to electrical navigation is the use of radio signals. In this implementation, the sensor plate is replaced by a phased array antenna, and each surgical instrument sensor is replaced by a low-power transmitter. Each instrument generates a uniquely coded signal specifically for positioning by measuring the phase and angle of the antenna array. The transmitter can be split axially such that the two halves at the distal tip of the instrument produce out-of-phase signals to allow identification of rotational orientation. An additional transmitter closer to the instrument will provide identification of the instrument's angle.

[0078] On the other hand, an infrared (IR) laser source is placed in the distal tip of the endovascular catheter, which protrudes axially into the body of the endovascular catheter. With the distal tip of the catheter pointing towards the desired surgical site, the IR penetration through the colon will be sufficient to allow detection of the IR light on the ventral side of the colonic wall, thus allowing laparoscopic tools with IR detection to identify the IR source and its 'attribution' at that location. This provides the surgeon with directional information for locating the intended surgical site. The diffusion pattern of the IR light within the colonic wall will allow laparoscopic tools to identify the brightest point associated with the center of the IR beam.

[0079] Additionally, methods for detecting the orientation of the distal tip of an endovascular catheter relative to the patient can inform the surgeon of the surgical site's position relative to the patient (e.g., whether the patient is supine or prone, left or right). Methods for detecting the orientation of the distal tip of an endovascular catheter include using two sufficiently spaced electromagnetic sensors to provide different 3D coordinates, or using an inertial measurement unit to detect the distal tip orientation relative to gravity. If rotation and angular information of the laparoscopic tools is available, one or more tools can be presented in the view along with navigation prompts.

[0080] Figure 7 This is a block diagram illustrating a robotic surgical system 700 according to various aspects of the present disclosure. The robotic surgical system 700 includes a first robotic arm 702 and a second robotic arm 704, respectively attached to robotic arm bases 706 and 708. The first robotic arm 702 and the second robotic arm 704 each include a first end effector 716 and a second end effector 718. The end effectors 716 and 718 may include robotic manipulators or grippers adapted to operate endoscopic catheters and tools of the present disclosure. The first end effector 716 operates one or more tools 712, including suture needle driving tools, grasping tools, and / or flexible endoscopes (not shown). The second end effector 718 operates a sheath or catheter 710, which may include one or more channels for receiving and guiding one or more tools 712. The robotic surgical system 700 may also include an electromagnetic (EM) generator 714 configured to generate an EM field sensed by an EM sensor incorporated into or disposed on a suture needle. Alternatively, the EM sensor may be incorporated into or disposed on the distal end portion of the actuator tool. The EM sensor outputs sensor measurement data that can be used to determine the position and orientation of the suture needle. In various respects, the EM generator 714 may be embedded in the operating table 715 or incorporated into a pad that can be placed between the operating table 715 and the patient 711.

[0081] The first and second robotic arms 702 and 704 are controllable to align end effectors 716 and 718 such that the proximal end portion of the conduit 710 is distal to the proximal end portion of one or more tools 712, and the one or more tools 712 are kept axially aligned with the conduit 710.

[0082] In one aspect, a first robotic arm 702 inserts a catheter 710 through, for example, a tracheal tube (not shown) into the mouth of a patient 711 and into the bronchial system of the patient 711. Then, a second robotic arm 704 inserts one or more tools 712 through the catheter 110 to a target within the bronchial system of the patient 711. The first and second robotic arms 702 and 704 can be axially moved relative to each other and moved in and out of the patient 711 under the control of a surgeon (not shown) at a console (not shown).

[0083] The navigation phase may include advancing catheter 710 along with one or more tools 712 into the patient 711, and then advancing one or more tools 712 beyond the distal end of catheter 710 to reach a desired destination, such as a target. Other navigation modes may be used, such as by using a guide line passing through the working channel of catheter 710. Surgeons may use visual guidance modalities or combinations of visual guidance modalities to assist navigation and perform suturing procedures, such as fluorescein microscopy, video, computed tomography (CT), or magnetic resonance imaging (MRI). In various aspects, one or more tools 712 are deployed through the longitudinally aligned working channel within catheter 710 to perform suturing procedures and any other desired surgical procedures. In various aspects, robotic arms 702, 704 include three joints 701 and three arm segments 705. In other aspects, robotic arms 702, 704 may include more or fewer than three joints 701 and three arm segments 705.

[0084] Figure 8 It is used for drawing control Figure 7 A block diagram of the robot control system 800 of the robotic surgical system 700. The robot control system 800 includes a control system 810 that controls the robotic surgical system 700. For example, the control system 810 can perform... Figure 10Method 1000. The control system 810 may interface with a display 822, a user controller 825, an endoscope camera 826, and an air blowing system 828. The control system 810 may be directly or indirectly coupled to the robotic surgical system 700, for example, via wireless communication. The control system 810 includes a processor 812, a memory 814 coupled to the processor 812, a random access memory (RAM) 816 coupled to the processor 812, and a communication interface 818 coupled to the processor 812. The processor 812 may include one or more hardware processors. The computer system 810 may be a stationary computing device (such as a personal computer) or a portable computing device (such as a tablet computer). Alternatively, the control system 810 may be integrated into one of the robotic arm bases 706, 708. The control system 810 may also interface with a user controller 825, which can be used by a surgeon to control the robotic arm system 824 to perform suturing procedures.

[0085] Those skilled in the art will understand that memory 814 can be a computer-readable storage medium accessible by processor 812. That is, a computer-readable storage medium can include non-transitory, volatile and non-volatile, removable and non-removable media implemented using any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, a computer-readable storage medium can include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technologies, CD-ROM, DVD, Blu-ray or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by processor 812.

[0086] An application stored in memory 814 can cause a user interface (not shown) to be displayed on display 822 when executed by processor 812. This user interface can be configured to present endoscopic images from endoscope camera 826 to the user. According to this disclosure, the user interface can also be configured to guide the user in selecting a target, particularly by identifying and marking targets in the displayed fluorescence examination 3D reconstruction or any other fluorescence examination image data.

[0087] The communication interface 818 can be configured to connect to a network, such as a local area network (LAN), wide area network (WAN), wireless mobile network, Bluetooth network, and / or the Internet, consisting of wired and / or wireless networks. The communication interface 818 can be used to connect the control system 810 to the endoscope camera 826. The communication interface 818 can also be used to receive image data and suture path planning data from the memory 814. The control system 810 may also include an input device (not shown), which can be any device that allows a user to interact with the control system 810, such as a mouse, keyboard, foot pedal, touchscreen, and / or voice interface. The control system 810 may also include an output module (not shown), which can include any connection port or bus, such as a parallel port, serial port, universal serial bus (USB), or any other similar connection port known to those skilled in the art.

[0088] Figure 9 It contains various tools. Figure 7 A perspective view of the distal portion of catheter 710. Catheter 710 includes a working channel 905 in which a suture needle driver tool 910, a grasping tool 920, and an endoscope 930 can be disposed. The suture needle driver tool 910 includes jaw members 912 and 914, which can be... Figure 7 The robotic end effector 718 is controlled to transfer the suture needle 915 back and forth between jaw members 912, 914. The gripping tool 920 includes jaw members 922, 924, which can be used with the suture needle 915 to perform suturing procedures, for example, to tie a knot after sutures have been placed. The endoscope 930 includes a camera 932, which can be used to capture video images of the surgical site. The endoscope 930 can be a monocular endoscope, a stereoscopic endoscope, a 3D endoscope, or any other suitable endoscopic camera for capturing clear video images of the defect to be sutured and surrounding tissue. One or more video images captured can be fused with other information to guide the surgeon and / or robotic surgical system in manipulating the actuator tool 910 and gripping tool 920 to perform suturing procedures. Planned needle trajectories, suture needle entry points, and / or critical structures to be avoided can be overlaid on the captured video images.

[0089] Figure 10This is a flowchart illustrating another method for performing robotic suturing surgery. At box 1002, suture needle path and key structural information are received. This key structural information may include or be derived from a three-dimensional preoperative image. The suture needle path can be generated in a path planning application, where the surgeon can mark entry and exit points on a two-dimensional or three-dimensional image showing the defect to be sutured, presented to the surgeon in a suitable user interface. Alternatively, the user interface may allow the user to select the suture spacing or interval and the suture length. At box 1004, images of the defect and surrounding tissue are received from an endoscopic imaging device. In various respects, a camera sensor may be mounted on a actuator tool or gripper. At box 1006, the suture needle path and key structural information are overlaid on the received image. The suture needle path or trajectory may include at least one of needle entry marks or needle exit marks.

[0090] In various aspects, a suture path can be received from a planning user interface that can display an image or representation of the defect and allow the surgeon to draw and / or mark the suture path on the image or representation of the defect. In some aspects, at least a portion of the suture path can be automatically generated. Automatic generation of the suture path may include determining the size of the defect to be sutured based on imaging of the defect (e.g., an image received from an endoscopic camera); receiving parameters of the suture pattern; and generating the suture path based on the determined size of the defect and the received parameters. Parameters of the suture pattern may include the distance between suture loops.

[0091] Key structural information may include a graphical representation of key structures near or adjacent to the defect to be sutured. The received images can be displayed in a user interface. For example, representations of blood vessels and organs can be displayed. The graphical representation of key structures can be displayed in such a way that defects in tubular anatomical structures (e.g., upper or lower gastrointestinal (GI) tracts), entry marks, suture needle paths, and suture needle representations are visible to the surgeon or other clinician. For example, the graphical representation of key structures can be displayed such that they are translucent or ghosted. In some aspects, if the distance to the key structure is less than a predetermined distance, the control system can enter a novice mode, which prevents further movement of the robotic arm. The system can also provide a way to manually override the novice mode, such as by activating physical buttons or switches.

[0092] At box 1008, the current position and orientation of the suture needle are determined. The current position and orientation of the suture needle may be determined based on an electromagnetic (EM) field sensed by at least one EM sensor incorporated into or disposed on the suture needle or actuator tool. If at least one EM sensor is incorporated into or disposed on the actuator tool, the position and orientation of the suture needle may be determined by: controlling the actuator tool to hold the suture needle in a predetermined position and predetermined orientation relative to the actuator tool; and calculating the position and orientation of the suture needle based on position and orientation information from the EM sensor and a predetermined geometric relationship between the actuator tool and the suture needle. Alternatively, the current position and orientation of the suture needle may also be determined based on 3D endoscopic images or ultrasound images. AI algorithms, such as image recognition algorithms, may be employed to determine the current position and orientation of the suture needle. The AI ​​algorithm may include a predictive algorithm that predicts the future position and orientation of the suture needle based on previous and current 3D endoscopic images or ultrasound images. The future position and orientation information may be used to determine and display where the suture needle will exit the tissue after passing through it.

[0093] At box 1009, method 1000 determines whether the suture needle is near the tissue. If it is determined that the suture needle is not near the tissue, boxes 1004 through 1008 are repeated. If it is determined that the suture needle is near the tissue, at box 1010, the location where the suture needle will exit the tissue is predicted based on the current position and orientation of the suture needle. At box 1012, based on the current position and orientation of the suture needle, the predicted location of the exit mark, showing where the suture needle will exit the tissue, is overlaid on the received endoscopic image.

[0094] At box 1014, at least one robotic arm is controlled to operate a suture needle driver tool to drive the suture needle based on the suture needle path overlaid on the received image and the current position and orientation of the suture needle. In various aspects, the at least one robotic arm may include a robotic end effector coupled to the suture needle driver tool.

[0095] At box 1015, method 1000 determines whether needle slippage has been detected. Needle slippage can be detected by detecting movement of the needle relative to the jaw members of the needle driver tool or the needle driver tool currently holding the needle. In various aspects, needle slippage can be detected or predicted and then compensated for using tissue tension data, which can be obtained from imaging, machine learning, pressure balloons used to expand tissue to measure tension, pressure sensors, and / or strain gauges. Pressure sensors or strain gauges can include single sensors incorporated into or disposed on a probe or catheter, or arrays of sensors arranged circumferentially around the probe or catheter. Pressure sensors can also be used to determine suture pressure. In other aspects, optical sensors or force sensors can be used to detect needle slippage.

[0096] In various aspects, the needle-driving tool may include an ultrasonic transducer. Instructions, when executed by a processor, may cause the processor to display the position of the suture needle relative to the tissue based on data output from the ultrasonic transducer. The needle-driving tool may include an ultrasonic transducer. Instructions, when executed by a processor, may cause the processor to determine the distance between the suture needle and a critical structure near the suture needle based on data output from the ultrasonic transducer, and to display a message on a display indicating the distance between the suture needle and the critical structure near the suture needle based on the data output from the ultrasonic transducer.

[0097] The needle-driving tool may include a sensor configured to sense pulling resistance on the suture when it is pulled. Instructions, when executed by a processor, may cause the processor to determine that the pulling force is greater than a threshold, and in response to determining that the pulling force is greater than the threshold, to reduce the pulling force. The needle-driving tool may also include a sensor configured to sense the pulling force on the suture when it is knotted. Instructions, when executed by a processor, may cause the processor to control the needle-driving tool and the gripping tool to knot with a predetermined pulling force based on the sensed pulling force.

[0098] It should be understood that the various aspects disclosed herein can be combined in combinations different from those specifically given in the specification and drawings. It should also be understood that, depending on the example, certain actions or events of any process or method described herein may be performed in a different order, or may be completely added, combined, or omitted (e.g., performing the technique may not require all the described actions or events). Furthermore, although for clarity some aspects of this disclosure are described as being performed by a single module or unit, it should be understood that the techniques of this disclosure can be performed by combinations of units or modules associated with, for example, a medical device.

[0099] In one or more examples, the described techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include non-transitory computer-readable media, which correspond to tangible media such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer).

[0100] The instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other physical structures suitable for implementing the described techniques. Furthermore, this technique can be fully implemented in one or more circuit or logic elements.

Claims

1. An intracavitary robotic system, the intracavitary robotic system comprising: At least one robotic arm; An endoscopic tool, the endoscopic tool being removably coupled to the at least one robotic arm, the endoscopic tool including an imaging device coupled to a distal end portion of the endoscopic tool; A needle driver tool, which is removably coupled to the at least one robotic arm; A gripping tool, said gripping tool being removably coupled to said at least one robotic arm; processor; and A memory storing instructions that, when executed by the processor, cause the processor to perform the following operations: Receive images from the imaging device; Overlay the suture needle path onto the received image; and The at least one robotic arm is controlled to operate the needle driver tool to drive the suture needle based on the suture needle path overlaid on the received image. When executed by the processor, the instruction further causes the processor to perform the following operations: Detecting needle slippage; and In response to the detection of slippage of the suture needle, the at least one robotic arm is controlled to operate the endoscopic tool and / or the needle driver tool to take into account the detected slippage.

2. The intracavitary robotic system of claim 1, further comprising a force sensor coupled to the grasping tool. The instructions, when executed by the processor, further cause the processor to perform the following operations: The force applied to the tissue by the grasping tool is determined based on the force measurement data output from the force sensor; and An alarm is generated in response to the determination that the force applied to the tissue is greater than a predetermined force.

3. The intracavitary robotic system according to claim 2, wherein the force sensor comprises force sensors distributed in an array.

4. The intracavitary robotic system of claim 2, wherein the instructions, when executed by the processor, further cause the processor to perform the following operations: Tissue type is determined by processing force measurement data output from the force sensor using machine learning-based algorithms; Receive crawler type information; and The predetermined force is determined based on the organization type and the crawling tool type information.

5. The intracavitary robotic system of claim 1, further comprising a force sensor coupled to the grasping tool. The instructions, when executed by the processor, further cause the processor to perform the following operations: Determine the organization type; Get the type of crawling tool; and The maximum force applied by the gripping tool is set based on the organization type and the gripping tool type.

6. The intracavitary robotic system of claim 1, wherein the suture needle path includes at least one of a needle entry mark or a needle exit mark.

7. The intracavitary robotic system according to claim 1, wherein the intracavitary robotic system further comprises: An electromagnetic (EM) field generator, the electromagnetic field generator being configured to generate an electromagnetic field; and At least one electromagnetic sensor, said at least one electromagnetic sensor being connected to the suture needle, The instructions, when executed by the processor, further cause the processor to track the position of the suture needle based on the electromagnetic field sensed by the at least one electromagnetic sensor.

8. The intracavitary robotic system of claim 1, wherein detecting slippage includes detecting movement of the suture needle relative to the needle actuator tool.

9. The intracavitary robotic system of claim 1, wherein the instructions, when executed by the processor, further cause the processor to perform the following operations: The suture needle path is adjusted based on the detected slippage; and Overlay the adjusted suture needle path onto the received image. Controlling the at least one robotic arm includes controlling the at least one robotic arm to operate the endoscopic tool and the needle driver tool based on an adjusted suture path overlaid on the received image.

10. The intracavitary robotic system of claim 9, further comprising a user controller, The instructions, when executed by the processor, further cause the processor to provide tactile feedback to the user controller in response to detecting the slippage of the suture needle.

11. The intracavitary robotic system of claim 9, further comprising a user controller, The instructions, when executed by the processor, further cause the processor to generate vibrations in the user controller in response to detecting slippage of the suture needle.

12. The intracavitary robotic system of claim 1, further comprising a pressure sensor, The instructions, when executed by the processor, further cause the processor to perform the following operations: Tissue tension data is generated based on the measurement data output from the pressure sensor; and Needle slippage is predicted based on the tissue tension data.

13. The intracavitary robotic system of claim 1, wherein the instructions, when executed by the processor, further cause the processor to perform the following operations: Determine the current position and orientation of the suture needle; Determine that the suture needle is near the tissue; and In response to determining that the suture needle is near the tissue, a marker showing where the suture needle will exit the tissue will be overlaid on the received image based on the current position and orientation of the suture needle.

14. The intracavitary robotic system of claim 13, wherein the instructions, when executed by the processor, further cause the processor to overlay a marker showing the planned location where the suture needle will exit the tissue onto the received image.

15. The intracavitary robotic system of claim 13, wherein the instructions, when executed by the processor, further cause the processor to display the critical structure on one side or behind the suture location in the received image.

16. The intracavitary robotic system of claim 13, wherein the instructions, when executed by the processor, further cause the processor to display at least one of the entry position, orientation, or depth of the suture needle to avoid access to critical structures.

17. The intracavitary robotic system of claim 13, wherein the instructions, when executed by the processor, further cause the processor to perform the following operations: Determine the amount of tissue resistance to the movement of the suture needle; and This indicates the amount of resistance in the tissue.

18. Instructions executed by a processor of an intracavitary robotic system, said instructions, when executed by the processor of the intracavitary robotic system, cause the processor to perform the following operations: Receive images from a camera mounted on the endoscope; Overlay the suture needle path onto the received image; and The robot's actuator tool is controlled to drive the suture needle based on the suture needle path overlaid on the received image. Detect the sliding of the suture needle; The suture needle path is adjusted based on the detected slippage; Overlay the adjusted suture needle path onto the received image; as well as The robot is controlled to operate the actuator tool based on the adjusted suture needle path overlaid on the received image.

Images