Systems and methods for self-alignment and adjustment of robotic endoscopes

Through the 3D depth map and autonomous adjustment technology of the robot endoscopy system, the problem of navigation and setup complexity of the endoscopy system is solved, and a more efficient and lower-cost diagnosis and treatment process is achieved.

CN118973513AInactive Publication Date: 2024-11-15NOAH MEDICAL CORP
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202380030084.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-31
Filing Date
2023-05-22
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing endoscopy systems have complexity and time-consuming problems in navigation and setup, especially in reusable endoscopy that requires frequent cleaning and disinfection, resulting in delays in surgery and unoptimized operational procedures.

Method used

The robot endoscopic system is adopted to automatically detect and track the operating environment by generating 3D depth maps, automatically adjust the system position, avoid collisions, and optimize the alignment of the instrument drive mechanism with the patient side, simplifying the setup and surgical process.

Benefits of technology

Improves surgical setup efficiency, reduces surgical delays, optimizes operating procedures, reduces the cost and complexity of the endoscopy, makes it suitable for single use, and improves diagnostic and therapeutic performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118973513A_ABST
    Figure CN118973513A_ABST
Patent Text Reader

Abstract

The invention provides a method for a robotic endoscope system. The method comprises the steps that a 3D depth map of the surrounding environment of the robot endoscope system is generated; autonomously actuating a self-propelled base of a robotic support system to a desired position relative to the sickbed based on the 3D depth map, where the robotic support system includes a robotic arm coupled at a proximal end to the self-propelled base and coupled at a distal end to the flexible endoscopic device via an instrument drive mechanism (IDM); and driving the robotic arm to align the IDM with a component coupled to or as part of the sickbed.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 345,287 filed on May 24, 2022 and U.S. Provisional Patent Application No. 63 / 347,179 filed on May 31, 2022, each of which is incorporated herein by reference in its entirety. Background Art

[0003] Endoscopy procedures use endoscopy to examine the inside of a hollow organ or body cavity. Unlike many other medical imaging techniques, an endoscope is inserted directly into an organ. Flexible endoscopes that rely on intuitive manipulation and control can be delivered for diagnosis and treatment of diseases accessible through any natural orifice in the body. Depending on clinical indications, endoscopes can be designated as bronchoscopes, ureteroscopes, colonoscopes, gastroscopes, otolaryngoscopes, and various other endoscopes. For example, flexible bronchoscopes can be used for lung cancer diagnosis and / or surgical treatment. However, a difficult problem with bronchoscopy is to reach the upper lobe of the lung while navigating through the airway. In another example, flexible endoscopy has been used to examine and treat gastrointestinal (GI) tract disorders without creating an opening in the patient. The endoscope is introduced into the upper GI tract or lower GI tract via the mouth or anus, respectively. A miniature camera at the distal end captures an image of the GI wall, which helps clinicians diagnose GI diseases. Simple surgical procedures (such as polypectomy and biopsy) can be performed by introducing a flexible tool via a working channel to reach the site of interest at the distal end.

[0004] Traditionally, endoscopes are reusable and may need to be thoroughly cleaned, disinfected, and / or sterilized after each procedure. In most cases, cleaning, disinfecting, and sterilizing can be aggressive processes that kill germs and / or bacteria. Such procedures may also be harsh on the endoscope itself. As a result, the design of such reusable endoscopes may often be complex, especially to ensure that the endoscope can withstand such harsh cleaning, disinfecting, and sterilization regimens. Periodic maintenance and repairs may often be required of such reusable endoscopes.

[0005] For instruments that are difficult to clean properly, low-cost disposable medical devices designated for single use have become popular. Single-use disposable devices can be packaged in sterile wrappers to avoid the risk of pathogenic cross-contamination of diseases such as HIV, hepatitis, and other pathogens. Hospitals generally welcome the convenience of single-use disposable products because they no longer have to worry about aging, overuse, breakage, malfunction, and sterilization of the product. Traditional endoscopes typically include a handle that the operator uses to manipulate the endoscope. For single-use endoscopes, the handle typically encapsulates a camera, expensive electronics, and mechanical structures at the proximal end to transmit video and allow the user to manipulate the endoscope via a user interface. This can lead to high costs for the handle of a single-use endoscope.

[0006] The process of setting up a medical robot can be time consuming and challenging due to the number of components to set up and the complex workflow. The setup time can result in longer room turnover times, which can cause delays in surgery. Examples of challenging steps in the current system workflow are: (1) positioning the robotic system in a position appropriate for surgery and (2) aligning the instrument drive mechanism with the patient side. These steps are challenging because the user must consider the placement of other equipment in the room required for surgery and gain a spatial understanding of the workspace of the robotic components while manually performing these steps. During surgery, if there are any changes in the room setup, the user must manually adjust the system to accommodate. Summary of the invention

[0007] The present disclosure recognizes a need for a robotic endoscopy platform or system that allows a robotic endoscopy system to autonomously adjust in response to a real-time operating environment. The present disclosure addresses the above needs by providing methods and systems that can detect and track the operating environment of the system (e.g., the external environment surrounding the system) and automatically adjust the system, thereby (1) simplifying the operator's workflow during system setup and / or during surgery, and (2) making the system configuration more optimized than placement performed manually, which may be considered acceptable but not optimal. In some embodiments, autonomous processing and / or movement of the system can include aligning an instrument drive mechanism (IDM) of a robotic endoscopy device with a patient-side support, aligning a robotic base station (e.g., a robotic cart) relative to a patient or hospital suite device, identifying other technologies / devices and automatically configuring compatibility, avoiding collisions between the system and the patient and / or other objects in the operating environment, self-adjusting a robotic arm or IDM based on real-time instrument flexion detection and monitoring of patient breathing / vital bodies, etc.

[0008] In one aspect of the present disclosure, a method for a robotic endoscope system is provided. The method includes: generating a 3D depth map of the environment surrounding the robotic endoscope system; autonomously actuating a self-propelled base of a robotic support system to a desired position relative to a patient bed based on the 3D depth map, wherein the robotic support system includes a robotic arm coupled to the self-propelled base at a proximal end and coupled to a flexible endoscope device via an instrument drive mechanism (IDM) at a distal end; autonomously actuating the arm to avoid collision; and actuating the robotic arm to align the IDM with a component coupled to or as part of the patient bed. Given a 3D depth map input, the method implemented to enable autonomous repositioning of system components requires algorithms for processing and filtering image data, mapping image data to inputs to a robotic control algorithm, and algorithms for obstacle avoidance.

[0009] It is recognized herein that a robotic endoscope is needed that allows surgical or diagnostic operations to be performed with improved performance and cost efficiency. It is also recognized herein that the device and system comprising an endoscope can be disposable and can not require a large number of cleaning procedures. The present disclosure provides a low-cost, single-use articulated endoscope for diagnosis and treatment in various applications (such as bronchoscopy, urology, gynecology, arthroscopy, orthopedics, otolaryngology, gastrointestinal endoscopy, neurosurgery and various other applications). In some embodiments, the present disclosure provides a single-use, robot-controlled disposable bronchoscope for use with a robotic system to achieve diagnostic assessment of damage at any position in the pulmonary anatomical structure. It should be noted that the endoscopic system provided can be used in various minimally invasive surgical procedures, therapeutic or diagnostic surgeries involving various types of tissues including heart, bladder and lung tissue, and can be used in other anatomical regions of the patient's body (such as the digestive system including but not limited to the esophagus, liver, stomach, colon, urethra or the respiratory system including but not limited to the bronchi, lungs and various other regions).

[0010] It should be noted that the provided autonomous configuration, alignment and collision avoidance methods, endoscopic assemblies and various components of the apparatus can be used in various minimally invasive surgical procedures, therapeutic or diagnostic procedures involving various types of tissue including heart, bladder and lung tissue, and can be used in other anatomical regions of the patient's body (such as the digestive system including but not limited to the esophagus, liver, stomach, colon, urethra or the respiratory system including but not limited to the bronchi, lungs and various other regions).

[0011] In one aspect, a method for controlling movement and operation of a robotic endoscopic system in an environment is provided. The method includes: generating a 3D depth map of the environment surrounding the robotic endoscopic system; autonomously actuating a self-propelled base of a robotic support system to a desired position relative to a patient bed based on the 3D depth map, wherein the robotic support system includes a robotic arm coupled to the self-propelled base at a proximal end and coupled to an instrument drive mechanism (IDM) at a distal end; and actuating the robotic arm to autonomously align the IDM with a component coupled to or as part of the patient bed.

[0012] In a related but independent aspect, a system for controlling a robotic endoscopic system is provided, the system comprising: a memory storing computer-executable instructions; one or more processors in communication with the robotic endoscopic system and configured to execute the computer-executable instructions to: generate a 3D depth map of the environment surrounding the robotic endoscopic system; autonomously actuate a self-propelled base of a robotic support system to a desired position relative to a patient bed based on the 3D depth map, wherein the robotic support system comprises a robotic arm coupled to the self-propelled base at a proximal end and to an instrument drive mechanism (IDM) at a distal end; and actuate the robotic arm to autonomously align the IDM with a component coupled to or part of the patient bed.

[0013] In some embodiments, a 3D depth map is generated based at least in part on the 3D point cloud data. In some cases, the method further includes processing the 3D depth map to detect a patient bed and calculate a position and orientation of the robotic support system relative to the patient bed. In some embodiments, the flexible endoscopic device is releasably coupled to the IDM after the IDM is aligned with a component coupled to or as part of the patient bed.

[0014] In some embodiments, the method further includes controlling movement of the robotic arm to move the IDM to a predetermined distance from a component coupled to or as part of the patient bed. In some cases, the method further includes loading a flexible endoscopic device to couple to the IDM at a proximal end and to the component at a distal end. In some cases, the method further includes automatically adjusting a position of the IDM relative to the component upon detecting a buckling event.

[0015] In some embodiments, the method further includes detecting and identifying objects in the environment and reconfiguring the robotic arm to avoid collision with the object while maintaining the position and orientation of the IDM. In some cases, the method further includes detecting buckling of a flexible catheter coupled to the IDM while inserting the flexible catheter into the patient. In some cases, the method further includes executing a responsive velocity control algorithm to control the velocity of the tip of the flexible catheter while reconfiguring the robotic arm to avoid collision with the object.

[0016] In some embodiments, the IDM autonomously aligns with the component based at least in part on the sensor data. In some cases, the sensor data is captured by an electromagnetic sensor. In some cases, the sensor data is captured by a camera that includes a fiducial marker placed on the component, and the 3D depth map includes at least the 3D position of the fiducial marker.

[0017] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in the art from the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. As will be appreciated, the present disclosure is capable of other and different embodiments, and its several details are capable of modification in various obvious respects, all without departing from the present disclosure. Therefore, the drawings and description are to be regarded as illustrative in nature, rather than restrictive.

[0018] Incorporation by reference

[0019] All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. In the event that the publications and patents or patent applications incorporated by reference conflict with the disclosure contained in the specification, the specification is intended to supersede and / or take precedence over any such conflicting material. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The novel features of the present invention are set forth in detail in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by referring to the following detailed description and accompanying drawings (also referred to herein as "figures") which set forth illustrative embodiments utilizing the principles of the present invention, in which:

[0021] Figure 1 A robotic platform according to some embodiments of the present disclosure is schematically illustrated.

[0022] Figure 2 An example of a robotic catheter assembly with an anti-buckling device is shown.

[0023] Figure 3 An example of a robotic endoscopy (eg, bronchoscopy) system capable of performing autonomous collision avoidance and self-alignment before and during an operation is shown.

[0024] Figure 4A An example of using optical sensors to build a 3D map of the operating environment is shown;

[0025] Figure 4B and Figure 4C An example of automatically aligning an IDM (Instrument Drive Mechanism) based on EM sensor data is shown.

[0026] Figure 5 and Figure 6 An example of autonomous alignment of the IDM (Instrument Drive Mechanism) with the patient-side support is shown.

[0027] Figure 7 An example of a self-propelled (eg, via one or more propulsion units such as wheels, rotors, propellers) robotic cart being autonomously positioned relative to a patient bed is shown.

[0028] Figure 8 An example of self-alignment of a robotic endoscopic system is shown.

[0029] Fig. 9 An example process for autonomous alignment of a robotic endoscope system is shown.

[0030] Figure 10-12 Examples of collision avoidance of a robotic endoscopic system with respect to one or more objects in an operating environment before and during a surgical procedure are shown.

[0031] Fig.11 An example of a robotic bronchoscope including a handle portion and a flexible elongated member is shown.

[0032] Fig.12 An example of an instrument drive mechanism providing a mechanical interface to the handle portion of a robotic bronchoscope is shown.

[0033] Fig.13 An example of a collision avoidance algorithm is shown.

[0034] Fig.14 and Fig.15 An example of a flexible endoscope is shown.

[0035] Fig.16 An example of an instrument drive mechanism providing a mechanical interface to the handle portion of a robotic bronchoscope is shown.

[0036] Fig.17 An example of the distal tip of an endoscope is shown.

[0037] Fig.18 An example distal portion of a catheter with integrated imaging and lighting devices is shown. DETAILED DESCRIPTION

[0038] Although various embodiments of the present invention have been shown and described herein, it will be readily understood by those skilled in the art that such embodiments are provided by way of example only. Many variations, changes and substitutions may occur to those skilled in the art without departing from the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed.

[0039] The embodiments disclosed herein can be combined in one or more of many ways to provide improved diagnosis and treatment for patients. The disclosed embodiments can be combined with existing methods and devices to provide improved treatment, such as with known lung diagnosis, surgical methods, and surgical methods of other tissues and organs. It should be understood that any one or more of the structures and steps described herein can be combined with any one or more additional structures and steps of the methods and devices described herein, and the accompanying drawings and supporting text provide descriptions according to the embodiments.

[0040] Although the exemplary embodiments will be primarily directed to devices or systems for bronchoscopy, those skilled in the art will appreciate that this is not intended to be limiting and that the devices described herein may be used for other therapeutic or diagnostic procedures and various anatomical regions of a patient's body. The provided devices or systems may be used for urology, gynecology, rhinology, otology, laryngoscopy, gastroenterology using an endoscope, a combination device including an endoscope and an instrument, an endoscope with a positioning function, and those skilled in the art will appreciate that this is not intended to be limiting and that the devices described herein may be used for other therapeutic or diagnostic procedures and other anatomical regions of a patient's body, such as the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testicles, bladder, ears, nose, mouth, soft tissues (such as bone marrow, adipose tissue, muscle, glandular and mucosal tissue, spinal cord and nerves tissue, cartilage), hard biological tissue (such as teeth, bones, etc.), as well as body cavities and ducts (such as sinuses, ureters, colon, esophagus, lungs, blood vessels and throat), and various other tissues in the form of: neuroendoscopes, encephaloscopes, ophthalmoscopes, otoscopes, rhinoscopes, laryngoscopes, gastroscopes, esophagoscopes, bronchoscopes, thoracoscopic, pleural scopes, angioscopes, mediastinoscopes, nephroscopes, gastroscopes, duodenoscopes, choledochoscopes, bile ductoscopes, laparoscopes, amnioscopic, ureteroscopes, hysteroscopes, cystoscopes, rectoscopes, colonoscopes, arthroscopes, salivary gland endoscopes, orthopedic endoscopes, etc., which are combined with various tools or instruments.

[0041] The systems and devices herein can be combined in one or more of many ways to provide improved diagnosis and treatment for patients. The systems and devices provided herein can be combined with existing methods and devices to provide improved treatment, such as with known lung diagnosis, surgical methods, and surgical methods of other tissues and organs. It should be understood that any one or more of the structures and steps described herein can be combined with any one or more additional structures and steps of the methods and devices described herein, and the accompanying drawings and supporting text provide descriptions according to embodiments.

[0042] Whenever the term "at least", "greater than", or "greater than or equal to" precedes the first value in a series of two or more values, the term "at least", "greater than", or "greater than or equal to" applies to each value in the series. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0043] Whenever the term "no more than," "less than," or "less than or equal to" precedes the first value in a series of two or more values, the term "no more than," "less than," or "less than or equal to" applies to every value in the series. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0044] As used herein, the terms distal and proximal may generally refer to a position referenced from the device and may be opposite to an anatomical reference. For example, a distal position of a main shaft or main catheter may correspond to a proximal position of a patient's elongated member, and a proximal position of a main sheath or main catheter may correspond to a distal position of a patient's elongated member.

[0045] As described above, setting up a robotic endoscopic system can be time consuming and challenging due to the complexity of the operating environment, the requirement for precise alignment between instruments and patient body parts, and various other reasons. The present disclosure provides methods and systems that can detect and track the operating environment of the system and automatically adjust the system, thereby simplifying the workflow during system setup and / or during surgery. In some embodiments, the autonomous processes and / or movements of the system can include aligning the instrument drive mechanism (IDM) of the robotic endoscopic device with the patient side support, aligning the robot base (robotic cart) relative to the patient or hospital suite equipment, identifying other technologies / equipment and automatically configuring compatibility, avoiding collisions between the system and the patient and / or other objects in the operating environment, self-adjusting the robotic arm or IDM based on real-time instrument flexion detection and monitoring of patient breathing / vital bodies, and other functions described elsewhere herein.

[0046] The operating environment of the robotic endoscopic system may include one or more objects. The robotic endoscopic system may be capable of detecting one or more objects in the operating environment, generating a 3D map with depth information, performing autonomous alignment of the instrument drive mechanism relative to the patient bed or body part, and self-adjusting its placement and configuration to avoid collision with the one or more objects. The one or more objects may include, for example, system accessories (e.g., system monitor, monitor pole), external monitors, patient beds, patients, imaging equipment (e.g., fluoroscopic C-arms), anesthesia carts, and other equipment or subjects (e.g., operators, surgeons) in the operating environment before or during a surgical procedure.

[0047] Figure 1 A robotic platform 100 is schematically shown. The platform may include a robotic endoscopic system including one or more flexible articulated surgical instruments 105, and a support device 110, such as a robotic manipulator (e.g., a robotic arm), for driving, supporting, positioning, or controlling the movement and / or operation of the robotic system. The robotic platform may also include peripheral devices and subsystems, such as an imaging system, which may assist and / or facilitate navigating an elongated member to a target location within a subject 120.

[0048] A robotic endoscopic system is provided for performing surgical procedures or diagnostics with improved performance at low cost. For example, the robotic endoscopic system may include a steerable catheter that may be completely disposable. Figure 1 As shown, the robotic endoscope system may include a steerable catheter assembly 105 and a robotic support system 110 for supporting or carrying the steerable catheter assembly. The steerable catheter assembly may be an endoscope. In some embodiments, the steerable catheter assembly may be a disposable robotic endoscope. In some embodiments, the robotic endoscope system may include an instrument drive mechanism (IDM) 103, which is attached to the distal end of a robotic arm 107 of the robotic support system. The instrument drive mechanism may be provided by any suitable controller device (e.g., a handheld controller), which may or may not include a robotic system. The instrument drive mechanism may provide a mechanical and electrical interface for the steerable catheter assembly 105. The mechanical interface may allow the steerable catheter assembly 105 to be releasably coupled to the instrument drive mechanism 103. For example, the handle portion 104 of the steerable catheter assembly may be attached to the instrument drive mechanism (IDM) via a quick installation / release device (e.g., a magnet, a spring-loaded level, etc.). In some cases, the steerable catheter assembly may be manually coupled to the instrument drive mechanism or released from the instrument drive mechanism without the use of tools. The instrument drive mechanism may be used to control an elongated member or robotic catheter assembly in two or more degrees of freedom (eg, articulation).

[0049] The robotic support system 110 may include a robotic arm 107 and a mobile base (e.g., a robotic cart) 109. The robotic arm 107 may enable positioning of a robotic catheter assembly or other robotic instrument. In some cases, the user interface, robotic control module, and robotic arm may be mounted to a mobile cart. The mobile cart may include various components, such as a rechargeable power supply in electrical communication with an electrical panel that provides a charging port for a portable electronic device, a converter, a transformer, and a surge protector for multiple AC and DC outlets as a power source for onboard equipment, including one or more computers storing user interface specific software.

[0050] The robotic arm 107 can have redundant degrees of freedom, allowing its elbow to be moved algorithmically or passively to a configuration that facilitates the positioning of the operator to initiate the robotic system or other robotic instruments. For example, the robotic arm can include multiple joints with redundant degrees of freedom so that the joints of the robotic arm can be driven into a series of different configurations for a given end effector position (e.g., IDM position). The redundant degrees of freedom can advantageously allow the robotic arm to self-adjust to an optimal configuration before or during surgery to avoid collisions with other objects in the operating environment. For example, the instrument drive mechanism can automatically align itself with the patient-side support (e.g., a support structure at the bed for holding and supporting the endoscopic device in place) during the setup process. During the setup process and the operation process, after detecting the movement of the patient-side support, the instrument drive mechanism (IDM) can automatically adjust accordingly to avoid collisions while maintaining the position of the IDM, thereby eliminating any interruption to the surgical workflow and avoiding misalignment. In another example, when the robotic arm is activated during a setup procedure or surgical operation, the system can detect unwanted proximity between any part of the robotic arm and other objects around it (e.g., the system's monitor), and the robotic arm can automatically reconfigure to move away from the monitor to avoid collision.

[0051] In some embodiments, in addition to autonomous movement of the robotic arm (e.g., automatically positioning the steerable catheter assembly 105 to an initial position (e.g., an access point) to access the target tissue), the robotic arm can also be passively moved by an operator. In this case, the operator can push the arm to any position and the arm will move compliantly. The robotic arm can also be controlled in a compliant mode to improve human-machine interaction. For example, compliant motion control of robotics can employ collision avoidance strategies, and position-force control can be designed to save unnecessary energy consumption while reducing the impact of possible collisions.

[0052] The steerable catheter assembly 105 may include a flexible elongate member coupled to a handle portion.The robotic endoscopy system may include an anti-buckling device 101 for preventing the elongate member from buckling during use.

[0053] Figure 2Another example of a robotic catheter assembly with an anti-buckling device 201 is shown. The maneuverable catheter assembly may include a handle portion 211, which may include components configured to process image data, provide power, or establish communication with other external devices. For example, the handle portion 211 may include circuits and communication elements that enable electrical communication between the maneuverable catheter assembly and the instrument drive mechanism 221 and any other external system or device. In another example, the handle portion 211 may include circuit elements, such as a power supply for powering the electronic devices (e.g., cameras and LED lights) of the endoscope. In some cases, the handle portion may be electrically communicated with the instrument drive mechanism 221 via an electrical interface (e.g., a printed circuit board) so that the communication module of the instrument drive mechanism can receive image / video data and / or sensor data and can transmit it to other external devices / systems. Alternatively or additionally, the instrument drive mechanism 221 may only provide a mechanical interface. The handle portion may be electrically communicated with a modular wireless communication device or any other user device (e.g., a portable / handheld device or a controller) to transmit sensor data and / or receive control signals.

[0054] The steerable catheter assembly can include a flexible elongated member 213 (i.e., catheter) coupled to the handle portion 211. In some embodiments, the flexible elongated member can include an axis, a steerable tip, and a steerable portion. The steerable catheter assembly can be a disposable robotic endoscope. In some cases, only the elongated member can be disposable. In some cases, at least a portion of the elongated member (e.g., an axis, a steerable tip, etc.) can be disposable. In some cases, the entire steerable catheter assembly including the handle portion and the elongated member can be disposable. The flexible elongated member and the handle portion are designed so that the entire steerable catheter assembly can be discarded at low cost.

[0055] The robotic endoscope can be releasably coupled to the instrument drive mechanism 221. The instrument drive mechanism 221 can be mounted to the arm of the robotic support system or any actuation support system as described above. The instrument drive mechanism can provide a mechanical and electrical interface for the robotic endoscope. The mechanical interface can allow the robotic endoscope to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the robotic endoscope can be attached to the instrument drive mechanism via a quick installation / release device (e.g., a magnet and a spring-loaded level). In some cases, the robotic endoscope can be manually coupled to the instrument drive mechanism or released from the instrument drive mechanism without the use of tools. In some embodiments, the instrument drive mechanism 221 can include a set of motors that are actuated to rotationally drive a set of pull wires of the catheter. The handle portion 211 of the catheter assembly can be mounted on the instrument drive mechanism so that its pulley assembly is driven by the group of motors. The number of pulleys can vary based on the pull wire configuration. In some cases, one, two, three, four or more pull wires can be used to articulate the catheter.

[0056] The handle portion can be designed to allow for disposable use of the robotic endoscope at a reduced cost. For example, classic manual and robotic endoscopes can have a cable at the proximal end of the endoscope handle. The cable typically includes an illumination fiber, a camera video cable, and other sensor fibers or cables, such as EM sensors or shape sensing fibers. Such complex cables can be expensive, increasing the cost of the endoscope. The provided robotic endoscope can have an optimized design, thereby allowing for a simplified structure and components while retaining mechanical and electrical functionality. In some cases, the handle portion of the robotic endoscope can be wirelessly designed while providing a mechanical / electrical interface for the catheter.

[0057] In some cases, the handle portion can be a housing or include components configured to process image data, provide power, or establish communication with other external devices. In some cases, the communication can be wireless communication. For example, the wireless communication can include Wi-Fi, radio communication, Bluetooth, IR communication, or other types of direct communication. This wireless communication capability can enable the robotic bronchoscope to operate in a plug-and-play manner and can be easily disposed of after a single use. In some cases, the handle portion can include circuit elements, such as a power supply for powering electronic devices (e.g., cameras and LED light sources) disposed within the robotic bronchoscope or catheter.

[0058] The handle portion can be designed with the catheter, so that cables or optical fibers can be omitted. For example, the catheter portion can be designed with a single working channel, allowing instruments to pass through a robotic bronchoscope, as well as low-cost electronic devices (e.g., a tip chip camera), an illumination source (e.g., a light emitting diode (LED)), and an EM sensor located at the optimal position according to the mechanical structure of the catheter. This can allow the design of the handle portion to be simplified. For example, by using LEDs for illumination, the termination of the handle portion can be based only on electric welding or wire crimping. For example, the handle portion can include a proximal plate, and the camera cable, LED cable, and EM sensor cable terminate at the proximal plate, and the proximal plate is connected to the interface of the handle portion and establishes an electrical connection with the instrument drive mechanism. As described above, the instrument drive mechanism is attached to the robot arm (robot support system) and provides a mechanical and electrical interface with the handle portion. This can advantageously improve assembly and implementation efficiency, and simplify the manufacturing process and cost. In some cases, the handle portion and the catheter can be discarded after a single use.

[0059] In some embodiments, the steerable catheter assembly can have a substantially integral design, wherein one or more assemblies can be integrated with the catheter, thereby simplifying assembly, manufacturing process, while retaining the kinematics and dynamic performance of the steerable catheter. As shown in the example, the steerable catheter assembly can include an elongated member 213 or a detection portion, which is brought to the vicinity of the tissue and / or region to be inspected. In some cases, the elongated member 213 can also be referred to as a catheter. The catheter 213 can include an internal structure, such as a working channel that allows an instrument to be inserted through. For example, the working channel can have a size such as approximately 2 millimeters in diameter, to be compatible with standard instruments. The working channel can have any other suitable size based on application.

[0060] The catheter 213 can be made of suitable materials to obtain desired flexibility or bending stiffness. In some cases, the material of the catheter can be selected so that it can maintain structural support for the internal structure (e.g., working channel) while having enough flexibility (e.g., being able to bend in all directions and orientations). For example, the catheter can be made of any suitable material, such as polyurethane, vinyl (e.g., polyvinyl chloride), nylon (e.g., vestamid, grillamid), polyurethane, polyethylene, polypropylene, polycarbonate, polyester, silicone elastomer, acetate fiber, etc. In some cases, the material can be a polymer material, a biocompatible polymer material, and the catheter can have enough flexibility to advance through a path with a small curvature without causing pain to the subject. In some cases, the catheter can include a sheath. The length of the sheath can be different from the length of the catheter. The sheath can be shorter than the catheter to provide desired support. Alternatively, the catheter can be a single-piece component substantially.

[0061] In some cases, the distal portion or tip of the catheter can be substantially flexible so that it can be turned to one or more directions (e.g., pitch, yaw). In some embodiments, the catheter can have a variable bending stiffness along the longitudinal axis. For example, the catheter can include multiple segments with different bending stiffness (e.g., flexibility, semi-rigidity and rigidity). The bending stiffness can be changed by selecting a material with different stiffness / rigidity, changing the structure in different segments, adding additional support components or any combination thereof. In some cases, the proximal end of the catheter does not need to be bent to a great extent, so the proximal end portion of the catheter can be reinforced with an additional mechanical structure (e.g., an additional material layer) to achieve greater bending stiffness. This design can provide support and stability for the catheter. In some cases, variable bending stiffness can be achieved by using different materials during catheter extrusion. This can advantageously allow different stiffness levels along the axis of the catheter during the extrusion manufacturing process without the need for additional fastening or assembling of different materials.

[0062] The distal portion of the catheter can be manipulated by one or more pull wires. The distal portion of the catheter can be made of any suitable material, such as a copolymer, a polymer, a metal or an alloy, so that it can be bent by the pull wire. In some embodiments, the proximal end or portion of one or more pull wires can be operably coupled to various mechanisms (e.g., gears, pulleys, etc.) in the handle portion of the catheter assembly. The pull wire can be a metal wire, cable or wire, or a polymer wire, cable or wire. The pull wire can also be made of natural or organic materials or fibers. The pull wire can be any type of suitable wire, cable or wire that can support various loads without deformation, significant deformation or breakage. The distal end or portion of one or more pull wires can be anchored or integrated into the distal portion of the catheter so that the control unit can apply force or tension to the distal portion, which can manipulate or articulate (e.g., upward, downward, pitch, yaw or any direction between the two) at least the distal portion (e.g., flexible portion) of the catheter.

[0063] As described above, the pull wire can be made of any suitable material, such as stainless steel (e.g., SS316), a metal, an alloy, a polymer, nylon, or a biocompatible material. The pull wire can be a wire, a cable, or a thread. In some embodiments, different pull wires can be made of different materials to change the load-bearing capacity of the pull wire. In some embodiments, different portions of the pull wire can be made of different materials to change the stiffness and / or load-bearing along the pull wire. In some embodiments, the pull wire can be used to transmit electrical signals.

[0064] The catheter can be sized so that one or more electronic components can be integrated into the catheter. For example, the outer diameter of the distal tip can be approximately 4 to 4.4 millimeters (mm), and the diameter of the working channel can be approximately 2 mm, so that one or more electronic components can be embedded in the catheter wall or catheter interstitium. However, it should be noted that based on different applications, the outer diameter can be any range less than 4 mm or greater than 4.4 mm, and the diameter of the working channel can be any range depending on the tool size or specific application.

[0065] One or more electronic components may include an imaging device, an illumination device, or a sensor. In some embodiments, the imaging device may be a video camera. The imaging device may include an optical element and an image sensor for capturing image data. The image sensor may be configured to generate image data in response to the wavelength of light. Various image sensors may be used to capture image data, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD). The imaging device may be a low-cost camera. In some cases, the image sensor may be disposed on a circuit board. The circuit board may be an imaging printed circuit board (PCB). The PCB may include a plurality of electronic components for processing image signals. For example, the circuit of a CCD sensor may include an A / D converter and an amplifier to amplify and convert an analog signal provided by the CCD sensor. Alternatively, the image sensor may be integrated with an amplifier and a converter to convert an analog signal into a digital signal, thereby eliminating the need for a circuit board. In some cases, the output of the image sensor or circuit board may be image data (digital signal), which may be further processed by a camera circuit or a camera processor. In some cases, the image sensor may include an optical sensor array.

[0066] The lighting device may include one or more light sources located at the distal tip. The light source may be a light emitting diode (LED), an organic LED (OLED), a quantum dot, or any other suitable light source. In some cases, the light source may be a miniaturized LED to achieve a compact design or dual tone flash LED lighting.

[0067] In some embodiments, the catheter can be designed to be flexible. When the flexible portion of the catheter is inserted into the patient through the endoscope by the extension mechanism, one or more portions can bend or flex.

[0068] An anti-buckling mechanism 201 can be coupled to the handle portion of the robotic endoscope to support the catheter. The anti-buckling mechanism is used to prevent buckling of the insertion shaft. The anti-buckling mechanism 201 can be a telescopic extension device with an internal mechanism to achieve anti-buckling of the catheter during insertion and removal. One end of the anti-buckling mechanism is detachably connected to the handle portion of the robotic bronchoscope, and the other end is detachably connected to a support surface 203. As shown in the example, the anti-buckling tube can be attached to a bracket on the instrument drive mechanism and can be removed and discarded after the procedure via a quick release mechanism. Figure 2 In the example shown, a support arm (e.g., an ET tube holder support arm) can be supported by a robotic mobile cart that supports the endotracheal tube holder and provides a support surface for the distal end of the anti-buckling tube to press against when it is compressed. The support arm can be controlled to rotate, translate vertically up and down, and / or telescope the cantilever for precise positioning over the patient's mouth and attachment to the endotracheal tube holder. The support arm positioning can be synchronized with the movement of the robotic arm so that the location of the catheter entry point can be tracked.

[0069] The anti-buckling mechanism may require a relatively linear trajectory to travel. In some cases, this trajectory may be ensured via alignment between the anti-buckling mechanism and the patient-side connector in the folded state. Figure 1 An example of a patient-side connector 121 and an IDM 103 is shown. For example, the patient-side connector can be secured to a patient-side bracket 123 (e.g., attached to a hospital bed). Alignment between the IDM and the patient-side connector / bracket can involve aligning a folded anti-buckling mechanism with the patient-side connector. The robotic arm can automatically move the IDM to a position that aligns the IDM with the patient-side connector. In some cases, alignment can include generating a 3D depth map of the operating environment, moving a mobile cart to a desired position relative to the hospital bed based on the depth map, and moving the IDM to a position and orientation aligned with the patient-side connector based on detection of the position / positioning of the patient-side connector.

[0070] Figure 3An example of a robotic endoscope (e.g., bronchoscope) system capable of performing autonomous collision avoidance and self-alignment before and during operation 300, 330 is shown. The robotic endoscope (e.g., bronchoscope) system 300 may include a steerable catheter assembly 320 and a robotic support system 310 for supporting or carrying the steerable catheter assembly. In some cases, the steerable catheter assembly may be a bronchoscope. The steerable catheter assembly may be the same as the endoscopic device described elsewhere herein. In some cases, the steerable catheter assembly may be a disposable robotic bronchoscope. In some embodiments, the robotic endoscope (e.g., bronchoscope) system 300 may include an instrument drive mechanism (IDM) 313, which is attached to the end of a robotic arm 311 of the robotic support system. The instrument drive mechanism may be provided by any suitable controller device (e.g., a handheld controller), which may or may not include a robotic system. The instrument drive mechanism may provide a mechanical and electrical interface for the steerable catheter assembly 320. The mechanical interface may allow the steerable catheter assembly 320 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the steerable catheter assembly can be attached to the instrument drive mechanism via a quick-install / release device (e.g., magnet, spring-loaded level, etc.). In some cases, the steerable catheter assembly can be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0071] The steerable catheter assembly 320 may include a handle portion 323, which may include components configured to process image data, provide power, or establish communication with other external devices. For example, the handle portion 323 may include circuits and communication elements to enable electrical communication between the steerable catheter assembly 320 and the instrument drive mechanism 313 and any other external system or device. In another example, the handle portion 323 may include circuit elements, such as a power supply for powering the electronic equipment (e.g., camera and LED light) of the endoscope. In some cases, the handle portion may be electrically communicated with the instrument drive mechanism 313 via an electrical interface (e.g., a printed circuit board) so that the communication module of the instrument drive mechanism can receive image / video data and / or sensor data and can transmit it to other external devices / systems. Alternatively or additionally, the instrument drive mechanism 313 may only provide a mechanical interface. The handle portion may be electrically communicated with a modular wireless communication device or any other user device (e.g., a portable / handheld device or a controller) to transmit sensor data and / or receive control signals.

[0072] Manipulable catheter assembly 320 can include a flexible elongated member 321 coupled to a handle portion. In some embodiments, the flexible elongated member can include a shaft, a manipulable tip, and a manipulable portion. Manipulable catheter assembly can be a disposable robotic bronchoscope. In some cases, only the elongated member can be disposable. In some cases, at least a portion of the elongated member (e.g., shaft, manipulable tip, etc.) can be disposable. In some cases, the entire manipulable catheter assembly 320 including the handle portion and the elongated member can be disposable. The flexible elongated member and the handle portion are designed so that the entire manipulable catheter assembly can be discarded at low cost.

[0073] In some embodiments, the provided bronchoscope system may also include accessories such as a user interface. As shown in the example system 330, during operation, one or more components of the system (such as a treatment interface module 331 (user console side) and / or a treatment control module 333 (patient and robot sides)) may be brought into the operating environment. One or more components or accessories may be added to or removed from the operating environment before or during the surgical operation. The robotic arm 311 may have redundant degrees of freedom so that the joints of the robotic arm may be driven into a range of different configurations for a given end effector position. For example, upon detection of the treatment interface module 311, the robotic arm 311 may automatically move into different configurations to avoid collision with the treatment interface module while the distal end of the arm of the IDM 313 maintains a specific state (e.g., a given position or velocity of the end effector). Details regarding collision avoidance will be described later in this document.

[0074] The treatment interface module can allow an operator or user to interact with the bronchoscope during the surgical procedure. In some embodiments, the treatment control module 333 can be a handheld controller. The treatment control module 333 can allow the user to control the speed of the bronchoscope tip, as described elsewhere herein. In some cases, the treatment control module can include a proprietary user input device and one or more additional elements that are detachably coupled to an existing user device to improve the user input experience. For example, a physical trackball or scroll wheel can replace or supplement the functionality of at least one of the virtual graphical elements (e.g., navigation arrows displayed on a touchpad) displayed on a graphical user interface (GUI) by giving it functionality similar to the replaced graphical elements. Examples of user devices can include, but are not limited to, mobile devices, smart phones / cellular phones, tablet computers, personal digital assistants (PDAs), notebook computers, desktop computers, media content players, and the like. Details about the user interface device and the user console will be described later in this article.

[0075] The robotic endoscope platform of this article can be capable of detecting one or more objects in the operating environment around the robotic endoscope system. In these cases, the detection of the operating environment can include generating an obstacle map. The obstacle map can be a three-dimensional (3D) map describing the location of the detected objects in 3D space.

[0076] A 3D map of the operating environment can be constructed based on the sensed data. In some cases, sensed data is received from one or more visual sensors, including depth information of the environment. The visual sensor may include a camera, a camera, a three-dimensional (3D) depth camera, a stereo camera, a depth camera, a red, green, and blue depth (RGB-D) camera, a time of flight (TOF) camera, an infrared camera, a charge coupled device (CCD) image sensor, or a complementary metal oxide semiconductor (CMOS) image sensor. For example, the visual sensor may include only one camera (monocular visual sensor). Alternatively, the visual sensor may include two (binocular visual sensor) or more cameras. The visual sensor may be provided on a robotic endoscope system, such as a robotic cart, a monitor, etc. Alternatively or additionally, the visual sensor may not be provided on the robotic endoscope system. For example, the visual sensor may be provided on a wall, a ceiling, or other location in an operating environment (e.g., a room). In an embodiment using multiple visual sensors, each sensor may be located in a different part of the robotic endoscope system, and the difference between the image data collected by each sensor may be used to provide depth information of the environment. Depth information may be used herein to refer to information about the distance of one or more objects from the robotic endoscope system and / or sensor. In embodiments using a single vision sensor, the depth information may be obtained by capturing image data at multiple different positions and orientations of the vision sensor and then reconstructing the depth information using a suitable image analysis technique (e.g., structure from motion).

[0077] Figure 4A An example of building a 3D map of an operating environment 400 using an optical sensor 401 (e.g., a camera) is shown. As described elsewhere herein, the operating environment of the system can generally include an environment external to the system (e.g., the room in which the system is located, within a certain distance, etc.) or an environment in which the system can move or operate. In some cases, the camera can be a plenoptic camera with a main lens and an additional microlens array (MLA). A plenoptic camera model can be used to calculate a depth map of the captured image data. In some cases, the image data captured by the camera can be a grayscale image with depth information at each pixel coordinate (i.e., a depth map). The camera can be calibrated to obtain intrinsic camera parameters (e.g., focal length, focal length, distance between the MLA and the image sensor, pixel size, etc.) to improve depth measurement accuracy. Other parameters (e.g., distortion coefficients) can also be calibrated to correct the image for metric depth measurement.

[0078] In some cases, the image data may be received and processed by one or more processors of the robotic endoscope system. For example, preprocessing of captured image data may be performed. In an embodiment, the preprocessing algorithm may include an image processing algorithm, such as image smoothing, to mitigate the effects of sensor noise, or image histogram equalization to enhance pixel intensity values. Next, the optical method described elsewhere herein may be used to generate a depth map of the operating environment 400. In some cases, computer vision (CV) techniques or computer vision systems may be used to process sensor data to extract high-level understanding of the operating environment, object detection, object classification, extraction of scene depth, estimation of relative position of objects, and extraction of the direction of objects in space. For example, passive methods that only require images may be used to generate CV output data. Passive methods may include, for example, object recognition, stereoscopic vision, monocular motion shape, shadow shape, and synchronous localization and mapping (SLAM). Alternatively, an active method may be used, which may require controlled light to be projected into a target scene, and the active method may include, for example, structured light and time of flight (ToF). In some cases, a description of the 3D scene can be generated using computer vision techniques (e.g., optical flow, computational stereo methods, iterative methods combined with predictive models, machine learning methods, predictive filtering, or any non-rigid registration method).

[0079] In some cases, a fiducial marker 403 may be used to align the IDM with the patient-side support. The fiducial marker may be placed on the bed or on the patient-side support 405. The fiducial marker may have a 2D shape or pattern. For example, the fiducial marker may be a 2D QR code, a grid, or any asymmetric shape. By acquiring images of the 2D fiducial marker (e.g., from different angles), the position and orientation of the patient-side support where the fiducial marker is placed in the camera frame may be determined (e.g., triangulated). Based on the known spatial relationship between the camera and the IDM, the orientation and position of the IDM relative to the patient-side support may be calculated. Alternatively, the fiducial marker may be a 3D fiducial marker so that the marker is visible and discernible at a wide range of angles. For example, the 3D fiducial marker may be located within the field of view of the imaging system so that the fiducial marker is always discernible regardless of the position of the optical sensor relative to the bed or the marker. The fiducial marker may have any suitable 2D / 3D shape (non-isotropic) or pattern so that the projection of the fiducial marker corresponding to the field of view / angle may be distinguished from the projection of another field of view / angle. Alternatively, the alignment of the IDM with the patient-side support may not require a fiducial marker. For example, the segmentation and / or object recognition methods described elsewhere in this document can be used to identify the patient-side bracket without the need for a fiducial marker. In some cases, independent of the 3D point cloud, the IDM can be aligned with the patient-side bracket using a 3D fiducial marker. For example, an image frame or video sequence containing a fiducial marker can be acquired and processed to identify the spatial relationship (e.g., direction, position) between the imaging device and the patient-side bracket (i.e., the fiducial marker). Based on the known geometric relationship between the IDM and the imaging device, the spatial relationship between the IDM and the patient-side bracket can be derived. Alternatively, the fiducial marker can be used in conjunction with a 3D depth map. For example, the identity of an object (e.g., a patient-side bracket) can be identified by a fiducial marker, and depth data can be assigned to the object based on the 3D depth map. In some cases, a 3D depth map can include a 3D point cloud. Alternatively, a 3D depth map can be generated based on optical image data. The 3D depth map may include at least one object (e.g., a fiducial marker) having depth information obtained by an optical method as described above.

[0080] In some cases, the imaging device can be used in combination with other types of sensors (e.g., proximity sensors, position sensors, positioning sensors, etc.) to improve the accuracy of the position information. For example, the sensing data can also include sensor data from one or more proximity sensors. The proximity sensor can be of any suitable type, such as an ultrasonic sensor (e.g., a wide-angle sensor, an array sensor) or a light detection and ranging (lidar) sensor. Lidar can be used to obtain three-dimensional information of the environment by measuring the distance to an object. The proximity sensor can also be provided at the robotic endoscopic system. The proximity sensor can be located near the visual sensor. Alternatively, the proximity sensor can be located on a portion of the robotic endoscopic system that is different from the portion used to carry the visual sensor.

[0081] In some cases, a 3D depth map can be generated using single-modal sensor data (e.g., image data, LiDAR, proximity data, etc.). Alternatively, a 3D depth map can be generated using multi-modal data. For example, a Kalman filter or deep learning model can be used to fuse image data and a 3D point cloud generated by a LiDAR system to generate a 3D map. The 3D map can then be used for automatic alignment of the IDM, self-localization of the robotic car, collision avoidance, and various other functions described elsewhere in this document.

[0082] In some embodiments, the autonomous alignment of the IDM with the patient-side support can be based on positioning sensors. For example, sensor signals can be acquired by an electromagnetic coil located on the IDM and an electromagnetic coil located on the patient-side support and an electromagnetic tracking system. The position and orientation of the IDM and the patient-side support can be detected, and the difference can be used to generate a command to move the robotic arm, thereby achieving autonomous alignment between the IDM and the patient-side support.

[0083] like Figure 4B As shown, the system may include an EM field generator 415 that emits an EM field in the environment. During surgery, the EM field generator may be placed next to the patient's torso, such as on a bed 407, on a robotic cart, or any other suitable location in the environment 410. The system may include a first EM sensor 411 located at the IDM for measuring the position and orientation of the IDM, and a second EM sensor 413 located at the patient-side support for measuring the position and orientation of the patient-side support. Figure 4C An example of a patient side support 431 and associated EM sensor 430 is shown. Return to Reference Figure 4B The system can utilize an EM field generator and two sets of EM sensors 411, 413 to locate the position and orientation of the IDM and the patient-side support in 3D space.

[0084] The position and orientation measured by the two sets of EM sensors can be represented in the field generator frame. The EM sensors can be 6DOF sensors (e.g., X, Y, Z, roll, pitch, yaw) that are capable of sensing the EM signals generated by the EM field generator and measuring the six degrees of freedom spatial data of the IDM and the patient-side support. Alternatively, a pair of 5DOF EM sensors can be located at the IDM and / or the patient-side support to measure position and orientation. For example, a single 5DOF sensor can generate X, Y, Z, pitch, and yaw signals, but not a roll signal. By placing 2 5DOF sensors inside the device so that they are fixed relative to each other and their central axes are not parallel to each other, a pair of 5DOF signals can be used to calculate roll information.

[0085] The system can then process the spatial data regarding the IDM and patient-side support to determine whether the cart is properly positioned for automated alignment and move the IDM to align with the patient-side support using, for example, closed-loop feedback (e.g., controlling robotic arm movement to correct IDM position / orientation based on EM sensor data) and / or other control methods described elsewhere herein.

[0086] Figure 5 An example of automatic alignment of an IDM (instrument drive mechanism) with a patient-side support is shown. In the illustrated example 510, the instrument drive mechanism may initially be positioned out of alignment with the patient-side support. During the setup process, actuators of multiple links / joints of the robotic arm may be actuated and automatically align the IDM with the patient-side support 520. In some cases, after alignment, the IDM may be moved to an appropriate position a predetermined distance 610 from the patient-side support 600 to load an instrument such as a catheter assembly or an endoscopic device, such as Figure 6 As shown. For example, the flexible endoscopic device can be coupled to the IDM at the proximal end of the endoscope and coupled to the patient-side support via a connector at the distal end. In some cases, the predetermined distance 610 can be generated based on the size of the endoscopic device or empirical data. By switching the robotic arm to passive mode, the user can be allowed to further manually adjust the position of the IDM as needed. For example, once the passive mode is enabled, the robotic arm can be placed in any configuration, position, or orientation by the user directly applying force and will maintain the desired position and orientation.

[0087] As described above, the robotic arm can have redundant degrees of freedom. For example, the robotic arm can have six, seven, eight, nine or more degrees of freedom (DOF) so that the IDM can be oriented in a five or six degree of freedom (DOF) space. For example, a robotic arm end effector (e.g., an IDM) that can be positioned with six degrees of freedom can have nine degrees of freedom (six end effector degrees of freedom - three for positioning, three for orientation - plus three degrees of freedom to comply with access point constraints), or ten or more degrees of freedom in some cases. A highly configurable robotic arm with more degrees of freedom than desired for a given end effector position can beneficially provide sufficient degrees of freedom to allow a range of joint states for the end effector position in the workspace. During surgery, if there is any movement of the patient-side support, the instrument drive mechanism can be able to automatically adjust accordingly to maintain alignment with the patient-side support, thereby eliminating any interruption to the surgical workflow and avoiding misalignment.

[0088] In some embodiments, a 3D depth map generated by the platform can be used for automated mobile / robotic cart placement. For example, a 3D depth map can include a description of the operating environment, such as identification of equipment, beds, human operators, patients, etc., and such a 3D depth map can be used to generate an optimal position for a mobile cart relative to the bed. As described above, computer vision (CV) techniques or computer vision systems can be used to extract a high-level understanding of the operating environment, object detection, object classification, extraction of scene depth and estimation of relative position of objects, extraction of the orientation of objects in space. 3D map information and sensor data (e.g., proximity sensors, imaging sensors) can be used to detect whether the robotic cart is located within an optimal area relative to the bed.

[0089] Figure 7An example of a self-propelled (e.g., via one or more propulsion units, such as wheels, rotors, propellers) robot cart being autonomously placed relative to a bed is shown. In some embodiments, the propulsion unit may include a plurality of wheels that may allow the robot cart to roll on an underlying surface. In some examples, two, three, or four wheels may be provided that may allow the robot cart to stand stably without moving. In some cases, stability may be achieved with the aid of one or more wheels or other stable platforms (e.g., gyroscopic platforms). The size of the wheels may be different or may be the same. In some cases, the diameter of the wheels may be at least about 1 cm, 2 cm, 3 cm, 4 cm, 5 cm, 8 cm, 9 cm, 10 cm, 15 cm, 20 cm, 25 cm, 30 cm, 35 cm, 40 cm, 45 cm, 50 cm, 55 cm, 60 cm, 65 cm, 70 cm, 75 cm, 80 cm, 85 cm, 90 cm, 95 cm, 100 cm, 150 cm, or 200 cm. The wheel surface may be smooth or patterned. The wheels may also allow the robot cart to move laterally and / or rotate in place. The robot cart may be capable of any combination of translational or rotational movement. The propulsion unit may be driven by one or more actuators. For example, a motor, engine, transmission system or any other component may be provided to assist in driving the propulsion of the robot cart.

[0090] Based on the 3D depth map, an optimal position for the robotic cart can be generated. The optimal position can be generated based on the size of the robotic cart, the size of the robotic arm (workspace), the size of the endoscopic device, and the 3D depth map. Real-time sensor data (e.g., proximity sensors) can be collected and used to determine whether the robotic cart is in the proper position relative to the patient bed. Figure 8 As shown, when it is detected that the self-propelled robotic cart is not in the proper position 810 relative to the patient bed, the robotic cart can automatically move to the proper position 820. The proper position, movement speed, movement acceleration, and movement trajectory can be calculated by one or more processors of the platform based at least in part on the 3D depth map.

[0091] Alternatively or additionally, the system may notify the user of the non-ideal placement of the robotic cart and may prompt the user to intervene, for example, a message, warning, or notification may be displayed on the screen, along with a suggestion for placing the robotic cart (e.g., specifying that the cart should be closer to the bed) and / or displaying a 2D / 3D depth map of the operating environment and the robotic system. For example, the robotic system may generate a preferred relative cart position and orientation relative to the current cart position and display an animation to the user to guide the user in changing the position of the robotic cart.

[0092] Fig. 9An example process 900 for autonomous alignment of a robotic endoscope system is shown. It should be noted that in the illustrated process, while lidar data (e.g., 3D point cloud) is used to detect objects in the operating environment in real time and obtain depth information, any other suitable sensor (e.g., stereo camera) and method described elsewhere herein may be used. In the exemplary process, runtime sensor data (e.g., 3D point cloud) may be captured as input 901. The runtime sensor data may include data captured by a lidar (light detection and ranging). The lidar may obtain three-dimensional information of the operating environment / scene by measuring the distance to an object. For example, a transmitting device of a lidar system may generate a sequence of light pulses emitted within a short time duration, so that the sequence of light pulses may be used to derive distance measurement points. The lidar system may provide three-dimensional (3D) imaging (e.g., 3D point cloud). In some cases, one or more processors of the robotic endoscope system may further process the 3D point cloud or 3D image for obstacle detection or collision avoidance 903. Various suitable image processing methods (e.g., image segmentation) may be used to identify objects such as a bed. Depth information from the 3D point cloud can be used to assign distances to points within the segmented region (e.g., bed area).

[0093] The position and orientation of the lidar sensor can be obtained based on the kinematic mapping, and this information, together with the depth information of the bed, is used to estimate the relative orientation and position between the bed and the robotic endoscope system. Based on the relative position and orientation, a movement path for moving the robotic cart to an optimal position relative to the bed can be generated. The optimal position can be automatically generated by the system without user intervention. In some cases, instead of or in addition to performing the robotic cart self-placement, the algorithm can also notify the user of an invalid position of the robotic cart relative to the bed, and the valid position of the robotic cart can be displayed to the user on the GUI. The user may be able to provide input via the GUI indicating the selection of a valid position for the robotic cart. Upon receiving the user confirmation, the system can generate a movement path for the robotic cart. Alternatively, the user can be guided to manually move the robotic cart to the desired position as described above.

[0094] As described above, any suitable method can be used to process real-time sensor data (e.g., 3D point clouds) for segmentation, object recognition, and collision avoidance. One or more objects can be segmented in the workspace of the robot arm. For example, deep learning techniques (e.g., automatic pipeline engines) can be provided to process lidar data. The pipeline engine may include multiple components or layers. The pipeline engine may be configured to preprocess a continuous stream of raw lidar data or batch data emitted from a lidar system. In some cases, the data may be processed so that it is fed into a machine learning analysis. In some cases, the data may be processed to provide details of different levels of understanding, which may include (non-limiting examples) size, weight, composition, identity, degree of collision risk, mobility, etc. In some cases, the pipeline engine may include multiple components to perform different functions to extract different levels of information from the 3D point cloud data. In some cases, the pipeline engine may also include basic data processing, such as data normalization, data tagging with metadata, labeling, data alignment, data segmentation, etc. In some cases, the processing method may be programmable by a developer who builds the pipeline through an API.

[0095] In some embodiments, the pipeline engine can use machine learning techniques to process data. In some embodiments, the raw lidar data can be provided to the first layer of the pipeline engine, which can use a deep learning architecture to extract one or more primitives of the target object, such as edges, corners, and surfaces. In some cases, the deep learning architecture can be a convolutional neural network (CNN). CNN systems typically consist of different types of layers: convolution, pooling, upsampling, and fully connected neural networks. In some cases, activation functions can be used in some layers, such as rectified linear units. In a CNN system, there can be one or more layers for each type of operation. The input data of the CNN system can be the data to be analyzed, such as 3D radar data. The simplest architecture of a convolutional neural network starts with an input layer (e.g., an image), followed by a series of convolutional layers and pooling layers, and finally a fully connected layer. In some cases, the convolutional layer is followed by a layer of ReLU activation function. Other activation functions can also be used, for example, saturated hyperbolic tangent, identity, binary step, logistic, inverse tangent, softsign, parametric rectified linear unit, exponential linear unit, softPlus, curved identity, softExponential, sine, Sinc, Gaussian, Sigmoid function, etc. Convolutional layers, pooling layers, and ReLU layers can act as learnable feature extractors, while fully connected layers act as machine learning classifiers.

[0096] In some cases, the convolutional layers and fully connected layers may include parameters or weights. These parameters or weights may be learned during a training phase. Gradient descent may be used to train the parameters so that the class scores calculated by the CNN are consistent with the labels in the training set for each 3D point cloud image. These parameters may be obtained from a back-propagation neural network training process, which may or may not be performed using the same hardware as the production or application process.

[0097] A convolutional layer may include one or more filters. When these filters see the same specific structure in the input data, they will activate. In some cases, the input data may be a 3D image, and in the convolutional layer, one or more filtering operations may be applied to the pixels of the image. A convolutional layer may include a set of learnable filters that slide spatially over the image, computing the dot product between the entries of the filter and the input image. The filtering operation may be implemented as a convolution of a kernel over the entire image. The kernel may include one or more parameters. The results of the filtering operations may be added together across channels to provide an output from the convolutional layer to the next pooling layer. A convolutional layer may perform high-dimensional convolutions. For example, a three-dimensional feature map or input 3D data is processed by a set of three-dimensional kernels in a convolutional layer.

[0098] The output produced by the first layer of the pipeline engine can be provided to the second layer, which is configured to extract understanding of the target object, such as shape, material, subsurface structure, etc. In some cases, the second layer can also be implemented using a machine learning architecture.

[0099] The output produced by the second layer can then be provided to the third layer of the pipeline engine, which is configured to perform interpretation and decision making, such as object recognition, separation, segmentation, collision avoidance, target dynamics (e.g., mobility), identity recognition, type classification, etc. In some cases, the dynamics or mobility of the object can be used to determine a collision avoidance solution.

[0100] The pipeline engine described herein can be implemented by one or more processors. In some embodiments, one or more processors can be programmable processors (e.g., central processing unit (CPU), graphics processing unit (GPU), general processing unit or microcontroller) in the form of fine-grained spatial architecture (e.g., field programmable gate array (FPGA), application specific integrated circuit (ASIC), and / or one or more advanced RISC machine (ARM) processors). In some embodiments, the processor can be a processing unit of a computer system.

[0101] In some cases, to mitigate the effects of noise, a Bayesian estimation technique (e.g., Kalman filtering) is applied to fit the point cloud to a rigid model of the bed to align the robotic cart with the bed. In some cases, if the target object (e.g., a patient bed) is not detected 907, the process can continue to notify the user that the robotic cart position is invalid (e.g., out of the appropriate area) 921. For example, if the bed is not detected, a notification can be displayed on the GUI to notify the user that the cart position is invalid.

[0102] If a target object (e.g., a bed) is detected 907, the method may perform model coordinate registration 909 by mapping the position and orientation of the bed model to the coordinates of the robot system (e.g., by pre-registration of the sensor and robot system coordinate systems). Next, the method may include an algorithm to determine whether the system is in an acceptable workspace. As an example, the algorithm may include calculating position and orientation errors 911 and comparing them to a threshold to determine whether the system is in an acceptable workspace.

[0103] In some cases, the system may provide an autonomous mode and a manual positioning mode. If the autonomous mode is once enabled (e.g., selected by a user) 913, the system may execute an active system positioning algorithm. The active system positioning algorithm may, for example, perform path planning 915 to generate a path from a current position to a target position, with appropriate movement speed, acceleration, etc. For example, generate or update a robot cart wheel path plan to eliminate the relative direction error between the desired direction and the sensed direction calculated in the previous operation 911. Execute the path 917 and control the robot wheel motion to move the robot cart to the desired direction and position.

[0104] If the system disables the active system positioning algorithm (e.g., in semi-automatic mode 913), the user can be notified of the validity status of the system positioning relative to the bed (e.g., acceptable or unacceptable position 919), and the user can manually control the position of the robotic cart.

[0105] Figure 10-12 An example of collision avoidance is shown. As described above, a real-time 3D map of the operating environment may include obstacle information and relative positions of one or more components of the robotic endoscope system. The system may determine a proximity threshold and may autonomously move one or more components of the system to avoid a collision. For example, Fig.10As shown, when the robotic arm is actuated during a surgical procedure, a sensor may detect an unwanted proximity between the arm and a system monitor 1000. The proximity is detected via analysis of sensor signals that may include depth information. In response to the proximity, a user may be notified of a potential collision via a UI warning. Alternatively or additionally, the robotic arm may automatically move away from the monitor, or the monitor may be actuated away from the arm 1010. For example, the monitor may be mounted to a robotic support that may be actuated to move the monitor in response to a control signal. As shown in example 1010, upon detection of a potential collision between the monitor and any portion of the robotic arm, the robotic support of the monitor and the robotic arm may be actuated away from each other. Alternatively, as shown in example 1010, the robotic support of the monitor and the robotic arm may be actuated away from each other. Fig.11 As shown, upon detection of a potential collision between the monitor and any portion of the robotic arm 1100 , the robotic arm may be actuated to move away from the monitor.

[0106] Fig.11 An example is shown when an operator moves a monitor into the workspace of the robotic arm, the system can detect that the proximity of the monitor is approaching a threshold and can automatically change the configuration of the robotic arm. The redundancy of the robotic arm can advantageously allow the configuration of the robotic arm to be changed while maintaining the position and orientation of the IDM. Fig.12 An example is shown of bringing a device such as a fluoroscopic C-arm 1207 into the workspace during a surgical procedure. The system can detect the device 1207 and the proximity to the device and then automatically adjust the configuration of the robotic arm and / or robotic cart to provide sufficient clearance for the C-arm to take fluoroscopic images of the patient and / or complete a tomosynthesis scan. The configuration of the robotic arm can change from a first configuration 1203 to a second configuration 1205 while the position and orientation of the IDM 1201 remains unchanged.

[0107] Fig.13 An example of a collision avoidance algorithm 1300 is shown. It should be noted that in the illustrated process, while LiDAR data (3D point cloud) is used to detect objects in the operating environment in real time and obtain depth information, any other suitable sensor (e.g., stereo camera) and method described elsewhere herein may be used. The steps of capturing 3D point cloud data and segmenting objects within the workspace to detect objects may be similar to the steps of FIG. Fig. 9 The steps described in . For example, the input data may include 3D point cloud data captured by a LiDAR device 1301, and deep learning techniques as described above may be used to perform object segmentation, object recognition 1303 to avoid collisions. For example, when an object (e.g., a display device, a monitor) is brought into the operating environment, a segmentation algorithm may be executed to detect and recognize the object. Depth information from the 3D point cloud may be used to assign distances to points within a segmented area (e.g., the area of ​​a monitor).

[0108] In some cases, signal filtering is performed to mitigate the effects of sensing uncertainty 1305. In some cases, when the detected object can be represented by a pre-computed model, a Bayesian filtering method, such as a Kalman filter, can be applied. For example, the pre-computed model can include an estimation algorithm for the object, where the sensing point cloud of the monitor is the input to the estimation algorithm, which estimates the state (position, orientation) of the monitoring device.

[0109] Next, the position and orientation of the point cloud are mapped to the arm-based coordinate system 1307. The positions of components of the robotic endoscopy system (e.g., the robotic arm) and the positions of points in the segmented monitor area can be registered in the same reference system to determine the minimum proximity between any part of the monitor and any part of the robotic arm (calculate the contact distance) 1309. The position and orientation of the robotic arm can be obtained based on the kinematic mapping.

[0110] In some cases, a proximity determination algorithm may be executed to determine a potential collision regardless of whether a minimum proximity violates a predetermined proximity threshold 1311. In some cases, upon determining that a predetermined proximity threshold is violated, the system may activate an admittance controller to reconfigure the configuration of the robot arm (e.g., by actuating one or more joints / links of the robot arm) 1313. For example, the admittance controller may execute an admittance control algorithm that defines the nearest collision point as a repulsive force and reconfigures the robot arm to increase the (minimum) distance between the robot arm and the monitor. An algorithm is executed to reposition the arm to avoid a collision before it occurs. In some cases, the admittance control algorithm may map a sensed proximity between any part of the robot arm and an object to a virtual force applied to the robot arm and calculate an output motion of the robot arm as if an equivalent contact force was applied to the robot arm (by contact). For example, the admittance control algorithm includes an input that maps the shortest distance between a link of the arm and an obstacle to a commanded task space velocity of the arm link. The geometric Jacobian matrix of the arm can be used to map task space motion commands to arm joint commands to achieve arm motions that increase the distance between the arm and obstacles.

[0111] If the degrees of freedom of the robotic system are greater than the degrees of freedom of the task, i.e., there is actuation redundancy, avoidance can be achieved via a redundancy resolution algorithm. To avoid, the elbow of the arm can be repositioned to prevent the collision without affecting the position and orientation of the end effector. Such an algorithm can be implemented by defining a secondary redundancy resolution task, in which the arm is repositioned so that the distance between the arm and the object (before the collision) increases. In some cases, when there is more than one degree of freedom in redundancy, the robotic commands can be selected to actuate the joints, for example, to minimize the joint velocity norm.

[0112] In some embodiments, the robotic endoscope system may be able to autonomously adjust the position of the IDM relative to the patient-side support based on buckling detection. As the flexible endoscope is advanced at the proximal end, the flexible endoscope may deform as it passes through turns and buckles during insertion of the flexible device into the anatomy. Deformation may occur during insertion as the flexible device may assume a minimum energy shape, which may be a shaft "snug" to the tissue. Buckling may occur when the distal portion of the shaft encounters resistance.

[0113] During retraction of a flexible device, when the system actuation direction is reversed (e.g., moving backward or retracting), the distance "lost to flexion" of the axis can become slack. This phenomenon will result in a perceptible dead zone or system delay (user input commanding the robotic endoscope tip to move does not map directly to the robotic endoscope tip motion). When the endoscope is flexed, retraction of the endoscope can result in robotic actuation with little translation of the endoscope tip.

[0114] Prolapse or kinking can cause potential damage as it can expose sharp edges of the kinked elongated device and complicate the surgical procedure. In addition, a bent or kinked elongated device can cause the system to lose position / shape control of the device during insertion and retraction and can block instrumentation passage. Furthermore, a prolapsed or kinked device may not provide adequate access to the target anatomy to perform the intended task.

[0115] The robotic endoscope system of this article can employ sensitive insertion and retraction speed control of a flexible endoscope. The responsive speed control method of this article can automatically correct for the difference in motion between the endoscope tip and the speed command (e.g., instrument drive mechanism (IDM) command). In some cases, the speed control of the endoscope can be performed in conjunction with the collision avoidance algorithm described above, so that the robotic arm can be autonomously reconfigured to avoid collisions with other objects, while the speed of the endoscope tip is maintained and unaffected.

[0116] Unlike conventional methods of buckling detection based on differences in endoscopic device position (e.g., expected position and measured tip position), the methods and systems herein can automatically correct for buckling / deformation during insertion and retraction based on velocity and velocity control commands measured at the endoscope tip. This advantageously avoids shape sensing or the use of additional imaging methods to determine the shape or position of the endoscopic device.

[0117] During the insertion of the endoscopic device, the endoscopic device can receive a speed command. The speed command can be provided by a user input. For example, a user can provide a control instruction via a control interface of the endoscopic device, indicating the desired / expected speed of the endoscope tip. The control interface can include various devices, such as a touch screen display, a joystick, a keyboard, and other interactive devices. The user can use the user input device to navigate and / or control the movement of the robot arm and the movement of the catheter (e.g., tip speed). The user input device can have any type of user interaction component, such as a button, a mouse, a joystick, a trackball, a touchpad, a pen, an image capture device, a motion capture device, a microphone, a touch screen, a handheld wrist gimbal, an exoskeleton glove, or other user interaction systems, such as a virtual reality system, an augmented reality system, etc.

[0118] User input for commanding the velocity of the endoscope tip may be received via an input device. For example, pressing a joystick may be mapped to an analog value indicating the velocity / speed of the tip. For example, half-pressing the joystick may be mapped to 3 mm / sec, fully pressing may be mapped to 6 mm / sec, and not pressing may be mapped to 0 mm / sec.

[0119] Based on a particular user input device, the user input may be processed to convert it into a desired / commanded velocity for the tip of the catheter / endoscope. Next, a tip velocity error is calculated. The tip velocity error is the difference between the desired / commanded tip velocity and the velocity of the endoscope tip. In some embodiments, the velocity of the endoscope tip may be measured based on sensor data. In some cases, the sensor data may include position and orientation information of the distal tip of the endoscope.

[0120] In some cases, the sensor signal can be acquired by a positioning sensor. For example, the sensor signal can be acquired by an electromagnetic coil located at the distal end, which is used with an electromagnetic tracking system to detect the position and orientation of the distal end of the endoscope. For example, a positioning sensor (e.g., an electromagnetic (EM) sensor) can be embedded in the distal tip of a catheter, and an EM field generator can be placed next to the patient's torso during the procedure. The EM field generator can locate the EM sensor position in 3D space, or can locate the EM sensor position and orientation in 5D or 6D space. The endoscope tip position measured by the EM sensor p e =EM sensor (tip) position, which can be represented in the field generator frame.

[0121] Next, the linear velocity of the endoscope tip can be calculated. The linear velocity can be calculated using a time derivative method (e.g., backward Euler derivative). In some cases, the endoscope tip velocity can be calculated as the filtered time derivative of the tip position projected in the forward direction (e.g., measured by an EM sensor represented in the field generator framework). This projection may be required because the velocity command provided by the user is based on the integrated position change occurring in the direction of n. In some cases, a low pass filter may be applied to generate the filtered time derivative data.

[0122] The endoscope tip velocity can be calculated using the projection matrix v n =nn T filt(dp e / dt) is the filtered time derivative of the tip position projected in the advancing direction of n (e.g., expressed in the field generator framework, measured by an EM sensor).

[0123] Where n is a unit vector indicating the direction of travel of the endoscope tip expressed in the field generator frame. There can be a mechanical offset between the EM sensor and the endoscope tip, and the mechanical offset can be calibrated for each endoscope device. dp e / dt represents the time derivative of the endoscope tip, nn T is the projection matrix that maps the above velocities to the direction of travel of the endoscope tip (i.e., ignoring velocities that are not in the direction of travel). n It is not affected by the joint because the endoscope tip translates due to the joint being orthogonal to n.

[0124] After calculating the endoscope tip velocity, the endoscope tip velocity error can be calculated and can be further processed for safety checks. In some cases, the safety check can include multiple checks. For example, multiple checks can include determining whether the endoscope tip is stationary (e.g., the tip velocity is approximately zero) while the insertion distance of the handle portion (e.g., IDM) of the endoscope exceeds a distance threshold. In another example, multiple checks can include determining whether the endoscope tip is retracted (e.g., a negative tip velocity error) when the handle portion (e.g., IDM) of the endoscope is inserted, and if so, prolapse may have occurred. In another example, multiple checks can include determining whether the insertion force exceeds a force threshold. The term "insertion distance" as used herein can refer to a distance along a navigation path.

[0125] The method may include closed-loop control of the tip speed to reduce the tip speed error. The tip speed of the endoscope can be controlled based on the tip speed error calculated at the operation, which is used as a feedback signal to command the movement of the IDM to perform closed-loop control of the tip speed. The user speed input can be mapped to a command to control the movement of the IDM (for example, the speed of moving the IDM along the insertion axis). The command can be a control signal to control the motor of the robotic arm, thereby controlling the movement of the IDM (i.e., the proximal end of the endoscope). The endoscope tip speed can be calculated based on the inverse kinematics of the robotic arm. The endoscope tip speed is then used to calculate the tip speed in the forward direction by projecting in the forward direction of n using the projection matrix as described above. In some cases, the feedback signal can be a projection of the tip speed processed by a low-pass filter.

[0126] Based on the control algorithm, motion commands are generated to activate the robotic arm, thereby affecting the tip velocity of the endoscope. Due to the tortuosity, buckling, and / or prolapse of the navigation path (as described above), the effects on the motion of the IDM (e.g., insertion speed, insertion distance) and the effects on the motion of the endoscope tip (e.g., tip velocity) may not be perfectly matched.

[0127] During retraction of the medical device, the previously determined deformation losses during insertion can be used as a feed-forward term to inversely compensate the retraction control. This reduces the deformation losses that occurred during insertion prior to the tip movement during retraction.

[0128] Fig.14 An example of a flexible endoscope 1400 according to some embodiments of the present disclosure is shown. Fig.14 As shown, the flexible endoscope 1400 may include a handle / proximal portion 1409 and a flexible elongated member to be inserted into the interior of the object. The flexible elongated member may be the same as the flexible elongated member described above. In some embodiments, the flexible elongated member may include a proximal shaft (e.g., an insertion shaft 1401), a manipulable tip (e.g., a tip 1405), and a manipulable segment (active bending segment 1403). The active bending segment and the proximal shaft segment may be the same as the active bending segment, the anti-prolapse passive segment, and the proximal shaft segment described elsewhere herein. Endoscope 1400 may also be referred to as a manipulable catheter assembly, as described elsewhere herein. In some cases, endoscope 1400 may be a single-use robotic endoscope. In some cases, the entire catheter assembly may be disposable. In some cases, at least a portion of the catheter assembly may be disposable. In some cases, the entire endoscope may be released from the instrument drive mechanism and may be discarded. In some embodiments, the endoscope may include different levels of stiffness along the axis to improve functional operation.

[0129] The endoscope or steerable catheter assembly 1400 may include a handle portion 1409 that may include one or more components configured to process image data, provide power, or establish communications with other external devices. For example, the handle portion may include circuitry and communication elements that enable electrical communication between the steerable catheter assembly 1400 and an instrument drive mechanism (not shown) and any other external systems or devices. In another example, the handle portion 1409 may include circuit elements, such as a power source for powering the electronics of the endoscope (e.g., a camera, an electromagnetic sensor, and an LED light).

[0130] One or more components located at the handle can be optimized so that expensive and complex components can be assigned to a robot support system, a handheld controller, or an instrument drive mechanism, thereby reducing costs and simplifying the design of a disposable endoscope. The handle portion or the proximal portion can provide an electrical and mechanical interface to allow electrical and mechanical communication with the instrument drive mechanism. The instrument drive mechanism can include a group of motors that are actuated to drive a group of pull wires of a catheter rotationally. The handle portion of the catheter assembly can be assembled to the instrument drive mechanism so that its pulley / capstan assembly is driven by the group of motors. The number of pulleys can vary based on the pull wire configuration. In some cases, one, two, three, four or more pull wires can be used to articulate a flexible endoscope or catheter.

[0131] The handle portion can be designed to allow the robotic bronchoscope to be disposable at a reduced cost. For example, a classic manual bronchoscope and a robotic bronchoscope can have a cable at the proximal end of the bronchoscope handle. The cable typically includes an illumination fiber, a camera video cable, and other sensor fibers or cables (such as an electromagnetic (EM) sensor or a shape sensing fiber). Such complex cables can be expensive, increasing the cost of the bronchoscope. The provided robotic bronchoscope can have an optimized design so that a simplified structure and components can be employed while retaining mechanical and electrical functionality. In some cases, the handle portion of the robotic bronchoscope can be designed without a cable while providing a mechanical / electrical interface for the catheter.

[0132] An electrical interface (e.g., a printed circuit board) can allow image / video data and / or sensor data to be received by a communication module of an instrument drive mechanism and can be transmitted to other external devices / systems. In some cases, the electrical interface can establish electrical communication without cables or wires. For example, the interface can include pins soldered to an electronic board such as a printed circuit board (PCB). For example, a socket connector (e.g., a female connector) is provided on the instrument drive mechanism as a mating interface. This can beneficially allow the endoscope to be quickly inserted into an instrument drive mechanism or a robot support without the use of additional cables. This type of electrical interface can also be used as a mechanical interface so that when the handle portion is inserted into the instrument drive mechanism, both mechanical coupling and electrical coupling are established. As an alternative or in addition, the instrument drive mechanism can only provide a mechanical interface. The handle portion can be electrically communicated with a modular wireless communication device or any other user device (e.g., a portable / handheld device or a controller) for transmitting sensor data and / or receiving control signals.

[0133] In some cases, the handle portion 1409 may include one or more mechanical control modules, such as a Luer interface 1411, for connecting an irrigation system / suction system. In some cases, the handle portion may include a lever / knob for articulation control. Alternatively, the articulation control may be located at a separate controller attached to the handle portion via the instrument drive mechanism.

[0134] The endoscope can be attached to a robotic support system or a handheld controller via an instrument drive mechanism. The instrument drive mechanism can be provided by any suitable controller device (e.g., a handheld controller), which may or may not include a robotic system. The instrument drive mechanism can provide a mechanical and electrical interface to the steerable catheter assembly 1400. The mechanical interface can allow the steerable catheter assembly 1400 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the steerable catheter assembly can be attached to the instrument drive mechanism via a quick installation / release tool (such as a magnet, a spring-loaded level, etc.). In some cases, the steerable catheter assembly can be manually coupled to the instrument drive mechanism or released from the instrument drive mechanism without the use of a tool.

[0135] In the illustrated example, the distal tip of the catheter or endoscope shaft is configured to articulate / bend in two or more degrees of freedom to provide a desired camera field of view or control the direction of the endoscope. As illustrated in this example, an imaging device (e.g., a camera), a position sensor (e.g., an electromagnetic sensor) 1407 is located at the tip of the catheter or endoscope shaft 1405. For example, the line of sight of the camera can be controlled by controlling the articulation of the active bending segment 1403. In some cases, the angle of the camera can be adjustable so that the line of sight can be adjusted without articulating the distal tip of the catheter or endoscope shaft or in addition to articulating the distal tip of the catheter or endoscope shaft. For example, the camera can be oriented at a certain angle (e.g., tilted) relative to the axial direction of the endoscope tip by means of an optical assembly.

[0136] The distal tip 1405 can be a rigid component that allows sensors (such as electromagnetic (EM) sensors), imaging devices (e.g., cameras), and other electronic components (e.g., LED light sources) to be positioned embedded at the distal tip.

[0137] In real-time EM tracking, an EM sensor consisting of one or more sensor coils in one or more locations and orientations embedded in a medical device (e.g., the tip of an endoscopic tool) measures changes in the EM field generated by one or more static EM field generators positioned at locations close to the patient. The position information detected by the EM sensor is stored as EM data. An EM field generator (or transmitter) can be placed close to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. For example, an EM field generator can be positioned close to the patient's torso during surgery to locate the EM sensor position in 3D space or can locate the EM sensor position and orientation in 5D or 6D space. This can provide visual guidance to the operator when driving the bronchoscope toward the target site.

[0138] The endoscope can have a unique design in the elongated members. In some cases, the active bending section 1403 and the proximal shaft of the endoscope can be composed of a single tube that includes a series of cutouts (e.g., notches, slits, etc.) along its length to allow for improved flexibility, desired stiffness, and anti-prolapse features (e.g., features for defining a minimum bend radius).

[0139] As described above, the active bending segment 1403 can be designed to allow bending (e.g., articulation) in two or more degrees of freedom. Greater degrees of bending, such as 180 degrees and 270 degrees (or other articulation parameters for clinical indications) can be achieved through the unique structure of the active bending segment. In some cases, a variable minimum bending radius along the axial axis of the slender member can be provided, so that the active bending segment or the passive segment can include two or more different minimum bending radii.

[0140] The articulation of the endoscope can be controlled by applying force to the distal end of the endoscope via one or more pull wires. One or more pull wires can be attached to the distal end of the endoscope. In the case of multiple pull wires, pulling one wire at a time can change the orientation of the distal tip to tilt it upward, downward, left, right, or in any desired direction. In some cases, the pull wire can be anchored at the distal tip of the endoscope, travel through the curved segment, and enter the handle where the pull wire is coupled to a drive assembly (e.g., a pulley). The handle pulley can interact with an output shaft from a robotic system.

[0141] In some embodiments, the proximal end or proximal portion of one or more pull wires can be operably coupled to various mechanisms (e.g., gears, pulleys, capstans, etc.) in the handle portion of the catheter assembly. The pull wire can be a metal wire, cable or filament, or it can be a polymer wire, cable or filament. The pull wire can also be made of natural or organic materials or fibers. The pull wire can be any type of suitable wire, cable or filament that can support various loads without deformation, significant deformation or breakage. The distal end / distal portion of one or more pull wires can be anchored or integrated into the distal portion of the catheter so that the control unit's operation of the pull wire can apply a force or tension to the distal portion, which can manipulate or articulate (e.g., upward, downward, pitch, yaw or any direction therebetween) at least the distal portion (e.g., flexible segment) of the catheter.

[0142] The pull wire can be made of any suitable material, such as stainless steel (e.g., SS316), a metal, an alloy, a polymer, nylon, or a biocompatible material. The pull wire can be a wire, a cable, or a filament. In some embodiments, different pull wires can be made of different materials to change the load-bearing capacity of the pull wire. In some embodiments, different segments of the pull wire can be made of different materials to change the stiffness and / or load-bearing capacity along the pull wire. In some embodiments, the pull wire can be used for the transmission of electrical signals.

[0143] The proximal design can improve the reliability of the device without introducing additional costs, thereby allowing a low-cost single-use endoscope. In another aspect of the present invention, a single-use robotic endoscope is provided. The robotic endoscope can be a bronchoscope and can be the same as the steerable catheter assembly described elsewhere herein. Conventional endoscopes can be complex in design and are typically designed to be reused after surgery, which requires thorough cleaning, disinfection, or sterilization after each surgery. Existing endoscopes are typically designed with complex structures to ensure that the endoscope can withstand the cleaning, disinfection, and sterilization processes. The provided robotic bronchoscope can be a single-use endoscope, which can beneficially reduce cross-contamination between patients and infected persons. In some cases, the robotic bronchoscope can be delivered to a medical practitioner in a pre-sterilized package and is intended to be discarded after a single use.

[0144] like Fig.15 As shown, the robotic bronchoscope 1520 may include a handle portion 1513 and a flexible elongated member 1511. In some embodiments, the flexible elongated member 1511 may include a shaft, a steerable tip, and a steerable / active bending section. The robotic bronchoscope 1520 may be used with Fig.14 The robotic bronchoscope may be a single-use robotic endoscope. In some cases, only the catheter may be disposable. In some cases, at least a portion of the catheter may be disposable. In some cases, the entire robotic bronchoscope may be released from the instrument drive mechanism and may be discarded. In some cases, the bronchoscope may include different levels of stiffness along its axis to improve functional operation. In some cases, the minimum bend radius along the axis may vary.

[0145] The robotic bronchoscope can be releasably coupled to an instrument drive mechanism 1520. The instrument drive mechanism 1520 can be mounted to an arm of a robotic support system, or can be mounted to any actuated support system as described elsewhere herein. The instrument drive mechanism can provide a mechanical and electrical interface to the robotic bronchoscope 1510. The mechanical interface can allow the robotic bronchoscope 1510 to be releasably coupled to the instrument drive mechanism. For example, a handle portion of the robotic bronchoscope can be attached to the instrument drive mechanism via a quick mount / release tool (such as a magnet and a spring-loaded level). In some cases, the robotic bronchoscope can be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0146] Fig.16An example of an instrument drive mechanism (IDM) 1620 is shown, which provides a mechanical interface to a handle portion 1613 of a robotic bronchoscope. As shown in this example, the instrument drive mechanism 1620 can include a set of motors that are actuated to rotationally drive a set of pull wires of a flexible endoscope or catheter. The handle portion 1613 of the catheter assembly can be assembled to the instrument drive mechanism so that its pulley assembly or capstan is driven by the set of motors. The number of pulleys can vary based on the pull wire configuration. In some cases, one, two, three, four or more pull wires can be used to articulate a flexible endoscope or catheter.

[0147] The handle portion can be designed to allow the robotic bronchoscope to be disposable at a reduced cost. For example, a classic manual bronchoscope and a robotic bronchoscope can have a cable at the proximal end of the bronchoscope handle. The cable typically includes an illumination fiber, a camera video cable, and other sensor fibers or cables (such as an electromagnetic (EM) sensor or a shape sensing fiber). Such complex cables can be expensive, increasing the cost of the bronchoscope. The provided robotic bronchoscope can have an optimized design so that a simplified structure and components can be employed while retaining mechanical and electrical functionality. In some cases, the handle portion of the robotic bronchoscope can be designed without a cable while providing a mechanical / electrical interface for the catheter.

[0148] Fig.17 An example of a distal tip 1700 of an endoscope is shown. In some cases, the distal portion or tip of the catheter 1700 can be substantially flexible so that it can be manipulated in one or more directions (e.g., pitch, yaw). The catheter can include a tip portion, a curved segment, and an insertion axis. In some embodiments, the catheter can have a variable bending stiffness along the longitudinal axis. For example, the catheter can include multiple segments with different bending stiffnesses (e.g., flexible, semi-rigid, and rigid). The bending stiffness can be changed by selecting materials with different stiffness / rigidity, changing the structure in different sections (e.g., incisions, patterns), adding additional support components, or any combination of the above. In some embodiments, the catheter can have a variable minimum bending radius along the longitudinal axis. Selecting different minimum bending radii at different locations along the catheter can be beneficial to provide anti-prolapse capabilities while still allowing the catheter to reach difficult-to-reach areas. In some cases, the proximal end of the catheter does not need to be highly curved, so the proximal portion of the catheter can be reinforced with additional mechanical structures (e.g., additional material layers) to achieve greater bending stiffness. This design can provide support and stability for the catheter. In some cases, variable bending stiffness can be achieved by using different materials during extrusion of the catheter. This can advantageously allow different stiffness levels along the axis of the catheter during the extrusion manufacturing process without requiring additional fastening or assembly of the different materials.

[0149] The distal portion of the catheter can be manipulated by one or more pull wires 1705. The distal portion of the catheter can be made of any suitable material (such as a copolymer, polymer, metal or alloy) so that it can be bent by the pull wire. In some embodiments, the proximal or terminal end of the one or more pull wires 1705 can be coupled to a drive mechanism (e.g., a gear, pulley, capstan, etc.) via an anchoring mechanism as described above.

[0150] The pull wire 1705 can be a metal wire, cable or silk, or it can be a polymer wire, cable or silk. The pull wire 1705 can also be made of natural or organic materials or fibers. The pull wire 1705 can be any type of suitable wire, cable or silk that can support various loads without deformation, significant deformation or breakage. The distal end or distal portion of one or more pull wires 1705 can be anchored or integrated into the distal portion of the catheter so that the control unit's operation of the pull wire can apply a force or tension to the distal portion, which can manipulate or articulate (e.g., up, down, pitch, yaw or in any direction therebetween) at least the distal portion (e.g., the flexible segment) of the catheter.

[0151] The size of the catheter can enable one or more electronic components to be integrated into the catheter. For example, the outer diameter of the distal tip can be around 4 to 4.4 millimeters (mm), and the diameter of the working channel can be around 2 mm, so that one or more electronic components can be embedded in the wall of the catheter. However, it should be noted that based on different applications, the outer diameter can be in any range less than 4 mm or greater than 4.4 mm, and the diameter of the working channel can be in any range depending on the tool size or specific application.

[0152] One or more electronic components may include an imaging device, an illumination device, or a sensor. In some embodiments, the imaging device may be a video camera 1713. The imaging device may include an optical element and an image sensor for capturing image data. The image sensor may be configured to generate image data in response to the wavelength of light. Various image sensors such as complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) may be used to capture image data. The imaging device may be a low-cost camera. In some cases, the image sensor may be provided on a circuit board. The circuit board may be an imaging printed circuit board (PCB). The PCB may include a plurality of electronic components for processing image signals. For example, a circuit for a CCD sensor may include an A / D converter and an amplifier to amplify and convert an analog signal provided by the CCD sensor. Alternatively, the image sensor may be integrated with an amplifier and a converter to convert an analog signal into a digital signal so that a circuit board may not be required. In some cases, the output of the image sensor or circuit board may be image data (digital signal) that may be further processed by a camera circuit or a processor of the camera. In some cases, the image sensor may include an array of optical sensors.

[0153] The lighting device may include one or more light sources 1711 located at the distal tip. The light source may be a light emitting diode (LED), an organic LED (OLED), a quantum dot, or any other suitable light source. In some cases, the light source may be a small LED or dual tone flash LED lighting for compact designs.

[0154] The imaging device and the lighting device may be integrated into the catheter. For example, the distal portion of the catheter may include a suitable structure that matches at least one size of the imaging device and the lighting device. The imaging device and the lighting device may be embedded in the catheter. Fig.18 An example distal portion of a catheter with an integrated imaging device and lighting device is shown. A camera may be located at the distal portion. The distal tip may have a structure for accommodating a camera, lighting device, and / or a position sensor. For example, a camera may be embedded in a cavity 1810 at the distal tip of the catheter. Cavity 1810 may be integrally formed with the distal portion of the cavity and may have a size that matches the length / width of the camera so that the camera may not move relative to the catheter. The camera may be adjacent to the working channel 1820 of the catheter to provide a near-field view of a tissue or organ. In some cases, the attitude or orientation of the imaging device may be controlled by controlling the rotational movement (e.g., rolling) of the catheter.

[0155] The power of the camera can be provided by a wired cable. In some cases, the cable line can be in a harness, which provides power for the lighting element or other circuit at the distal tip of the camera and the catheter. The camera and / or light source can be powered by a power supply located at the handle portion via a wire, a copper wire, or any other suitable tool that travels through the length of the catheter. In some cases, the real-time image or video of the tissue or organ can be wirelessly transmitted to an external user interface or display. Wireless communication can be WiFi, Bluetooth, RF communication, or other forms of communication. In some cases, the image or video captured by the camera can be broadcast to multiple devices or systems. In some cases, the image and / or video data from the camera can be transmitted to the processor located in the handle portion via a wire, a copper wire, or any other suitable tool along the length of the catheter. The image or video data can be transmitted to an external device / system via the wireless communication component in the handle portion. In some cases, the system can be designed to make the line invisible to the operator or the line is not exposed to the operator.

[0156] In conventional endoscopy, the illumination light can be provided by a fiber optic cable that transmits the light of a light source located at the proximal end of the endoscope to the distal end of the robotic endoscope. In some embodiments of the present disclosure, a small LED lamp can be used and embedded in the distal portion of the catheter to reduce design complexity. In some cases, the distal portion may include a structure 1430 having a size that matches the size of the small LED light source. As shown in the illustrated example, two cavities 1430 can be formed integrally with the catheter to accommodate two LED light sources. For example, the outer diameter of the distal tip can be about 4 to 4.4 millimeters (mm), and the diameter of the working channel of the catheter can be about 2mm, so that two LED light sources can be embedded at the distal end. The outer diameter can be in any range less than 4mm or greater than 4.4mm, and the diameter of the working channel can be in any range according to the size of the tool or the specific application. Any number of light sources can be included. The internal structure of the distal portion can be designed to accommodate any number of light sources.

[0157] In some cases, each LED can be connected to a power cord that can extend to the proximal handle. In some embodiments, the LED can be welded to separate power cords that are subsequently bundled together to form a single strand. In some embodiments, the LED can be welded to a pull cord that supplies power. In other embodiments, the LED can be crimped or directly connected to a single pair of power cords. In some cases, a protective layer such as a thin layer of biocompatible glue can be applied to the front surface of the LED to provide protection while allowing light to be emitted. In some cases, an additional cover 1431 can be placed at the forward end face of the distal tip to provide precise positioning of the LED and sufficient space for glue. The cover 1831 can be composed of a transparent material that matches the refractive index of the glue so that the illumination light can be unblocked.

[0158] Although preferred embodiments of the present invention have been shown and described herein, it will be readily understood to those skilled in the art that such embodiments are provided only as examples. Without departing from the present invention, those skilled in the art will now conceive of many variations, changes and substitutions. It should be understood that various alternatives to the embodiments of the present invention described herein may be adopted when implementing the present invention. The appended claims are intended to define the scope of the present invention, and thus cover methods and structures within the scope of these claims and their equivalents.

Claims

1. A method for controlling a robotic endoscope system, the method include: generating a 3D depth map of the environment surrounding the robotic endoscope system; autonomously actuating a self-propelled base of a robotic support system to a desired position relative to a patient bed based on the 3D depth map, wherein the robotic support system comprises a robotic arm coupled to the self-propelled base at a proximal end and to an instrument drive mechanism (IDM) at a distal end; as well as The robotic arm is actuated to autonomously align the IDM with a component coupled to or part of the patient bed. 2 . The method of claim 1 , wherein the 3D depth map is generated at least in part based on 3D point cloud data.

3. The method of claim 2, further comprising processing the 3D depth map to detect the patient bed and calculate the position and orientation of the robotic support system relative to the patient bed.

4. The method of claim 1, wherein a flexible endoscopic device is releasably coupled to the IDM after the IDM is aligned with the component coupled to or as part of the patient bed.

5. The method of claim 1, further comprising controlling movement of the robotic arm to move the IDM to a predetermined distance from the component coupled to or as part of the patient bed.

6. The method of claim 5, further comprising loading a flexible endoscopic device to couple to the IDM at a proximal end and to the assembly at a distal end.

7. The method of claim 5, further comprising automatically adjusting a position of the IDM relative to the assembly when a buckling event is detected.

8. The method of claim 1, further comprising detecting and identifying objects in the environment and reconfiguring the robotic arm to avoid collision with the objects while maintaining the position and orientation of the IDM.

9. The method of claim 8, further comprising detecting buckling of a flexible catheter coupled to the IDM while the flexible catheter is inserted into a patient.

10. The method of claim 9, further comprising executing a responsive velocity control algorithm to control the velocity of the tip of the flexible catheter while reconfiguring the robotic arm to avoid collision with the object.

11. The method of claim 1 , wherein the IDM autonomously aligns with the component based at least in part on sensor data. The method of claim 11 , wherein the sensor data is captured by an electromagnetic sensor.

13. The method of claim 11, wherein the sensor data is captured by a camera that includes a fiducial marker placed on the component, and wherein the 3D depth map includes at least the 3D position of the fiducial marker.

14. A system for controlling a robotic endoscope system, the system include: a memory storing computer executable instructions; one or more processors in communication with the robotic endoscopic system and configured to execute the computer-executable instructions to: generating a 3D depth map of the environment surrounding the robotic endoscope system; autonomously actuating a self-propelled base of a robotic support system to a desired position relative to a patient bed based on the 3D depth map, wherein the robotic support system comprises a robotic arm coupled to the self-propelled base at a proximal end and to an instrument drive mechanism (IDM) at a distal end; as well as The robotic arm is actuated to autonomously align the IDM with a component coupled to or part of the patient bed.

15. The system of claim 14, wherein the 3D depth map is generated based at least in part on 3D point cloud data.

16. The system of claim 14, wherein the one or more processors are further configured to process the 3D depth map to detect the patient bed and calculate a position and orientation of the robotic support system relative to the patient bed.

17. The system of claim 14, wherein a flexible endoscopic device is releasably coupled to the IDM after the IDM is aligned with the component coupled to or as part of the patient bed.

18. The system of claim 14, wherein the one or more processors are further configured to control movement of the robotic arm to move the IDM to a predetermined distance from the component coupled to or as part of the bed.

19. The system of claim 18, wherein the one or more processors are further configured to load a flexible endoscopic device to couple to the IDM at a proximal end and to couple to the assembly at a distal end.

20. The system of claim 18, wherein the one or more processors are further configured to automatically adjust a position of the IDM relative to the assembly when a buckling event is detected.

21. The system of claim 14, wherein the one or more processors are further configured to detect and identify objects in the environment and reconfigure the robotic arm to avoid collision with the objects while maintaining a position and orientation of the IDM.

22. The system of claim 21, wherein the one or more processors are further configured to detect buckling of a flexible catheter coupled to the IDM when the flexible catheter is inserted into a patient.

23. The system of claim 22, wherein the one or more processors are further configured to execute a responsive velocity control algorithm to control the velocity of the tip of the flexible catheter while reconfiguring the robotic arm to avoid collision with the object.

24. The system of claim 14, wherein the IDM autonomously aligns with the component based at least in part on sensor data.

25. The system of claim 14, wherein the sensor data is captured by an electromagnetic sensor.

26. The system of claim 14, wherein the sensor data is captured by a camera that includes a fiducial marker placed on the component, and wherein the 3D depth map includes at least the 3D position of the fiducial marker.

Citation Information

Patent Citations

  • System for robotic-assisted endolumenal surgery and related methods

    CN105939647A

  • Navigation of tubular networks

    CN108778113A

  • Flexible robot operation system for endoscopic treatment and using method thereof

    CN109431603A

  • Patient introducer alignment

    CN110602976A

  • Surgical robot adjusting system and method, medium and terminal

    CN112263332A