System and method for self-alignment and adjustment of a robotic endoscope

Abstract

A method for a robotic endoscope system is provided. The method includes generating a 3D depth map of the environment surrounding the robotic endoscope system, autonomously actuating a mobile base of a robotic support system to a desired location relative to a patient bed based on the 3D depth map, where the robotic support system includes a robotic arm coupled to the mobile base at a proximal end and coupled to a flexible endoscope device via an instrument drive mechanism (IDM) at a distal end, and actuating the robotic arm to align the IDM with a component coupled to the patient bed or that is part of the patient bed.

Description

Technical Field

[0001] (Cross-reference)

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 345,287, filed May 24, 2022, and U.S. Provisional Patent Application No. 63 / 347,179, filed May 31, 2022, each of which is hereby incorporated by reference in its entirety.

Background Art

[0002]

[0002] In endoscopic procedures, the interior of a hollow organ or body cavity is examined using an endoscope. Unlike many other medical imaging techniques, an endoscope is inserted directly into an organ. Flexible endoscopes that can provide intuitive manipulation and control are useful for the diagnosis and treatment of diseases accessible through any natural orifice of the body. Depending on the clinical application, an endoscope may be designated as a bronchoscope, ureteroscope, colonoscope, gastroscope, otolaryngology endoscope, and various others. For example, a flexible bronchoscope may be used for the diagnosis and / or surgical treatment of lung cancer. However, one of the challenges in bronchoscopy is reaching the upper lobe of the lung while passing through the airway. In another example, flexible endoscopy is used to examine and treat diseases of the gastrointestinal (GI) tract without the need to create an opening in the patient's body. The endoscope is introduced into the upper or lower GI tract from the mouth or anus, respectively. A small camera at the distal end captures images of the digestive tract wall that assist the clinician in diagnosing GI tract diseases. Simple surgical procedures (such as polyp resection and biopsy) can be performed by introducing a flexible tool through the working channel and reaching the distal end to the site of interest.

[0003]

[0003] Conventionally, endoscopes are manufactured to be reusable and may require thorough cleaning, disinfection, and / or sterilization after each procedure. In most cases, cleaning, disinfection, and sterilization may be an aggressive process to kill bacteria and / or germs. Such procedures can also be harsh on the endoscope itself. Therefore, the design of such reusable endoscopes often becomes complex, especially to ensure that the endoscope can withstand such harsh cleaning, disinfection, and sterilization protocols. Regular maintenance and repair of such reusable endoscopes are often required.

[0004]

[0004] Low-cost disposable medical devices designated for single use are becoming popular as an alternative to instruments that are difficult to clean properly. Single-use disposable devices may be packaged in a sterile wrapper to avoid the risk of cross-contamination of pathogenic diseases such as HIV, hepatitis, and other pathogens. Hospitals generally welcome the convenience of single-use disposable products because they no longer need to worry about product lifespan, excessive use, breakage, malfunction, and sterilization. Conventional endoscopes often include a handle that an operator uses to manipulate the endoscope. In the case of a single-use endoscope, the handle typically encloses a camera, expensive electronics, and a mechanical structure at the proximal end to transmit images and enable the user to operate the endoscope via a user interface. This can make the handle for a single-use endoscope costly.

[0005]

[0005] The setup process of a medical robot can be time-consuming and difficult because there are many accessories to set up and the workflow is complex. Due to the setup time, the room turnover time becomes long, and there may be delays in the procedures. Examples of difficult steps in the current system workflow are (1) positioning the robot system in a location suitable for the procedure, and (2) aligning the instrument drive mechanism to the patient side. These steps are difficult because the user needs to consider the arrangement of other equipment in the room required for the procedure and spatially understand the working space of the robot components while manually performing these steps. If there are changes in the room setup during the procedure, the user needs to manually adjust the system accordingly.

Summary of the Invention

[0006]

[0006] This specification recognizes the need for a robotic endoscope platform or system that enables autonomous self-adjustment of a robotic endoscope system according to a real-time operating environment. The present disclosure addresses the above need by providing a method and system that can detect and track the operating environment of the system (e.g., the external environment surrounding the system) and automatically adjust the system, thereby (1) simplifying the operator's workflow during system setup and / or during treatment, and (2) enabling an optimal system configuration compared to a manually completed arrangement that may be acceptable but not optimal. In some embodiments, the autonomous process and / or movement of the system may include alignment of the instrument drive mechanism (IDM) of the robotic endoscope device to the patient-side mount, alignment of the robot-based station (e.g., robotic cart) to the patient or hospital suite equipment, automatic configuration for recognition and compatibility of other technologies / devices, collision avoidance between the system and the patient and / or other objects in the operating environment, automatic adjustment of the robotic arm or IDM based on real-time buckling detection of the instrument, and monitoring of the patient's respiration / vitals, etc.

[0007]

[0007] In one aspect of the present disclosure, a method for a robotic endoscope system is provided. The method includes generating a 3D depth map of the environment surrounding the robotic endoscope system, autonomously operating a mobile base of a robotic support system to a desired location relative to a patient bed based on the 3D depth map, where the robotic support system includes a robotic arm coupled to the mobile base at a proximal end and coupled to a flexible endoscope device via an instrument drive mechanism (IDM) at a distal end, operating to autonomously operate the arm for collision avoidance, and operating the robotic arm to align the IDM with a component coupled to or part of the patient bed. The method implemented to enable autonomous repositioning of system components given 3D depth map input requires algorithms for processing and filtering image data, algorithms for mapping the image data to an input of a robotic control algorithm, and algorithms for obstacle removal.

[0008]

[0008] Also recognized herein are devices and systems with endoscopes that may be disposable and may not require extensive cleaning procedures. The present disclosure provides low-cost single-use articulatable endoscopes for diagnostic and therapeutic use in bronchoscopy, urology, gynecology, arthroscopy, orthopedics, otolaryngology, gastrointestinal endoscopy, neurosurgery, and various other applications. In some embodiments, the present disclosure provides a single-use disposable robotic control bronchoscope for use with a robotic system that enables diagnostic evaluation of lesions at any location in the anatomical structure of the lung. It should be noted that the provided endoscope system can be used in various minimally invasive surgical procedures, treatments or diagnostic procedures involving various types of tissues including the heart, bladder, lung tissue, and other anatomical regions of the patient's body such as the digestive system including but not limited to the esophagus, liver, stomach, colon, urinary tract, or the respiratory system including but not limited to the bronchi, lungs, etc.

[0009]

[0009] The provided autonomous configuration, alignment, and collision avoidance methods, endoscopic components, and various components of the device can be used in various minimally invasive surgical procedures, treatments, or diagnostic procedures, including various types of tissues including heart, bladder, and lung tissues, and in other anatomical regions of the patient's body such as the digestive system including but not limited to the esophagus, liver, stomach, colon, urinary tract, or the respiratory system including but not limited to the bronchus, lung, etc. It should be noted that they can be used in these areas.

[0010]

[0010] In one aspect, a method for controlling a robotic endoscopic system that moves and operates within an environment is provided. This method includes generating a 3D depth map of the environment surrounding the robotic endoscopic system, and based on the 3D depth map, autonomously actuating the mobile base of the robotic support system to a desired location relative to the patient bed, where the robotic support system includes a robotic arm coupled to the mobile base at the proximal end and an instrument drive mechanism (IDM) at the distal end, and actuating the robotic arm to autonomously align the IDM with a component coupled to or part of the patient bed.

[0011]

[0011] Related but in another aspect, a system for controlling a robotic endoscopic system is provided. This system includes a memory storing computer-executable instructions, and one or more processors configured to communicate with the robotic endoscopic system and execute the computer-executable instructions to generate a 3D depth map of the environment surrounding the robotic endoscopic system, and based on the 3D depth map, autonomously actuate the mobile base of the robotic support system to a desired location relative to the patient bed, where the robotic support system includes a robotic arm coupled to the mobile base at the proximal end and an instrument drive mechanism (IDM) at the distal end, and actuate the robotic arm to autonomously align the IDM with a component coupled to or part of the patient bed.

[0012]

[0012] In some embodiments, the 3D depth map is generated at least in part based on 3D point cloud data. Optionally, the method further includes processing the 3D depth map to detect a patient bed and calculating the position and orientation of the robotic support system relative to the patient bed. In some embodiments, the IDM is releasably coupled to a component that is coupled to or is part of the patient bed after the flexible endoscopic device is aligned with the component.

[0013]

[0013] In some embodiments, the method further includes controlling the movement of the robotic arm to move the IDM to a predetermined distance from a component that is coupled to or is part of the patient bed. Optionally, the method further includes loading a flexible endoscopic device that is coupled to the IDM at the proximal end and to the component at the distal end. Optionally, the method further includes automatically adjusting the position of the IDM relative to the component upon detection of a buckling event.

[0014]

[0014] In some embodiments, the method further includes detecting and recognizing objects in the environment and reconfiguring the robotic arm to avoid collisions with the objects while maintaining the position and orientation of the IDM. Optionally, the method further includes detecting buckling of a flexible catheter coupled to the IDM while the flexible catheter is inserted into the patient's body. In some examples, the method further includes executing a response speed control algorithm to control the speed of the tip of the flexible catheter while reconfiguring the robotic arm to avoid collisions with the objects.

[0015]

[0015] In some embodiments, the IDM is autonomously aligned with the component based at least in part on sensor data. Optionally, the sensor data is captured by an electromagnetic sensor. Optionally, the sensor data is captured by a camera that includes fiducial markers disposed on the component, and the 3D depth map includes at least the 3D positions of the fiducial markers.

[0016]

[0016] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in the art from the following detailed description, which illustrates only exemplary embodiments of the present disclosure. As will be understood, the present disclosure is capable of other different embodiments and some of the details thereof are capable of modifications in various obvious respects without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.

[0017] (Incorporated by reference)

[0017] All publications, patents, and patent applications mentioned in this specification are hereby incorporated by reference into this specification to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated as being incorporated by reference. To the extent that the incorporated publications and patents or patent applications conflict with the disclosure contained herein, this specification is intended to supersede and / or take precedence over such conflicting matters.

Brief Description of the Drawings

[0018]

[0018] The novel features of the present invention are set forth in detail in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description, which illustrates exemplary embodiments in which the principles of the present invention are utilized, and to the accompanying drawings (also referred to herein as "Figure" and "FIG.").

[0019]

Figure 1

[0019] A robot platform according to some embodiments of the present disclosure is schematically shown.

Figure 2

[0020] An example of a robotic catheter assembly with a buckling prevention device is shown.

Figure 3

[0021] An example of a robotic endoscope (e.g., bronchoscope) system capable of performing autonomous collision avoidance and self-alignment before and during surgery is shown.

Figure 4A

[0022] An example of constructing a 3D map of the operating environment using an optical sensor is shown.

Figure 4B

[0023] An example of automatic alignment of an IDM (instrument drive mechanism) based on EM sensor data is shown.

Figure 4C

[0023] An example of automatic alignment of an IDM (instrument drive mechanism) based on EM sensor data is shown.

Figure 5

[0024] An example of autonomous alignment of an IDM (instrument drive mechanism) with respect to a patient-side mount is shown.

Figure 6

[0024] An example of autonomous alignment of an IDM (instrument drive mechanism) with respect to a patient-side mount is shown.

Figure 7

[0025] An example of a self-propelled (e.g., via one or more propulsion units such as wheels, rotors, propellers) robotic cart that autonomously positions itself with respect to a patient bed is shown.

Figure 8

[0026] An example of self-alignment of a robotic endoscope system is shown.

Figure 9

[0027] An exemplary process of autonomous alignment of a robotic endoscope system is shown.

Figure 10 - 12

[0028] Examples of collision avoidance of a robotic endoscope system with respect to one or more objects in the operating environment before and during surgery are shown.

Figure 11

[0029] An example of a robotic bronchoscope having a handle portion and a flexible elongated member is shown.

Figure 12

[0030] An example of an instrument drive mechanism that provides a mechanical interface with the handle portion of a robotic bronchoscope is shown.

Figure 13

[0031] An example of a collision avoidance algorithm is shown.

Figure 14

[0032] An example of a flexible endoscope is shown.

Figure 15

[0032] An example of a flexible endoscope is shown.

Figure 16

[0033] An example of an instrument drive mechanism that provides a mechanical interface with the handle portion of a robotic bronchoscope is shown.

Figure 17

[0034] An example of the distal tip of an endoscope is shown.

Figure 18

[0035] An exemplary distal portion of a catheter in which an imaging device and an illumination device are integrated is shown.

Best Mode for Carrying Out the Invention

[0020]

[0036] Although various embodiments of the present invention are shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will envision numerous variations, modifications, and substitutions without departing from the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be used.

[0021]

[0037] The embodiments disclosed herein can be combined in one or more ways among many ways to provide improved diagnosis and treatment to patients. The disclosed embodiments can be combined with existing methods and devices, for example, in combinations with known methods of lung diagnosis, surgery, and surgery of other tissues and organs, to provide improved treatment. Any one or more of the structures and processes described herein can be combined with any one or more additional structures and processes of the methods and devices described herein, and it should be understood that the drawings and the supporting text provide an explanation according to the embodiments.

[0022]

[0038] Exemplary embodiments are primarily directed to devices or systems for bronchoscopy, but those skilled in the art will understand that this is not intended to be limiting. Also, the devices described herein may be used in various anatomical regions of a patient's body for other therapeutic or diagnostic procedures. The devices or systems provided can be used in urology, gynecology, rhinology, otology, laryngoscopy, gastroenterology using an endoscope, a composite device including an endoscope and an instrument, or an endoscope with a localization function. Those skilled in the art will understand that this is not intended to be limiting. Also, the devices described herein, in combination with various tools or instruments, can be in the form of a neuroendoscope, a brain endoscope, an ophthalmoscope, an otoscope, a nasal endoscope, a laryngoscope, a gastroscope, an esophagoscope, a bronchoscope, a thoracoscope, a pleuroscope, an angioscope, a mediastinoscope, a nephroscope, a gastroscope, a duodenoscope, a cholangioscope, a bile duct scope, a laparoscope, an amnioscope, a ureteroscope, a hysteroscope, a cystoscope, a proctoscope, a colonoscope, an arthroscope, a salivary duct endoscope, an orthopedic endoscope, and others, for other therapeutic or diagnostic procedures in other anatomical regions of a patient's body, such as the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testes, bladder, ears, nose, mouth, bone marrow, adipose tissue, muscle, gland and mucosal tissue, spinal cord and nerve tissue, soft tissues such as cartilage, hard biological tissues such as teeth and bones, and body lumens and passages such as sinuses, ureters, colon, esophagus, lung passages, blood vessels and pharynx, and various others.

[0023]

[0039] The systems and devices herein can be combined in one or more ways among many ways to provide improved diagnosis and treatment to patients. The embodiments provided herein can be combined with existing methods and devices to provide improved treatment, such as in combination with known methods of lung diagnosis, surgery, and surgery of other tissues and organs. Any one or more of the structures and processes described herein can be combined with any one or more additional structures and processes of any one or more of the methods and devices described herein, and it should be understood that the drawings and the supporting text provide an explanation according to the embodiments.

[0024]

[0040] When the terms "at least", "greater than", or "above" are in front of the first numerical value of a series of two or more numerical values, the terms "at least", "greater than", or "above" always apply to each numerical value of that series of numerical values. For example, 1, 2, or above 3 is equivalent to 1 or above, 2 or above, or 3 or above.

[0025]

[0041] When the terms "not exceeding", "less than", or "below" are in front of the first numerical value of a series of two or more numerical values, the terms "not exceeding", "less than", or "below" always apply to each numerical value of that series of numerical values. For example, 3, 2, or below 1 is equivalent to 3 or below, 2 or below, or 1 or below.

[0026]

[0042] As used herein, the terms distal and proximal may generally refer to locations relative to a device and may be opposite to anatomical references. For example, the distal location of a main shaft or catheter may correspond to the proximal location of a patient's elongate member, and the proximal location of a main sheath or catheter may correspond to the distal location of a patient's elongate member.

[0027]

[0043] As described above, the setup of a robotic endoscope system can be time-consuming and difficult due to the complexity of the operating environment, the need for accurate alignment between the instrument and the patient's body part, and various other reasons. The present disclosure simplifies the workflow during system setup and / or treatment by providing methods and systems that can detect and track the operating environment of the system and automatically adjust the system. In some embodiments, the autonomous processes and / or operations of the system may include alignment of the instrument drive mechanism (IDM) of the robotic endoscope device to the patient-side mount, alignment of the robot base (robot cart) to the patient or hospital suite equipment, automatic configuration for recognition and compatibility with other technologies / devices, collision avoidance between the system and the patient and / or other objects in the operating environment, automatic adjustment of the robotic arm or IDM based on real-time buckling detection of the instrument and monitoring of the patient's respiration / vitals, and other functions described elsewhere in this specification.

[0028]

[0044] The operating environment of the robotic endoscope system may include one or more objects. The robotic endoscope system can detect one or more objects in the operating environment, generate a 3D map including depth information, perform autonomous alignment of the instrument drive mechanism with respect to the patient bed or body part, and automatically adjust its placement and orientation to avoid collisions with one or more objects. The one or more objects may include, for example, system accessories (such as a system monitor, monitor pole), external monitors, patient beds, patients, imaging devices (such as a fluoroscopy c-arm), anesthesia carts, and other equipment or subjects (such as an operator, surgeon) in the operating environment before or during surgery.

[0029]

[0045] FIG. 1 schematically shows a robotic platform 100. This platform may include a robotic endoscope system with one or more flexible articulatable surgical instruments 105, and a support device 110 such as a robotic manipulator (such as a robotic arm) for driving, supporting, positioning, or controlling the movement and / or operation of the robotic system. The robotic platform may further include peripheral devices and subsystems such as an imaging system that can assist and / or facilitate the guidance of an elongate member to a target site within the body of a subject 120.

[0030]

[0046] A robotic endoscope system is provided for performing low-cost and improved-performance surgical procedures or diagnoses. For example, the robotic endoscope system may include a fully disposable steerable catheter. As shown in FIG. 1, the robotic endoscope system may include a steerable catheter assembly 105 and a robotic support system 110 for supporting or carrying the steerable catheter assembly. The steerable catheter assembly may be an endoscope. In some embodiments, the steerable catheter assembly may be a single-use robotic endoscope. In some embodiments, the robotic endoscope system may include an instrument drive mechanism (IDM) 103 attached to the distal end of the robotic arm 107 of the robotic support system. The instrument drive mechanism may be provided by any suitable controller device (e.g., a handheld controller) that may or may not include the robotic system. The instrument drive mechanism may provide a mechanical and electrical interface with the steerable catheter assembly 105. The mechanical interface may enable the steerable catheter assembly 105 to be releasably coupled to the instrument drive mechanism 103. For example, the handle portion 104 of the steerable catheter assembly may be attached to the instrument drive mechanism (IDM) via a quick attachment / detachment means such as a magnet, a spring-loaded lever, etc. In some cases, the steerable catheter assembly may be manually coupled or detached from the instrument drive mechanism without using tools. The instrument drive mechanism may be used to control an elongate member or a robotic catheter assembly with two or more degrees of freedom (e.g., joints).

[0031]

[0047] The robot support system 110 may include a robotic arm 107 and a mobile base (e.g., a robot cart) 109. The robotic arm 107 may initiate the positioning of a robotic catheter assembly or other robotic equipment. In some cases, a user interface, a robot control module, and the robotic arm may be attached to the moving cart. The moving cart includes one or more computers that store application-specific software for the user interface, and provides charging ports for portable electronic devices, converters, transformers, and surge protection devices, with respect to multiple AC and DC receptacles that serve as a power source for on-board equipment, and may include various elements such as a rechargeable power source that electrically communicates with an electrical panel.

[0032]

[0048] The robotic arm 107 may have redundant degrees of freedom that move its elbow algorithmically or passively to enable a convenient posture for the operator to initiate positioning of the robotic system or other robotic instruments. For example, the robotic arm may be provided with a plurality of joints having redundant degrees of freedom so that the joints of the robotic arm can be driven in a range of various postures with respect to a predetermined end effector position (e.g., the IDM position). The redundant degrees of freedom may advantageously enable the robotic arm to automatically adjust to an optimal posture to avoid collisions with other objects in the operating environment before or during a procedure. For example, the instrument drive mechanism may automatically align with a patient-side mount (e.g., a support structure of a patient bed for holding and supporting an endoscopic device in a predetermined position) during a setup procedure. During the setup procedure and the operation procedure, when the movement of the patient-side mount is detected, the instrument drive mechanism (IDM) can automatically adjust accordingly to avoid collisions while maintaining the position of the IDM, eliminating interruptions in the procedure workflow and avoiding misalignments. In another example, when the robotic arm is activated during a setup procedure or a surgical operation, the system may detect an undesirable approach of any part of the robotic arm and other surrounding objects (e.g., a monitor of the system), and the robotic arm may be automatically reconfigured and moved away from the monitor to avoid collisions.

[0033]

[0049] In some embodiments, in addition to the autonomous movement of the robotic arm, such as automatically positioning the steerable catheter assembly 105 to an initial position (e.g., an access point) to access a target tissue, the robotic arm may be moved passively by the operator. In such a case, the operator can push the arm at any position, and the arm moves compliant. The robotic arm can also be controlled in a compliance mode to improve the human-robot interaction. For example, the compliance motion control of robotic technology may adopt a collision avoidance strategy, and the position force control may be designed to save unnecessary energy consumption while reducing the impact of the possibility of collision.

[0034]

[0050] The manipulable catheter assembly 105 may comprise a flexible elongate member coupled to a handle portion. The robotic endoscope system may comprise a buckling prevention device 101 for preventing buckling of the elongate member during use.

[0035]

[0051] FIG. 2 shows another example of a robotic catheter assembly with a buckling prevention device 201. The manipulable catheter assembly may comprise a handle portion 211 that may include components configured to process image data, supply power, or establish communication with other external devices. For example, the handle portion 211 may include circuitry and communication elements that enable electrical communication between the manipulable catheter assembly and an instrument drive mechanism 221, and any other optional external system or device. In another example, the handle portion 211 may include circuit elements such as a power source for supplying power to the electronics of the endoscope (e.g., a camera and LED lights). In some cases, the handle portion may be in electrical communication with the instrument drive mechanism 221 via an electrical interface (e.g., a printed circuit board) such that image / video data and / or sensor data can be received by the communication module of the instrument drive mechanism and transmitted to other external devices / systems. Alternatively or in addition, the instrument drive mechanism 221 may provide only a mechanical interface. The handle portion may be in electrical communication with a modular wireless communication device or other user device (e.g., a portable / handheld device or a controller) for transmitting sensor data and / or receiving control signals.

[0036]

[0052] The steerable catheter assembly may comprise a flexible elongated member 213 (i.e., the catheter) coupled to the handle portion 211. In some embodiments, the flexible elongated member may include a shaft, a steerable tip, and a steerable section. The steerable catheter assembly may be a single-use robotic endoscope. In some cases, only the elongated member may be disposable. In some cases, at least a portion of the elongated member (e.g., the shaft, the steerable tip, etc.) may be disposable. In some cases, the entire steerable catheter assembly including the handle portion and the elongated member may be disposable. The flexible elongated member and the handle portion are designed to enable the disposal of the entire steerable catheter assembly at low cost.

[0037]

[0053] The robotic endoscope may be releasably coupled to the instrument drive mechanism 221. The instrument drive mechanism 221 may be attached to the arm of the robotic support system or to any actuation support system as described above. The instrument drive mechanism may provide a mechanical and electrical interface with the robotic endoscope. The mechanical interface may enable the robotic endoscope to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the robotic endoscope may be attached to the instrument drive mechanism via a quick attachment / detachment means such as a magnet or a spring-loaded lever. In some cases, the robotic endoscope may be manually coupled or detached from the instrument drive mechanism without using tools. In some embodiments, the instrument drive mechanism 221 may comprise a set of motors actuated to rotationally drive a set of pull wires of the catheter. The handle portion 211 of the catheter assembly may be attached to the instrument drive mechanism such that its pulley assembly is driven by the set of motors. The number of pulleys may vary depending on the configuration of the pull wires. In some cases, one, two, three, four, or more pull wires may be used to articulate the catheter.

[0038]

[0054] The handle part may be designed so that the robotic endoscope can be disposed of at low cost. For example, typical manual and robotic endoscopes may have a cable at the proximal end of the endoscope handle. The cable often includes illumination fibers, a camera video cable, and other sensor fibers or cables such as EM sensors and shape detection fibers. Such complex cables are expensive and may increase the cost of the endoscope. The provided robotic endoscope may have an optimized design that allows for the adoption of a simplified structure and components while maintaining mechanical and electrical functions. In some cases, the handle part of the robotic endoscope may adopt a cable-free design while providing a mechanical / electrical interface with the catheter.

[0039]

[0055] In some cases, the handle part may house or include components configured to process image data, supply power, or establish communication with other external devices. In some cases, the communication may be wireless communication. For example, wireless communication may include Wi-Fi, wireless communication, Bluetooth, IR communication, or other types of direct communication. Such wireless communication capabilities may enable the robotic bronchoscope to function in a plug-and-play manner and, conveniently, can be discarded after a single use. In some cases, the handle part may include circuit elements such as a power source for supplying power to electronic devices (e.g., a camera and an LED light source) disposed within the robotic bronchoscope or catheter.

[0040]

[0056] The handle part may be designed to be interlocked with the catheter so as to eliminate cables and fibers. For example, the catheter part may adopt a design having a single working channel that allows the instrument to pass through a robotic bronchoscope, low-cost electronic devices such as a chip-on-chip camera, a light source such as a light-emitting diode (LED), and an EM sensor located at an optimal location according to the mechanical structure of the catheter. This may enable a simplified design of the handle part. For example, by using an LED for illumination, the termination in the handle part can be based only on electrical soldering or wire crimping. For example, the handle part may include a proximal plate where a camera cable, an LED cable, and an EM sensor cable terminate, while the proximal plate connects to the interface of the handle part and establishes an electrical connection with the instrument drive mechanism. As described above, the instrument drive mechanism is attached to a robotic arm (robotic support system) and provides a mechanical and electrical interface with the handle part. This may advantageously improve assembly and implementation efficiency and simplify the manufacturing process and cost. In some cases, the handle part interlocked with the catheter may be disposed of after a single use.

[0041]

[0057] In some embodiments, the steerable catheter assembly may have a substantially integrated design in which one or more components can be integrated with the catheter, thereby simplifying the assembly and manufacturing process while maintaining the kinematic dynamic performance of the steerable catheter. As shown in the example, the steerable catheter assembly may include an elongated member 213 or a probing part that is brought close to the tissue and / or area to be examined. The elongated member 213 may sometimes also be called a catheter. The catheter 213 may include an internal structure such as a working channel that allows a tool to be inserted. As an example, the working channel may have a dimension such as a diameter of approximately 2 mm that is compatible with standard tools. The working channel may have any other suitable dimension based on the application example.

[0042]

[0058] The catheter 213 may be composed of a material suitable for a desired flexibility or bending rigidity. In some cases, the material of the catheter may be selected such that it can maintain structural support for an internal structure (e.g., a working channel) and is substantially flexible (e.g., can be bent in various directions and orientations). For example, the catheter can be made of any suitable material such as urethane, vinyl (such as polyvinyl chloride), nylon (vestamid, grillamid, etc.), pellethane, polyethylene, polypropylene, polycarbonate, polyester, silicon, silicone elastomer, acetate, etc. In some cases, the material may be a polymeric material, a biocompatible polymeric material, and the catheter may be flexible enough to pass through a path with a small curvature without causing pain to the subject. In some cases, the catheter may comprise a sheath. The sheath may not be the same length as the catheter. The sheath may be shorter than the catheter to provide the desired support. Alternatively, the catheter may be a substantially one-piece component.

[0043]

[0059] In some cases, the distal portion or tip of the catheter may be substantially flexible so as to be steerable in one or more directions (e.g., pitch, yaw). In some embodiments, the catheter may have variable bending stiffness along its longitudinal axis. For example, the catheter may comprise a plurality of segments having different bending stiffnesses (e.g., flexible, semi-rigid, and rigid). The bending stiffness may vary by selecting materials having different stiffnesses / hardnesses, varying the structure in different segments, adding additional support components, or any combination of the above. In some cases, since the proximal end of the catheter need not be highly bent, the proximal end of the catheter may be reinforced by additional mechanical structures (e.g., additional material layers) to obtain greater bending stiffness. Such designs may provide support and stability to the catheter. In some cases, variable bending stiffness may be obtained by using different materials during extrusion of the catheter. This advantageously may allow different stiffness levels along the shaft of the catheter in an extrusion manufacturing process without additional fastening or assembly of different materials.

[0044]

[0060] The distal portion of the catheter may be manipulated by one or more pull wires. The distal portion of the catheter may be made of any suitable material such as a copolymer, polymer, metal or alloy so that it can be bent by the pull wire. In some embodiments, the proximal end or proximal portion of one or more pull wires may be operably coupled to various mechanisms (such as gears, pulleys, etc.) in the handle portion of the catheter assembly. The pull wire may be a metal wire, cable or thread, or may be a polymer wire, cable, or thread. The pull wire may also be made of natural or organic materials or fibers. The pull wire may be any type of suitable wire, cable or thread capable of supporting various types of loads without deformation, significant deformation, or breakage. Since the distal end or distal portion of one or more pull wires may be fixed or integrated with the distal portion of the catheter, the operation by the control unit of the pull wire may apply a force or tension to the distal portion that can manipulate (e.g., in any direction such as up, down, pitch, yaw, or intermediate) or articulate at least the distal portion (e.g., the flexible section) of the catheter.

[0045]

[0061] As described above, the pull wire may be made of any suitable material such as stainless steel (e.g., SS316), metal, alloy, polymer, nylon or biocompatible material. The pull wire may be a wire, cable or thread. In some embodiments, different pull wires may be made of different materials to vary the load-bearing capacity of the pull wire. In some embodiments, different sections of the pull wire may be made of different materials to vary the stiffness and / or load-bearing along the pull. In some embodiments, the pull wire may be utilized for the transfer of electrical signals.

[0046]

[0062] The catheter may have dimensions such that one or more electronic components can be integrated into the catheter. For example, the outer diameter of the distal tip may be approximately 4 to 4.4 millimeters (mm), and the diameter of the working channel may be approximately 2 mm such that one or more electronic components can be embedded in the wall of the catheter or in the catheter lumen. However, it should be noted that based on different applications, the outer diameter can be in any range less than 4 mm or greater than 4.4 mm, and the diameter of the working channel can be in any range according to the dimensions of the tool or a particular application.

[0047]

[0063] The one or more electronic components may include an imaging device, a lighting device, or a sensor. In some embodiments, the imaging device may be a video camera. The imaging device may comprise an optical element and an image sensor for capturing image data. The image sensor may be configured to generate image data in response to the wavelength of light. Various image sensors for capturing image data, such as complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD), may be employed. The imaging device may be a low-cost camera. In some cases, the image sensor may be provided on a circuit board. The circuit board may be an imaging printed circuit board (PCB). The PCB may comprise a plurality of electronic elements for processing the image signal. For example, the circuit for a CCD sensor may comprise an A / D converter and an amplifier for amplifying and converting the analog signal provided by the CCD sensor. Optionally, the image sensor may be integrated with an amplifier and a converter for converting the analog signal to a digital signal such that a circuit board may not be required. In some cases, the output of the image sensor or the circuit board may be image data (digital signal) and may be further processable by a camera circuit or a processor of the camera. In some cases, the image sensor may comprise an array of optical sensors.

[0048]

[0064] The lighting device may comprise one or more light sources disposed at the distal tip. The light source may be a light emitting diode (LED), an organic LED (OLED), a quantum dot, or any other suitable light source. In some cases, the light source may be a small LED for compact design or a dual-tone flash LED lighting.

[0049]

[0065] In some embodiments, the catheter may be designed to be flexible. When the flexible portion of the catheter is inserted into the patient through the endoscope by an extension mechanism, one or more sections may bend or buckle.

[0050]

[0066] To support the catheter, the anti-buckling mechanism 201 may be coupled to the handle portion of the robotic endoscope. The anti-buckling mechanism is used to prevent buckling of the insertion shaft. The anti-buckling mechanism 201 may be a telescopic extension device having an internal mechanism for achieving anti-buckling of the catheter during insertion and withdrawal. The anti-buckling mechanism may be removably connected to the handle portion of the robotic bronchoscope at one end and may be removably connected to the support surface 203 at the other end. As shown in the example, the anti-buckling tube may be attached to a bracket on the instrument drive mechanism and may be removable and disposable after the procedure via a quick detachment mechanism. In the example shown in FIG. 2, a support arm (e.g., an ET tube mount support arm) supports the endotracheal tube mount and may be supported by a robotic mobile cart that provides a support surface for the distal end of the anti-buckling tube to abut when the anti-buckling tube is compressed. The support arm may be rotated and controlled to translate vertically up and down, and / or may be a boom arm that expands and contracts to be accurately positioned above the patient's mouth and attachable to the endotracheal tube mount. The positioning of the support arm may be synchronized with the movement of the robotic arm so as to track the location of the catheter entry point.

[0051]

[0067] The buckling prevention mechanism may require a relatively straight trajectory for movement. In some cases, such a trajectory may be ensured by aligning the buckling prevention mechanism in a folded state with the patient-side connector. FIG. 1 shows an example of the patient-side connector 121 and the IDM 103. For example, the patient-side connector may be fixed to the patient-side mount 123 (e.g., attached to the patient bed). Aligning the IDM with the patient-side connector / mount may sometimes include arranging the folded buckling prevention mechanism in line with the patient-side connector. The robotic arm may automatically move the IDM to a position where the IDM is aligned with the patient-side connector. In some cases, the alignment may include generating a 3D depth map of the operating environment, moving the transfer cart to a desired position relative to the patient bed based on the depth map, and transitioning the IDM to a position and orientation aligned with the patient-side connector based on the detection of the location / position of the patient-side connector.

[0052]

[0068] Figure 3 shows an example of a robotic endoscope (e.g., bronchoscopy) system (300, 330) that can perform autonomous collision avoidance and self - alignment before and during surgery. The robotic endoscope (e.g., bronchoscopy) system 300 may include a steerable catheter assembly 320 and a robotic support system 310 for supporting or transporting the steerable catheter assembly. In some cases, the steerable catheter assembly may be a bronchoscope. The steerable catheter assembly may be the same as the endoscope devices described elsewhere in this specification. In some cases, the steerable catheter assembly may be a single - use robotic bronchoscope. In some embodiments, the robotic endoscope (e.g., bronchoscopy) system 300 may include an instrument drive mechanism (IDM) 313 attached to the end of the robotic arm 311 of the robotic support system. The instrument drive mechanism may be provided by any suitable controller device (e.g., a handheld controller) that may or may not include the robotic system. The instrument drive mechanism may provide a mechanical and electrical interface with the steerable catheter assembly 320. The mechanical interface may enable the steerable catheter assembly 320 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the steerable catheter assembly may be attached to the instrument drive mechanism via quick - attach / detach means such as magnets, spring - loaded levers, etc. In some cases, the steerable catheter assembly may be manually coupled or detached from the instrument drive mechanism without using tools.

[0053]

[0069] The steerable catheter assembly 320 may comprise a handle portion 323 that may include components configured to process image data, supply power, or establish communication with other external devices. For example, the handle portion 323 may include circuitry and communication elements that enable electrical communication between the steerable catheter assembly 320 and the instrument drive mechanism 313, as well as any other optional external system or device. In another example, the handle portion 323 may include circuit elements such as a power source for supplying power to the electronics of an endoscope (e.g., a camera and LED lights). In some cases, the handle portion may be in electrical communication with the instrument drive mechanism 313 via an electrical interface (e.g., a printed circuit board) such that image / video data and / or sensor data can be received by the communication module of the instrument drive mechanism and transmitted to other external devices / systems. Alternatively or in addition, the instrument drive mechanism 313 may provide only a mechanical interface. The handle portion may be in electrical communication with a modular wireless communication device or other user device (e.g., a portable / handheld device or a controller) to transmit sensor data and / or receive control signals.

[0054]

[0070] The steerable catheter assembly 320 may comprise a flexible elongate member 321 coupled to the handle portion. In some embodiments, the flexible elongate member may include a shaft, a steerable tip, and a steerable section. The steerable catheter assembly may be a single-use robotic bronchoscope. In some cases, only the elongate member may be disposable. In some cases, at least a portion of the elongate member (e.g., the shaft, the steerable tip, etc.) may be disposable. In some cases, the entire steerable catheter assembly 320, including the handle portion and the elongate member, may be disposable. The flexible elongate member and the handle portion are designed such that the entire steerable catheter assembly can be disposed of at low cost.

[0055]

[0071] In some embodiments, the provided bronchoscope system may also include accessories such as a user interface. As shown in the exemplary system 330, during the surgery, one or more components of the system, such as the treatment interface module 331 (on the user console side) and / or the treatment control module 333 (on the patient and robot side), may be brought into the operating environment. One or more components or accessories may be added to or removed from the operating environment before or during the surgery. The robotic arm 311 may have redundant degrees of freedom such that the joints of the robotic arm can be driven to various postures with respect to a given end effector position. For example, when the treatment interface module 311 is detected, the robotic arm 311 may automatically shift to a different posture to avoid collision with the treatment interface module while the distal end of the arm of the IDM 313 maintains a specific state (e.g., a given position or velocity of the end effector). Details of collision avoidance will be described later in this specification.

[0056]

[0072] The treatment interface module may enable an operator or user to interact with the bronchoscope during a surgical procedure. In some embodiments, the treatment control module 333 may be a handheld controller. The treatment control module 333 may enable a user to control the speed of the tip of the bronchoscope, as described elsewhere herein. The treatment control module may optionally comprise a unique user input device and one or more add-on elements removably coupled to an existing user device to enhance the user input experience. For example, a physical trackball or roller may replace or complement the functionality of at least one virtual graphical element (e.g., navigation arrows displayed on a touchpad) displayed on a graphical user interface (GUI) by providing a similar functionality to the replaced graphical element. Examples of user devices may include, but are not limited to, mobile devices, smartphones / cell phones, tablets, personal digital assistants (PDAs), laptop or notebook computers, desktop computers, media content players, etc. Details of the user interface device and user console are described later herein.

[0057]

[0073] The robotic endoscope platform herein may be able to detect one or more objects in an operating environment surrounding a robotic endoscope system. In this case, the detection of the operating environment may include generating an obstacle map. The obstacle map may be a three-dimensional (3D) map representing the positions of the objects detected in a three-dimensional space.

[0058]

[0074] The 3D map of the operating environment may be constructed based on sensing data. In some cases, the sensing data including depth information of the environment is received from one or more vision sensors. The vision sensors may include a camera, a video camera, a three-dimensional (3D) depth camera, a stereo camera, a depth camera, a red-green-blue depth (RGB-D) camera, a time-of-flight (TOF) camera, an infrared camera, a charge-coupled device (CCD) image sensor, or a complementary metal-oxide-semiconductor (CMOS) image sensor. For example, the vision sensor may include only one camera (monocular vision sensor). Alternatively, the vision sensor may include two (binocular vision sensor) or more cameras. The vision sensor may be disposed on a robot endoscope system such as a robot cart, a monitor, etc. Alternatively or additionally, the vision sensor may not be disposed on the robot endoscope system. For example, the vision sensor may be disposed on a wall, a ceiling or other places of the operating environment (e.g., a room). In an embodiment where multiple vision sensors are used, each sensor may be located at a different part of the robot endoscope system, and the differences between the image data collected by each sensor can be used to provide depth information of the environment. The depth information can be used herein to refer to information regarding the distance from the robot endoscope system and / or the sensor to one or more objects. In an embodiment where a single vision sensor is used, depth information can be obtained by capturing image data at multiple different positions and orientations of the vision sensor and reconstructing the depth information using appropriate image analysis techniques (e.g., structure from motion).

[0059]

[0075] FIG. 4A shows an example of constructing a 3D map of an operating environment 400 using an optical sensor 401 such as a camera. As described elsewhere in this specification, the operating environment of the system may generally include an environment external to the system (e.g., the room where the system is located, within a certain proximity, etc.) or an environment in which the system can move or operate. In some cases, the camera may be a plenoptic camera having a main lens and an additional microlens array (MLA). The plenoptic camera model may be used to calculate a depth map of the captured image data. In some cases, the image data captured by the camera may be a grayscale image (i.e., a depth map) that includes depth information at each pixel coordinate. The camera may be calibrated to improve depth measurement accuracy by obtaining internal camera parameters such as focal length, focus distance, distance between the MLA and the image sensor, pixel size, etc. Other parameters such as distortion coefficients may also be calibrated to correct the image for metric depth measurement.

[0060]

[0076] In some cases, the image data may be received and processed by one or more processors of the robotic endoscope system. For example, preprocessing of the captured image data may be performed. In one embodiment, the preprocessing algorithm may include image processing algorithms such as image smoothing to reduce the influence of sensor noise, or image histogram equalization to enhance pixel intensity values. Next, the optical approach described elsewhere in this specification may be employed to generate a depth map of the operating environment 400. In some cases, computer vision (CV) techniques or computer vision systems are used to process the sensing data in order to elicit a high-level understanding of the operating environment, object detection, object classification, extraction of scene depth and estimation of the relative position of objects, and extraction of the orientation of objects in space. For example, the CV output data may be generated using a passive method that requires only an image. Passive methods may include, for example, object recognition, stereovision, monocular shape from motion, shape from shading, and simultaneous localization and mapping (SLAM). Alternatively, an active method that may require projecting controlled light onto the target scene may be utilized, and active methods may include, for example, structured light and time-of-flight (ToF). In some cases, computer vision techniques such as optical flow, computational stereo methods, iterative methods combined with prediction models, machine learning methods, prediction filtering, or any non-rigid registration method may be used to generate a description of the 3D scene.

[0061]

[0077] In some cases, a reference marker 403 may be used to align the IDM to the patient-side mount. The reference marker may be disposed on the patient bed or the patient-side mount 405. The reference marker may have a 2D shape or pattern. For example, the reference marker may be a 2D QR code, grid, or any asymmetric shape. By acquiring an image of the 2D reference marker (e.g., from various angles), the position and orientation of the patient-side mount on which the reference marker is disposed within the camera frame can be determined (e.g., by triangulation). Based on the known spatial relationship between the camera and the IDM, the orientation and position of the IDM with respect to the patient-side mount can be calculated. Alternatively, the reference marker may be a 3D reference marker such that the marker is visible and distinguishable at a wide range of angles. For example, the 3D reference marker may be positioned within the field of view of the imaging system such that the reference marker is always distinguishable regardless of the position of the optical sensor with respect to the patient bed or the marker. The reference marker may have any suitable 2D / 3D shape (anisotropic) or pattern such that the projection of the reference mark corresponding to the field of view / angle is distinguishable from the projection of another field of view / angle. Alternatively, alignment of the IDM to the patient-side mount may not require a reference marker. For example, the patient-side mount may be recognized without a reference marker using segmentation and / or the object recognition methods described elsewhere in this specification. In some cases, the 3D reference marker may be used to align the IDM to the patient-side mount independently of using the 3D point cloud. For example, a series of image frames or videos including the reference marker may be acquired and processed to identify the spatial relationship (e.g., orientation, position) between the imaging device and the patient-side mount (i.e., the reference marker). Based on the known geometric relationship between the IDM and the imaging device, the spatial relationship between the IDM and the patient-side mount may be derived. Alternatively, the reference marker may be used together with a 3D depth map. For example, the object identity (e.g., the patient-side mount) may be identified by the reference marker, and depth data may be assigned to the object based on the 3D depth map. In some cases, the 3D depth map may include a 3D point cloud.Alternatively, the 3D depth map may be generated based on optical image data. The 3D depth map may at least include an object (such as a reference marker) containing depth information obtained using the above optical method.

[0062]

[0078] In some cases, the imaging device may be used together with other types of sensors (such as proximity sensors, location sensors, position sensors, etc.) to improve the accuracy of the position information. For example, the sensing data may further include sensor data from one or more proximity sensors. The proximity sensor may be of any suitable type such as an ultrasonic sensor (e.g., a wide-angle sensor, an array sensor) or a light detection and ranging (Lidar) sensor. Lidar may be used to obtain three-dimensional information of the environment by measuring the distance to an object. The proximity sensor may also be disposed in a robot endoscope system. The proximity sensor may be located near the vision sensor. Alternatively, the proximity sensor may be located in a part of the robot endoscope system different from the part used to mount the vision sensor.

[0063]

[0079] In some cases, the 3D depth map may be generated using single-modal sensor data (such as image data, Lidar, proximity data, etc.). Alternatively, the 3D depth map may be generated using multi-modal data. For example, the image data and 3D point cloud generated by a Lidar system may be fused using a Kalman filter or a deep learning model to generate a 3D map. Then, the 3D map may be used for automatic alignment of IDM, self-positioning of the robotic vehicle, collision avoidance, and various other functions described elsewhere in this specification.

[0064]

[0080] In some embodiments, the autonomous alignment of the IDM to the patient mount may be based on a positioning sensor. For example, the sensor signals may be acquired by electromagnetic coils located on the IDM and electromagnetic coils located on the patient mount, as well as an electromagnetic tracking system. The positions and orientations of the IDM and the patient mount may be detected, and the difference may be used to generate a command to move the robotic arm, thereby achieving the autonomous alignment of the IDM and the patient mount.

[0065]

[0081] As shown in FIG. 4B, the system may include an EM field generator 415 that propagates an EM field in the environment. The EM field generator may be located adjacent to the patient's torso during the procedure, for example, on the bed 407, the robotic cart, or other suitable location within the environment 410. The system may include a first EM sensor 411 located on the IDM to measure the position and orientation of the IDM, and a second EM sensor 413 located on the patient mount to measure the position and orientation of the patient mount. FIG. 4C shows an example of a patient mount 431 and an associated EM sensor 430. Returning to FIG. 4B, the EM field generator and the two sets of EM sensors 411, 413 may be utilized by the system to identify the positions and orientations of the IDM and the patient mount in 3D space.

[0066]

[0082] The positions and orientations measured by the two sets of EM sensors may be represented in the submitted generator frame. The EM sensors may be 6DOF sensors (e.g., X, Y, Z, roll, pitch, yaw) that can sense the EM signals generated by the EM field generator and measure the 6-degree-of-freedom spatial data of the IDM and the patient mount. Alternatively, a pair of 5DOF EM sensors may be located on the IDM and / or the patient mount to measure the position and orientation. For example, one 5DOF sensor may generate signals for X, Y, Z, pitch, and yaw other than roll. By arranging two 5DOF sensors within the device fixed relative to each other such that their central axes are not parallel to each other, roll information can be calculated using the pair of 5DOF signals.

[0067]

[0083] Next, spatial data regarding the IDM and the patient-side mount is processed by the system to determine whether the cart is properly positioned for automatic alignment. For example, closed-loop feedback (e.g., controlling the movement of the robotic arm based on EM sensor data to correct the position / orientation of the IDM) and / or other control methods described elsewhere in this specification may be used to move the IDM into alignment with the patient-side mount.

[0068]

[0084] FIG. 5 shows an example of autonomous alignment of the IDM (instrument drive mechanism) with respect to the patient-side mount. In the illustrated example 510, the instrument drive mechanism may initially be in a misaligned position with respect to the patient-side mount. During the setup procedure, actuators of multiple links / joints of the robotic arm may be activated to automatically align the IDM with the patient-side mount 520. In some cases, after alignment, the IDM may be moved to an appropriate position at a predetermined distance 610 from the patient-side mount 600 for loading instruments such as a catheter assembly or an endoscopic device, as shown in FIG. 6. For example, a flexible endoscopic device may be coupled to the IDM at the proximal end of the endoscope and to the patient-side mount via a connector at the distal end. In some cases, the predetermined distance 610 may be generated based on the dimensions of the endoscopic device or empirical data. The user may be permitted to further manually adjust the position of the IDM as needed by switching the robotic arm to a passive mode. For example, enabling the passive model allows the user to directly apply force to position the robotic arm in any posture, position, or orientation, and the desired position and posture will be maintained.

[0069]

[0085] As described above, a robotic arm may have redundant degrees of freedom. For example, a robotic arm may have 6, 7, 8, 9, or more degrees of freedom (DOF) to enable the IDM to be oriented in a 5 or 6 degree-of-freedom (DOF) space. For example, a robotic arm end effector (e.g., IDM) that can be positioned with 6 degrees of freedom may, in some cases, have 9 degrees of freedom (3 degrees of freedom for position, 3 degrees of freedom for orientation, and 3 degrees of freedom to comply with access site constraints), or 10 or more degrees of freedom. A highly configurable robotic arm having more degrees of freedom than required for a particular end effector position can advantageously provide sufficient degrees of freedom to enable various joint states of the end effector position within the workspace. When the patient-side mount moves during the procedure, the instrument drive mechanism may automatically adjust accordingly to maintain alignment with the patient-side mount, eliminating interruptions in the procedure workflow and avoiding misalignment.

[0070]

[0086] In some embodiments, the 3D depth map generated by the platform may be used for the automatic placement of the mobile / robotic cart. For example, the 3D depth map may include descriptions of the operating environment such as the identification of equipment, patient beds, human operators, patients, etc., and such a 3D depth map can be used to generate the optimal location of the mobile cart relative to the patient bed. As described above, computer vision (CV) technology or a computer vision system may be used to elicit a high-level understanding of the operating environment, object detection, object classification, extraction of scene depth and estimation of the relative position of objects, and extraction of the orientation of objects in space. 3D map information and sensor data (e.g., proximity sensors, imaging sensors) can be used to detect whether the robotic cart is within an optimal zone relative to the patient bed.

[0071]

[0087] FIG. 7 shows an example of a self-propelled robotic cart (e.g., via one or more propulsion units such as wheels, rotors, propellers, etc.) that autonomously positions itself relative to a patient bed. In some embodiments, the propulsion unit may comprise a plurality of wheels that may enable the robotic cart to roll on the surface of a support structure. In some examples, two, three, or four wheels may be provided, and the robotic cart may be able to stand stably when not in motion. In some cases, stabilization may be effected with the aid of one or more wheels or other stabilization platforms such as a gyroscope platform. The wheels may be of different or the same size. In some cases, the diameter of the wheels may be at least about 1 cm, 2 cm, 3 cm, 4 cm, 5 cm, 8 cm, 9 cm, 10 cm, 15 cm, 20 cm, 25 cm, 30 cm, 35 cm, 40 cm, 45 cm, 50 cm, 55 cm, 60 cm, 65 cm, 70 cm, 75 cm, 80 cm, 85 cm, 90 cm, 95 cm, 100 cm, 150 cm, or 200 cm. The wheels may have a smooth surface or a treaded surface. The wheels may also be configured to enable the robotic cart to move laterally and / or rotate in place. The robotic cart may be configured to enable any combination of translational and rotational movement. The propulsion unit may be driven with the assistance of one or more actuators. For example, a motor, engine, drive train, or other component that may assist in driving the propulsion force of the robotic cart may be provided.

[0072]

[0088] Based on the 3D depth map, the optimal location of the robotic cart may be generated. The optimal location may be generated based on the dimensions of the robotic cart, the dimensions of the robotic arm (working space), the dimensions of the endoscopic device, and the 3D depth map. Real-time sensor data (e.g., proximity sensors) may be collected and used to determine whether the robotic cart is in an appropriate location relative to the patient bed. As shown in FIG. 8, when it is detected (810) that the self-propelled robotic cart is not in an appropriate location relative to the patient bed, the robotic cart may automatically move to the appropriate location 820. The appropriate location, movement speed, movement acceleration, and movement trajectory may be calculated by one or more processors of the platform based at least in part on the 3D depth map.

[0073]

[0089] Alternatively or additionally, the system may notify the user that the placement of the robotic cart is not optimal and prompt the user to intervene. For example, a message, warning, or notification may be displayed on the screen along with a proposal regarding the placement of the robotic cart (e.g., explicitly indicating to move the cart closer to the bed) and / or a display of the 2D / 3D depth map of the operating environment and the robotic system. For example, the robotic system may generate a preferred relative cart position and orientation based on the current cart position and display an animation to guide the user in changing the position of the robotic cart.

[0074]

[0090] Figure 9 shows an exemplary process 900 for autonomous alignment of a robotic endoscope system. In the illustrated process, Lidar data (e.g., 3D point cloud) is used for real-time detection of objects in the operating environment and acquisition of depth information, but note that any other suitable sensors (e.g., stereo cameras) and methods described elsewhere in this specification can also be utilized. In the exemplary process, runtime sensor data (e.g., 3D point cloud) may be captured as input 901. The runtime sensor data may include data captured by Lidar (light detection and ranging). Lidar may acquire three-dimensional information of the operating environment / scene by measuring the distance to objects. For example, the oscillator of a Lidar system may generate a series of optical pulses that are oscillated within a short time period so as to derive distance measurement points using the series of optical pulses. The Lidar system may provide three-dimensional (3D) imaging (e.g., 3D point cloud). In some cases, the 3D point cloud or 3D image may be further processed by one or more processors of the robotic endoscope system for obstacle detection or collision avoidance (903). Various suitable image processing methods (e.g., image segmentation) may be utilized to recognize objects such as a patient bed. Depth information from the 3D point cloud may be used to assign distances to points within the segmented region (e.g., the region of the patient bed).

[0075]

[0091] The position and orientation of the Lidar sensor may be obtained based on kinematic mapping, and such information, together with the depth information of the bed, is used to estimate the relative orientation and position between the bed and the robotic endoscope system. Based on the relative position and orientation, a movement path may be generated to move the robotic cart to an optimal placement with respect to the patient bed. The optimal placement may be automatically generated by the system without user intervention. In some cases, instead of or in addition to performing the self-placement of the robotic cart, the algorithm may notify the user of invalid locations of the robotic cart with respect to the patient bed, and the appropriate locations of the robotic cart may be displayed to the user on the GUI. The user may be able to provide an input via the GUI indicating the selection of an appropriate location of the robotic cart. Upon receiving the user's confirmation, the system may generate a movement path for the robotic cart. Alternatively, the user may be guided to manually move the robotic cart to the desired location as described above.

[0076]

[0092] As described above, any suitable method may be employed to process real-time sensor data (e.g., 3D point clouds) for segmentation, object recognition, and collision avoidance. One or more objects may be segmented in the working space of the robotic arm. For example, a deep learning method such as an automated pipeline engine may be provided for processing lidar data. The pipeline engine may be composed of a plurality of components or layers. The pipeline engine may be configured to preprocess a continuous stream of raw lidar data or batch data transmitted from the Lidar system. In some cases, the data may be processed so that it can be incorporated into machine learning analysis. In some cases, the data may be processed to provide details at various levels of understanding, which may include, by way of non-limiting example, dimensions, weight, composition, identity, degree of collision risk, mobility, etc. In some cases, the pipeline engine may be composed of a plurality of components that perform various functions for extracting various levels of information from 3D point cloud data. In some cases, the pipeline engine may further include various basic data processing such as data normalization, labeling of data with metadata, tagging, data alignment, data segmentation, etc. In some cases, the processing method may be programmable through an API by a developer who constructs the pipeline.

[0077]

[0093] In some embodiments, the pipeline engine may utilize machine learning techniques for data processing. In some embodiments, raw Lidar data may be supplied to a first layer of a pipeline engine that can extract primitives such as edges, corners, surfaces, etc. of one or more target objects using a deep learning architecture. In some cases, the deep learning architecture may be a convolutional neural network (CNN). A CNN system typically consists of different types of layers, namely, convolutional, pooling, upscaling, and fully connected neural networks. In some cases, activation functions such as rectified linear units may be used in some layers. In a CNN system, there may be one or more layers for each type of operation. The input data of a CNN system may be data to be analyzed such as 3D radar data. The simplest architecture of a convolutional neural network starts with an input layer (e.g., an image), followed by a series of convolutional layers and pooling layers, and ends with a fully connected layer. In some cases, a layer of ReLU activation function follows the convolutional layer. Various other activation functions can also be used, such as saturated hyperbolic tangent, identity, binary step, logistic, inverse tangent, soft sign, parametric rectified linear unit, exponential linear unit, softplus, bent identity, soft exponential, sinusoid, sinc, Gaussian, sigmoid functions. The convolutional layer, pooling layer, and ReLU layer may function as learnable feature extractors, while the fully connected layer functions as a machine learning classifier.

[0078]

[0094] In some cases, the convolutional layer and the fully connected layer may contain parameters or weights. These parameters or weights may be learned during the training phase. The parameters may be trained using gradient descent so that the class scores calculated by the CNN match the labels in the training set of each 3D point cloud image. The parameters may be obtained from an error backpropagation neural network training process that may or may not be executed using the same hardware as the production process or application process.

[0079]

[0095] The convolutional layer may include one or more filters. These filters are to operate when seeing the same specific structure within the input data. In some cases, the input data is a 3D image, and in the convolutional layer, one or more filter operations may be applied to the pixels of the image. The convolutional layer may include a set of learnable filters that slide spatially over the image and calculate the dot product between the entries of the filter and the input image. The filter operation may be implemented as the convolution of the kernel over the entire image. The kernel may include one or more parameters. The results of the filter operation may be summed across channels to provide the output from the convolutional layer to the next pooling layer. The convolutional layer may perform high-dimensional convolution. For example, a 3D feature map or input 3D data may be processed by a group of 3D kernels in the convolutional layer.

[0080]

[0096] The output generated by the first layer of the pipeline engine may be supplied to a second layer configured to elicit an understanding of the target object such as shape, material, subsurface structure, etc. In some cases, the second layer may also be implemented using a machine learning architecture.

[0081]

[0097] The output generated by the second layer is then supplied to a third layer of the pipeline engine configured to perform interpretations and decision-making such as object recognition, separation, segmentation, collision avoidance, target dynamics (e.g., mobility), identity recognition, type classification, etc. In some cases, the dynamics or mobility of the object may be used to determine a collision avoidance scheme.

[0082]

[0098] The pipeline engines described in this specification may be implemented by one or more processors. In some embodiments, the one or more processors may be programmable processors (e.g., central processing units (CPUs), graphics processing units (GPUs), general-purpose processors or microcontrollers) in the form of fine-grained spatial architectures such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and / or one or more advanced RISC machines (ARM) processors. In some embodiments, the processor may be a processing device of a computer system.

[0083]

[0099] In some cases, to reduce the noise effect, a Bayesian estimation method (e.g., Kalman filtering) is applied to fit the point cloud to the rigid body model of the bed to align the robotic cart with the bed. In some cases, when the target object (e.g., patient bed) is not detected (907), the process may proceed to notify (921) the user that the position of the robotic cart is invalid (e.g., outside the appropriate area). For example, when the bed is not detected, a notification indicating that the position of the cart is invalid may be displayed on the GUI to notify the user.

[0084]

[0100] When a target object such as a patient bed is detected (907), this method may proceed to model coordinate registration 909 by mapping the position and orientation of the patient bed model to the coordinates of the robotic system (e.g., by pre-registration of the coordinate frames of the sensor and the robotic system). Next, this method may include an algorithm to determine whether the system is within an acceptable workspace. As an example, this algorithm may include calculating (911) the position and orientation error and comparing it with a threshold to determine whether the system is within an acceptable workspace.

[0085]

[0101] In some cases, the system may provide both an autonomous mode and a manual positioning mode. When the autonomous mode is enabled once (e.g., selected by the user) (913), the system may execute an active system positioning algorithm. The active system positioning algorithm may execute a path plan 915, for example, to generate a path in the target direction from the current position to the target position using an appropriate movement speed, acceleration, etc. For example, the wheel path plan of the robotic cart is generated or updated to eliminate the relative orientation error between the desired orientation and the sensed orientation calculated in the previous operation 911. The path is executed (917), and the movement of the wheels of the robot is controlled to move the robotic cart to the desired orientation and position.

[0086]

[0102] When the system disables the active system positioning algorithm in a semi-autonomous mode or the like (913), the user may be informed (919) of the state of the effectiveness of the system positioning relative to the bed, for example, an acceptable position or an unacceptable position, and the user may manually control the position of the robotic cart.

[0087]

[0103] Figures 10 through 12 illustrate examples of collision avoidance. As described above, the real-time 3D map of the operating environment may include obstacle information and the relative positions with respect to one or more components of the robotic endoscope system. The system may determine a proximity threshold and autonomously move one or more components of the system to avoid a collision. For example, as shown in FIG. 10, when the robotic arm is operating during a procedure, the sensor may detect an undesirable proximity between the arm and the monitor of the system 1000. The proximity is detected by analyzing a sensor signal that may include depth information. In response to the proximity, the user may be informed of a potential collision by a UI warning. Alternatively or additionally, the robotic arm may be automatically moved away from the monitor, or the monitor may be actuated away from the arm 1010. For example, the monitor may be attached to a robotic support that may be actuated to move the monitor in response to a control signal. As shown in Example 1010, when the possibility of a collision between the monitor and any part of the robotic arm is detected, both the robotic support of the monitor and the robotic arm may be actuated to move away from each other. Alternatively, as shown in FIG. 11, when the possibility of a collision between the monitor and any part of the robotic arm 1100 is detected, the robotic arm may be actuated to move away from the monitor.

[0088]

[0104] FIG. 11 shows an example where, when an operator moves a monitor into the working space of a robotic arm, the system may detect that the proximity to the monitor is approaching a threshold value and may automatically change the posture of the robotic arm. Due to the redundancy of the robotic arm, advantageously, it may be possible to change the posture of the robotic arm while maintaining the position and orientation of the IDM. FIG. 12 shows an example where, during a procedure, a device such as a fluoroscopic C-arm 1207 is brought into the working space. The system detects the device 1207 and the proximity to the device and may automatically adjust the posture of the robotic arm and / or the robotic cart to provide sufficient clearance for the C-arm to take a fluoroscopic image of the patient and / or complete a tomosynthesis sweep. The posture of the robotic arm may change from a first posture 1203 to a second posture 1205 while the position and orientation of the IDM 1201 remain unchanged.

[0089]

[0105] FIG. 13 shows an example of a collision avoidance algorithm 1300. In the illustrated process, Lidar data (e.g., 3D point cloud) is used for real-time detection of objects in the operating environment and acquisition of depth information, but it should be noted that any other suitable sensors (e.g., stereo cameras) and methods described elsewhere in this specification can also be utilized. The steps for capturing 3D point cloud data and segmenting objects in the working space for object detection may be the same as those described in FIG. 9. For example, the input data may include 3D point cloud data captured by a Lidar device 1301, and the above-described deep learning method may be used to perform object segmentation and object recognition 1303 for collision avoidance. For example, when an object (e.g., a display device, a monitor) is brought into the operating environment, a segmentation algorithm may be executed to detect and recognize the object. Depth information from the 3D point cloud may be used to assign distances to points within the segmented region (e.g., the region of the monitor).

[0090]

[0106] In some cases, signal filtering is performed to reduce the impact of sensing uncertainty (1305). In some cases, when the detected object can be represented by a pre-computed model, a Bayesian filtering method such as Kalman filtering may be applied. For example, the pre-computed model may include an object estimation algorithm, and the sensed point cloud of the monitor is an input to the estimation algorithm that estimates the state (position, orientation) of the monitoring device.

[0091]

[0107] Next, the position and orientation of the point cloud are mapped to the arm-based coordinate frame (1307). The positions of the components of a robotic endoscope system such as a robotic arm and the positions of the points within the segmented monitor area may be registered in the same reference frame to determine the minimum proximity between any part of the monitor and any part of the robotic arm (calculate the contact distance (1309)). The position and orientation of the robotic arm may be obtained based on kinematic mapping.

[0092]

[0108] In some cases, a proximity determination algorithm may be executed to determine the likelihood of a collision (1311) as to whether the minimum proximity violates a predetermined proximity threshold. In some cases, if it is determined that the predetermined proximity threshold is violated, the system may activate an admittance controller to reconfigure the pose of the robotic arm (1313) (e.g., by actuating one or more joints / links of the robotic arm). For example, the admittance controller may execute an admittance control algorithm that defines the most proximate collision point as a repulsive force and reconfigure the robotic arm to increase the (minimum) distance between the robotic arm and the monitor. An algorithm for repositioning the arm is executed to avoid a collision before it occurs. In some cases, the admittance control algorithm may map the sensed proximity between any part of the robotic arm and an object to a virtual force applied to the robotic arm and calculate the output motion of the robotic arm as if an equivalent contact force had been applied to the robotic arm (through contact). For example, the admittance control algorithm may include mapping the shortest distance between an arm link and an obstacle to an input to the commanded task space velocity of the arm link. Using the geometric Jacobian of the arm, the task space motion command may be mapped to an arm joint command to achieve an arm motion that increases the distance between the arm and the obstacle.

[0093]

[0109] When the robotic system has more degrees of freedom than the task, i.e., when there is actuation redundancy, avoidance may be implemented via a redundancy resolution algorithm. For example, the elbow of the arm may be repositioned to prevent a collision without affecting the position and orientation of the end effector. Such an algorithm may be implemented by defining a secondary redundancy resolution task that repositions the arm so that the distance between the arm and the object (before the collision) is increased. In some cases, when there are multiple degrees of freedom in the redundancy, the robot command may be chosen to actuate the joints such that, for example, the norm of the joint velocity is minimized.

[0094]

[0110] In some embodiments, the robotic endoscope system may be able to autonomously adjust the position of the IDM relative to the patient-side mount based on buckling detection. During insertion of the flexible device into the anatomical structure, when the proximal end of the flexible endoscope is pushed, the flexible endoscope may deform and buckle as it passes through the bending angle. The deformation may occur during insertion because the flexible device may assume the minimum energy shape that can be a "close contact" with the tissue of the shaft. Buckling may occur when the distal portion of the shaft encounters resistance.

[0095]

[0111] During retraction of the flexible device, if the system operating direction reverses (e.g., moving backward or retracting), the "lost due to buckling" distance of the shaft may be alleviated. This phenomenon results in the recognition of a dead zone or system delay (the user input that commands the movement of the tip of the robotic endoscope is not directly mapped to the movement of the tip of the robotic endoscope). When the endoscope buckles, the endoscope may be retracted such that the robot operates with little movement of the tip of the endoscope.

[0096]

[0112] Eversion or torsion may cause potential damage because it exposes the sharp edges of the twisted elongated device and complicates the surgical procedure. Also, a bent or twisted elongated device may cause loss of device position / shape control during insertion and retraction into the system and may prevent the passage of the instrument. Furthermore, an everted or twisted device may not be able to provide sufficient arm length towards the target anatomical structure to perform the intended task.

[0097]

[0113] The robotic endoscope system of this specification may employ responsive insertion and retraction speed control of a flexible endoscope. The responsive speed control method of this specification may automatically correct the difference in movement between the tip of the endoscope and a speed command (e.g., an instrument drive mechanism (IDM) command). In some cases, the speed control of the endoscope may be such that while the robotic arm autonomously reconfigures to avoid collisions with other objects, the speed of the tip of the endoscope is maintained and may be executed together with the above collision avoidance algorithm so as not to be affected.

[0098]

[0114] Unlike conventional buckling detection methods based on the difference in the position of the endoscope device (e.g., the predicted position and the measured tip position), the methods and systems of this specification may automatically correct buckling / deformation during insertion and retraction based on the speed measured at the tip of the endoscope and the speed control command. Thereby, advantageously, shape sensing for determining the shape or position of the endoscope device and the use of additional imaging approaches are avoided.

[0099]

[0115] During insertion of the endoscope device, the endoscope device may receive a speed command. The speed command may be provided by user input. For example, the user may provide a control command indicating the desired / predicted speed of the tip of the endoscope via a control interface of the endoscope device. The control interface may include various devices such as a touch screen monitor, a joystick, a keyboard, and other interactive devices. The user may be able to navigate and / or control the movement of the robotic arm and the movement of the catheter (e.g., tip speed) using a user input device. The user input device may have any type of user interactive component such as a button, a mouse, a joystick, a trackball, a touchpad, a pen, an imaging device, a motion capture device, a microphone, a touch screen, a handheld list gimbal, an exoskeleton glove, or other user interactive systems such as a virtual reality system, an augmented reality system.

[0100]

[0116] User input for commanding the speed of the endoscope tip may be received via an input device. For example, the depression of a joystick may be mapped to an analog value indicating the tip speed / velocity. For example, a half - depression of the joystick may be mapped to 3 mm / s, a full - depression may be mapped to 6 mm / s, and no depression may be mapped to 0 mm / s.

[0101]

[0117] Based on a particular user input device, the user input may be processed and converted to a desired / commanded speed of the tip of the catheter / endoscope. Next, a tip speed error is calculated. The tip speed error is the difference between the desired / commanded tip speed and the speed of the endoscope tip. In some embodiments, the speed of the endoscope tip may be measured based on sensor data. In some cases, the sensor data may include position and orientation information of the distal tip of the endoscope.

[0102]

[0118] In some cases, the sensor signal may be obtained by a positioning sensor. For example, the sensor signal may be obtained by an electromagnetic coil located at the distal end used in an electromagnetic tracking system to detect the position and orientation of the distal end of the endoscope. For example, a positioning sensor such as an electromagnetic (EM) sensor may be embedded at the distal tip of the catheter, and the EM field generator may be placed adjacent to the patient's torso during the procedure. The EM field generator may identify the position of the EM sensor in 3D space, or may identify the position and orientation of the EM sensor in 5D or 6D space. The tip position of the endoscope measured by the EM sensor, p e =EM sensor (tip) position may be represented in the field generator frame.

[0103]

[0119] Next, the linear velocity of the endoscope tip may be calculated. The linear velocity may be calculated using a time differentiation method such as backward Euler differentiation. In some cases, the endoscope tip velocity is calculated as the filtered time derivative of the tip position (e.g., measured by an EM sensor and represented in the submitted generator frame) and may be projected in the direction of travel. This projection may be necessary because the velocity command provided by the user is based on the integrated position change occurring in the n direction. In some cases, a low-pass filter may be applied to generate the filtered time derivative data.

[0104]

[0120] The endoscope tip velocity is calculated as the filtered time derivative of the tip position (e.g., measured by an EM sensor and represented in the submitted generator frame) and the projection matrix v n = nn T filt(dp e / dt) may be used to project it in the direction of travel of n.

[0105]

[0121] Here, n is a unit vector indicating the direction of travel of the endoscope tip, represented in the field generator frame. There may be a mechanical offset between the EM sensor and the scope tip, and the mechanical offset may be calibrated for each endoscope device. dp e / dt represents the time derivative of the endoscope tip, and nn T is a projection matrix that maps the aforementioned velocity in the direction of travel of the endoscope tip (i.e., velocities other than the direction of travel are ignored). The velocity v n may not be affected by the joints because the movement of the endoscope tip by the joints is orthogonal to n.

[0106]

[0122] After the endoscopic tip speed is calculated, an endoscopic tip speed error may be calculated and may be further processed for safety checks. In some cases, the safety check may include multiple checks. For example, multiple checks may include determining whether the endoscopic tip is stationary (e.g., the tip speed is approximately zero) while the insertion distance of the handle portion of the endoscope (e.g., IDM) exceeds a distance threshold. In another example, multiple checks may include determining whether the endoscopic tip is retracting (e.g., a negative tip speed error) while the handle portion of the endoscope (e.g., IDM) is being inserted, and in the case of retraction, an escape may have occurred. In yet another example, multiple checks may include determining whether the insertion force exceeds a force threshold. As used herein, the term "insertion distance" may refer to the distance along the navigation path.

[0107]

[0123] This method may include closed-loop control of the tip speed and may reduce the tip speed error. The tip speed of the endoscope may be controlled based on the tip speed error calculated during operation, and the tip speed error may be used as a feedback signal to command the movement of the IDM. The user speed input may be mapped to a command for controlling the movement of the IDM (e.g., the speed for moving the IDM along the insertion axis). The command may be a control signal for controlling the motor of the robotic arm, thereby controlling the movement of the IDM (i.e., the proximal end of the endoscope). The endoscopic tip speed may be calculated based on the inverse kinematics of the robotic arm. The endoscopic tip speed is then used, as described above, to calculate the tip speed in the direction of travel by projection onto the direction of travel of n using the projection matrix. In some cases, the feedback signal may be the projection of the tip speed processed by a low-pass filter.

[0108]

[0124] Based on the control algorithm, an operation command for operating the robotic arm is generated, thereby affecting the tip speed of the endoscope. As described above, the effects on the operation of the IDM (e.g., insertion speed, insertion distance) and the operation of the endoscope tip (e.g., tip speed) may not exactly match due to the meandering, buckling, and / or escape of the navigation path.

[0109]

[0125] During the retraction of the medical device, it may be possible to perform backlash correction of the retraction control by using the previously determined deformation loss during insertion as a feedforward term. Thereby, the deformation loss occurring during insertion is reduced prior to the movement of the tip during retraction.

[0110]

[0126] FIG. 14 shows an example of a flexible endoscope 1400 according to some embodiments of the present disclosure. As shown in FIG. 14, the flexible endoscope 1400 may include a handle / proximal portion 1409 and a flexible elongated member that is inserted into the subject. The flexible elongated member may be the same as those described above. In some embodiments, the flexible elongated member may include a proximal shaft (e.g., insertion shaft 1401), an operable tip (e.g., tip 1405), and an operable section (active bending section 1403). The active bending section and the proximal shaft section may be the same as those described elsewhere in this specification. The endoscope 1400 may also be referred to as an operable catheter assembly as described elsewhere in this specification. In some cases, the endoscope 1400 may be a single-use robotic endoscope. In some cases, the entire catheter assembly may be disposable. In some cases, at least a portion of the catheter assembly may be disposable. In some cases, the entire endoscope may be removed from the instrument drive mechanism and disposed of. In some embodiments, the endoscope may include various levels of stiffness along the shaft to improve functional operation.

[0111]

[0127] The endoscope or steerable catheter assembly 1400 may include a handle portion 1409 that may include one or more components configured to process image data, supply power, or establish communication with other external devices. For example, the handle portion may include circuitry and communication elements that enable electrical communication between the steerable catheter assembly 1400 and an instrument drive mechanism (not shown), as well as any other external system or device. In another example, the handle portion 1409 may include circuit elements such as a power source for supplying power to the electronics of the endoscope (e.g., a camera, an electromagnetic sensor, and LED lights).

[0112]

[0128] One or more components located in the handle may be optimized to allocate expensive and complex components to a robotic support system, a handheld controller, or an instrument drive mechanism, thereby reducing costs and simplifying the design of a disposable endoscope. The handle portion or proximal portion may provide electrical and mechanical interfaces that enable electrical communication and mechanical communication with the instrument drive mechanism. The instrument drive mechanism may include a set of motors that are actuated to rotationally drive a set of pull wires of the catheter. The handle portion of the catheter assembly may be attached to the instrument drive mechanism such that its pulley / capstan assembly is driven by the set of motors. The number of pulleys may vary depending on the configuration of the pull wires. In some cases, one, two, three, four, or more pull wires may be used to articulate a flexible endoscope or catheter.

[0113]

[0129] The handle portion may be designed to enable the robotic bronchoscope to be disposed of at a low cost. For example, typical manual and robotic bronchoscopes may have a cable at the proximal end of the bronchoscope handle. The cable often includes fiber optic cables for illumination, camera video cables, and other sensor fibers or cables such as electromagnetic (EM) sensors or shape sensing fibers. Such complex cables are expensive and may increase the cost of the bronchoscope. The provided robotic bronchoscope may have an optimized design that allows for the adoption of a simplified structure and components while maintaining mechanical and electrical functions. In some cases, the handle portion of the robotic bronchoscope may adopt a cableless design while providing a mechanical / electrical interface with the catheter.

[0114]

[0130] The electrical interface (e.g., a printed circuit board) may enable image / video data and / or sensor data to be received by the communication module of the instrument drive mechanism and transmitted to other external devices / systems. In some cases, the electrical interface may establish electrical communication without cables or wires. For example, the interface may include pins soldered to an electronic substrate such as a printed circuit board (PCB). For example, a receptacle connector (e.g., a female connector) may be provided as a mating interface on the instrument drive mechanism. This may advantageously allow the endoscope to be quickly inserted into the instrument drive mechanism or robotic support without using additional cables. Such a type of electrical interface may also function as a mechanical interface such that both a mechanical coupling and an electrical coupling are established when the handle portion is inserted into the instrument drive mechanism. Alternatively, or in addition, the instrument drive mechanism may provide only a mechanical interface. The handle portion may be in electrical communication with a modular wireless communication device or other user device (e.g., a portable / handheld device or controller) to transmit sensor data and / or receive control signals.

[0115]

[0131] In some cases, the handle portion 1409 may include one or more mechanical control modules such as a luer 1411 for interfacing with an irrigation system / suction system. In some cases, the handle portion may include a lever / knob for joint control. Alternatively, the joint control may be located on another controller attached to the handle portion via an instrument drive mechanism.

[0116]

[0132] The endoscope may be attached to a robotic support system or a hand-held controller via an instrument drive mechanism. The instrument drive mechanism may be provided by any suitable controller device (such as a hand-held controller) that may or may not include a robotic system. The instrument drive mechanism may provide a mechanical and electrical interface with the steerable catheter assembly 1400. The mechanical interface may be able to releasably couple the steerable catheter assembly 1400 to the instrument drive mechanism. For example, the handle portion of the steerable catheter assembly may be attached to the instrument drive mechanism via quick attachment / release means such as magnets, spring levels, etc. In some cases, the steerable catheter assembly may be manually coupled or released from the instrument drive mechanism without using tools.

[0117]

[0133] In the illustrated example, the distal tip of the catheter or endoscope shaft is configured to articulate / flex in two or more degrees of freedom to provide a desired camera field of view or to control the direction of the endoscope. As shown in the example, an imaging device (e.g., a camera), a position sensor (e.g., an electromagnetic sensor) 1407 is located at the tip of the catheter or endoscope shaft 1405. For example, the line of sight of the camera may be controlled by controlling the articulation of the active bending section 1403. In some cases, the angle of the camera may be adjustable so that the line of sight can be adjusted without articulating the distal tip of the catheter or endoscope shaft, or in addition to articulating it. For example, the camera may be oriented (e.g., tilted) at an angle with respect to the axial direction of the tip of the endoscope with the help of an optimal component.

[0118]

[0134] The distal tip 1405 may be a rigid component that enables positioning of sensors such as electromagnetic (EM) sensors, imaging devices (e.g., cameras), and other electronic components (e.g., LED light sources) embedded in the distal tip.

[0119]

[0135] In real-time EM tracking, an EM sensor composed of one or more sensor coils embedded at one or more locations and orientations of a medical device (e.g., the tip of an endoscope tool) measures changes in the EM field generated by one or more static EM field generators placed close to the patient. The position information detected by the EM sensor is stored as EM data. The EM field generator (or transmitter) may be placed close to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensors. The magnetic field may induce a small current in the sensor coils of the EM sensor, which is analyzed to determine the distance and angle between the EM sensor and the EM field generator. For example, the EM field generator may be placed close to the patient's torso during the procedure to identify the position of the EM sensor in 3D space, or to identify the position and orientation of the EM sensor in 5D or 6D space. This may provide a visual guide to the operator when driving the bronchoscope towards the target site.

[0120]

[0136] An endoscope may have an elongate member with a unique design. In some cases, the active bending section 1403 and the proximal shaft of the endoscope may be composed of a single tube with a series of cuts (e.g., reliefs, slits, etc.) incorporated along its length to enable improved flexibility, desirable rigidity, and an anti-escape function (e.g., a function that defines a minimum bending radius).

[0121]

[0137] As described above, the active bending section 1403 may be designed to enable bending in two or more degrees of freedom (e.g., articulation). Due to the unique structure of the active bending section, a greater degree of flexion such as 180 degrees or 270 degrees (or other articulation parameters for clinical applications) can be achieved. In some cases, a variable minimum bending radius along the axis of the elongate member may be provided such that the active bending section can include two or more different minimum bending radii.

[0122]

[0138] The articulation of the endoscope may be controlled by applying a force to the distal end of the endoscope via one or more pull wires. One or more pull wires may be attached to the distal end of the endoscope. In the case of multiple pull wires, pulling one wire at a time may change the orientation of the distal tip in any direction such as up, down, left, right, or as required. In some cases, the pull wire may be fixed at the distal tip of the endoscope, pass through the bending section, and enter the handle where it is coupled to a drive component (e.g., a pulley). This handle pulley may interact with an output shaft from a robotic system.

[0123]

[0139] In some embodiments, the proximal end or proximal portion of one or more pull wires may be operably coupled to various mechanisms (e.g., gears, pulleys, capstans, etc.) in the handle portion of the catheter assembly. The pull wire may be a metal wire, cable or thread, or it may be a polymer wire, cable or thread. The pull wire may also be made of natural or organic materials or fibers. The pull wire may be any type of suitable wire, cable or thread capable of supporting various types of loads without deformation, significant deformation, or breakage. Since the distal end / distal portion of one or more pull wires may be fixed or integrated with the distal portion of the catheter, operation by the control unit of the pull wire may apply a force or tension to the distal portion that can steer or articulate at least the distal portion of the catheter (e.g., in any direction up, down, pitch, yaw, or intermediate).

[0124]

[0140] The pull wire may be made of any suitable material such as stainless steel (e.g., SS316), metal, alloy, polymer, nylon or biocompatible material. The pull wire may be a wire, cable or thread. In some embodiments, different pull wires may be made of different materials to vary the load-bearing capacity of the pull wire. In some embodiments, different sections of the pull wire may be made of different materials to vary the stiffness and / or load-bearing along the pull. In some embodiments, the pull wire may be utilized for the transfer of electrical signals.

[0125]

[0141] Proximal design can improve the reliability of a device without incurring extra costs, realizing a low-cost single-use endoscope. In another aspect of the present invention, a single-use robotic endoscope is provided. The robotic endoscope may be a bronchoscope and may be the same as the steerable catheter assembly described elsewhere in this specification. Conventional endoscopes may have a complex design and are usually designed for reuse after a procedure, requiring thorough cleaning, disinfection, or sterilization after each procedure. Existing endoscopes are often designed with a complex structure so that the endoscope can withstand the cleaning, disinfection, and sterilization processes. The provided robotic bronchoscope may be a single-use endoscope that can advantageously reduce cross-contamination and infections between patients. In some cases, the robotic bronchoscope may be delivered to the physician in a pre-sterilized package and is intended to be disposed of after a single use.

[0126]

[0142] As shown in FIG. 15, the robotic bronchoscope 1510 may include a handle portion 1513 and a flexible elongated member 1511. In some embodiments, the flexible elongated member 1111 may include a shaft, a steerable tip, and a steerable / active bending section. The robotic bronchoscope 1510 may be the same as the steerable catheter assembly described in FIG. 14. The robotic bronchoscope may be a single-use robotic endoscope. In some cases, only the catheter may be disposable. In some cases, at least a portion of the catheter may be disposable. In some cases, the entire robotic bronchoscope may be released from the instrument drive mechanism and may be disposed of. In some cases, the bronchoscope may include varying levels of stiffness along the shaft to improve functional operation. In some cases, the minimum bending radius along the shaft may vary.

[0127]

[0143] The robotic bronchoscope may be releasably coupled to the instrument drive mechanism 1520. The instrument drive mechanism 1520 may be attached to the arm of the robotic support system or to any of the actuation support systems described elsewhere in this specification. The instrument drive mechanism may provide a mechanical and electrical interface with the robotic bronchoscope 1510. The mechanical interface may enable the robotic bronchoscope 1510 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the robotic bronchoscope may be attached to the instrument drive mechanism via quick attachment / detachment means such as magnets or spring-loaded levers. In some cases, the robotic bronchoscope may be manually coupled or detached from the instrument drive mechanism without using tools.

[0128]

[0144] FIG. 16 shows an example of an instrument drive mechanism 1620 that provides a mechanical interface with the handle portion 1613 of the robotic bronchoscope. As shown in this example, the instrument drive mechanism 1620 may include a set of motors that are actuated to rotationally drive a set of pull wires of a flexible endoscope or catheter. The handle portion 1613 of the catheter assembly may be attached to the instrument drive mechanism such that its pulley assembly or capstan is driven by the set of motors. The number of pulleys may vary depending on the configuration of the pull wires. In some cases, one, two, three, four, or more pull wires may be used to articulate the flexible endoscope or catheter.

[0129]

[0145] The handle portion may be designed to allow the robotic bronchoscope to be disposed of at low cost. For example, typical manual and robotic bronchoscopes may have a cable at the proximal end of the bronchoscope handle. The cable often includes fiber optic for illumination, a camera video cable, and other sensor fibers or cables such as electromagnetic (EM) sensors or shape sensing fibers. Such complex cables are expensive and can increase the cost of the bronchoscope. The provided robotic bronchoscope may have an optimized design that allows for the adoption of a simplified structure and components while maintaining mechanical and electrical functions. In some cases, the handle portion of the robotic bronchoscope may adopt a cable-free design while providing a mechanical / electrical interface with the catheter.

[0130]

[0146] FIG. 17 shows an example of the distal tip 1700 of an endoscope. In some cases, the distal portion or tip of the catheter 1700 may be substantially flexible so that it can be maneuvered in one or more directions (e.g., pitch, yaw). The catheter may include a tip portion, a bending section, and an insertion shaft. In some embodiments, the catheter may have variable bending stiffness along its longitudinal axis. For example, the catheter may include a plurality of sections having different bending stiffnesses (e.g., flexible, semi-rigid, and rigid). The bending stiffness may vary by selecting materials having different stiffnesses / hardnesses, changing the structure (e.g., cuts, patterns) in different segments, adding additional support components, or any combination of the above. In some embodiments, the catheter may have a variable minimum bending radius along its longitudinal axis. By selecting different minimum bending radii at different locations along the catheter, advantageously, an anti-escape function may be provided while the catheter is still able to reach areas that are difficult to reach. In some cases, since the proximal end of the catheter does not need to be highly bent, the proximal portion of the catheter may be reinforced by an additional mechanical structure (e.g., an additional material layer) to obtain a greater bending stiffness. Such a design may provide support and stability to the catheter. In some cases, the variable bending stiffness may be obtained by using different materials during the extrusion of the catheter. This advantageously may allow different stiffness levels along the shaft of the catheter in an extrusion manufacturing process without additional fastening or assembly of different materials.

[0131]

[0147] The distal portion of the catheter may be maneuvered by one or more pull wires 1705. The distal portion of the catheter may be made of any suitable material such as a copolymer, polymer, metal, or alloy so that it can be bent by the pull wires. In some embodiments, the proximal end or termination of one or more pull wires 1705 may be coupled to a drive mechanism (e.g., gears, pulleys, capstans, etc.) via the above-described fixation mechanism.

[0132]

[0148] The pull wire 1705 may be a metal wire, cable or thread, or it may be a polymer wire, cable or thread. The pull wire 1705 may also be made of natural or organic materials or fibers. The pull wire 1705 may be any suitable type of wire, cable or thread capable of supporting various types of loads without deformation, significant deformation or breakage. Since the distal end or portion of one or more pull wires 1705 may be fixed or integrated with the distal portion of the catheter, the operation by the control unit of the pull wire may apply a force or tension to the distal portion that can steer or articulate (e.g., in any direction such as up, down, pitch, yaw, or intermediate) at least the distal portion of the catheter (e.g., the flexible section).

[0133]

[0149] The catheter may have dimensions such that one or more electronic components can be integrated into the catheter. For example, the outer diameter of the distal tip may be approximately 4 to 4.4 millimeters (mm), and the diameter of the working channel may be approximately 2 mm such that one or more electronic components can be embedded in the wall of the catheter. However, it should be noted that based on different applications, the outer diameter can be in any range less than 4 mm or greater than 4.4 mm, and the diameter of the working channel can be in any range according to the dimensions of the tool or the specific application.

[0134]

[0150] One or more electronic components may include an imaging device, a lighting device, or a sensor. In some embodiments, the imaging device may be a video camera 1713. The imaging device may include an optical element and an image sensor for capturing image data. The image sensor may be configured to generate image data in response to the wavelength of light. Various image sensors for capturing image data, such as complementary metal-oxide semiconductor (CMOS) or charge-coupled device (CCD), may be employed. The imaging device may be a low-cost camera. In some cases, the image sensor may be provided on a circuit board. The circuit board may be an imaging printed circuit board (PCB). The PCB may include a plurality of electronic components for processing the image signal. For example, the circuit for a CCD sensor may include an A / D converter and an amplifier for amplifying and converting the analog signal provided by the CCD sensor. Optionally, the image sensor may be integrated with an amplifier and a converter for converting the analog signal to a digital signal so that a circuit board may not be required. In some cases, the output of the image sensor or the circuit board may be image data (digital signal) and may be further processable by a camera circuit or a processor of the camera. In some cases, the image sensor may include an array of optical sensors.

[0135]

[0151] The lighting device may include one or more light sources 1711 disposed at the distal tip. The light source may be a light-emitting diode (LED), an organic LED (OLED), a quantum dot, or any other suitable light source. In some cases, the light source may be a small LED for a compact design or a dual-tone flash LED lighting.

[0136]

[0152] The imaging device and the lighting device may be integrated into the catheter. For example, the distal portion of the catheter may include a suitable structure that matches at least the dimensions of the imaging device and the lighting device. The imaging device and the lighting device may be embedded in the catheter. FIG. 18 shows an exemplary distal portion of a catheter in which the imaging device and the lighting device are integrated. The camera may be located at the distal portion. The distal tip may have a structure for accommodating the camera, the lighting device, and / or the position sensor. For example, the camera may be embedded in the cavity 1810 at the distal tip of the catheter. The cavity 1810 may be formed integrally with the distal portion of the cavity and may have dimensions that match the length / width of the camera so that the camera does not move relative to the catheter. The camera may be adjacent to the working channel 1820 of the catheter to provide a close view of the tissue or organ. In some cases, the posture or orientation of the imaging device may be controlled by controlling the rotational movement (e.g., roll) of the catheter.

[0137]

[0153] Power to the camera may be provided by a wired cable. In some cases, the cable wire may be within a wire bundle that provides power not only to an illumination element or other circuitry at the distal tip of the catheter, but also to the camera. The camera and / or light source may be powered from a power source located in the handle portion via wires, copper wires, or any other suitable means extending along the length of the catheter. In some cases, real-time images or videos of the tissue or organ may be wirelessly transmitted to an external user interface or display. The wireless communication may be WiFi, Bluetooth, RF communication, or other forms of communication. In some cases, the images or videos captured by the camera may be broadcast to multiple devices or systems. In some cases, the image and / or video data from the camera may be transmitted along the length of the catheter to a processor located within the handle portion via wires, copper wires, or any other suitable means. The image or video data may be transmitted to an external device / system via a wireless communication component within the handle portion. In some cases, the system may be designed such that the wires are not visible or exposed to the operator.

[0138]

[0154] In conventional endoscopy, the illumination light may be provided by a fiber optic cable that transmits the light of a light source located at the proximal end of the endoscope to the distal end of the robotic endoscope. In some embodiments of the present disclosure, in order to reduce the complexity of the design, small LED lights may be used and may be embedded in the distal portion of the catheter. In some cases, the distal portion may include a structure 1430 having dimensions that match the dimensions of the small LED light source. As shown in the illustrated example, two cavities 1430 may be formed integrally with the catheter to accommodate two LED light sources. For example, the outer diameter of the distal tip may be about 4 to 4.4 millimeters (mm), and the diameter of the working channel of the catheter may be about 2 mm so that two LED light sources can be embedded at the distal end. The outer diameter may be in any range smaller than 4 mm or larger than 4.4 mm, and the diameter of the working channel may be in any range according to the dimensions of the tool or a particular application. Any number of light sources may be included. The internal structure of the distal portion may be designed to accommodate any number of light sources.

[0139]

[0155] In some cases, each of the LEDs may be connected to a power wire that can extend to the proximal handle. In some embodiments, the LEDs may be soldered to separate power wires that are later bundled to form a single twisted wire. In some embodiments, the LEDs may be soldered to a pull wire that supplies power. In other embodiments, the LEDs may be directly crimped or connected to a pair of power wires. In some cases, a protective layer, such as a thin layer of biocompatible adhesive, may be applied to the front of the LED to provide protection while allowing light emission. In some cases, an additional cover 1431 may be disposed on the front end face of the distal tip to provide accurate positioning of the LED and sufficient space for the adhesive. The cover 1831 may be composed of a transparent material that matches the refractive index of the adhesive so that the illumination light is not blocked.

[0140]

[0156] Preferred embodiments of the present invention are shown and described herein, but it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Here, numerous variations, modifications, and alternative forms will occur to those skilled in the art without departing from the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be used in practicing the present invention. The following claims define the scope of the present invention, and it is intended that methods and structures within the scope of these claims and their equivalents be covered by these claims.

Claims

1. A method for controlling a robotic endoscope system, comprising: generating a 3D depth map of the environment surrounding the robotic endoscope system; autonomously operating a mobile base of a robotic support system to a desired location relative to a patient bed based on the 3D depth map, wherein the robotic support system comprises a robotic arm coupled to the mobile base at a proximal end and to an instrument drive mechanism (IDM) at a distal end; and operating the robotic arm to autonomously align the IDM with a component coupled to the patient bed or that is part of the patient bed. A method as described above.

2. The method of claim 1, wherein the 3D depth map is generated at least in part based on 3D point cloud data.

3. The method of claim 2, further comprising processing the 3D depth map to detect the patient bed and calculating a position and orientation of the robotic support system relative to the patient bed.

4. The method of claim 1, wherein a flexible endoscope device is releasably coupled to the IDM after the IDM is aligned with the component coupled to the patient bed or that is part of the patient bed.

5. The method of claim 1, further comprising controlling movement of the robotic arm to move the IDM to a predetermined distance from the component coupled to the patient bed or that is part of the patient bed.

6. The method of claim 5, further comprising loading a flexible endoscope device coupled to the IDM at a proximal end and to the component at a distal end.

7. The method of claim 5, further comprising automatically adjusting the position of the IDM relative to the component upon detection of a buckling event.

8. The method of claim 1, further comprising detecting and recognizing an object within the environment and reconfiguring the robotic arm to avoid a collision with the object while maintaining the position and orientation of the IDM.

9. The method of claim 8, further comprising detecting buckling of a flexible catheter coupled to the IDM while the flexible catheter is inserted into a patient's body.

10. The method of claim 9, further comprising executing a response speed control algorithm to control the speed of a tip of the flexible catheter while reconfiguring the robotic arm to avoid a collision with the object.

11. The method of claim 1, wherein the IDM is autonomously aligned with respect to the component, at least in part, based on sensor data.

12. The method of claim 11, wherein the sensor data is captured by an electromagnetic sensor.

13. The sensor data is captured by a camera including a reference marker disposed on the component, The method of claim 11, wherein the 3D depth map includes at least the 3D position of the reference marker.

14. A system for controlling a robotic endoscope system, comprising: A memory storing computer-executable instructions; Communicating with the robotic endoscope system and executing the computer-executable instructions to Generate a 3D depth map of the environment surrounding the robotic endoscope system; Autonomously actuating a self-propelled base of a robotic support system to a desired location relative to a patient bed based on the 3D depth map, the robotic support system comprising a robotic arm coupled to the self-propelled base at a proximal end and an instrument drive mechanism (IDM) at a distal end; and Actuating the robotic arm to autonomously align the IDM with a component coupled to or part of the patient bed; One or more processors configured to perform; A system comprising.

15. The system of claim 14, wherein the 3D depth map is generated at least in part based on 3D point cloud data.

16. The system of claim 14, wherein the one or more processors are further configured to process the 3D depth map to detect the patient bed and calculate a position and orientation of the robotic support system relative to the patient bed.

17. The system of claim 14, wherein a flexible endoscope device is releasably coupled to the IDM after the IDM is aligned with the component coupled to or part of the patient bed.

18. The system of claim 14, wherein the one or more processors are further configured to control movement of the robotic arm to move the IDM to a predetermined distance from the component coupled to or part of the patient bed.

19. The system of claim 18, wherein the one or more processors are further configured to load a flexible endoscope device coupled to the IDM at a proximal end and to the component at a distal end.

20. The system of claim 18, wherein the one or more processors are further configured to automatically adjust the position of the IDM relative to the component upon detection of a buckling event.

21. The system of claim 14, wherein the one or more processors are further configured to detect and recognize an object within the environment and reconfigure the robotic arm to avoid a collision with the object while maintaining the position and orientation of the IDM.

22. The system of claim 21, wherein the one or more processors are further configured to detect buckling of a flexible catheter coupled to the IDM while the flexible catheter is inserted into a patient's body.

23. The system of claim 22, wherein the one or more processors are further configured to execute a response speed control algorithm to control the speed of the tip of the flexible catheter while reconfiguring the robotic arm to avoid a collision with the object.

24. The system of claim 14, wherein the IDM is autonomously aligned relative to the component, at least in part, based on sensor data.

25. The system of claim 14, wherein the sensor data is captured by an electromagnetic sensor.

26. The system of claim 14, wherein the sensor data is captured by a camera including a reference marker disposed on the component, and the 3D depth map includes at least the 3D position of the reference marker.

Images