Systems and methods for responsive insertion and retraction of a robotic endoscope

Abstract

A method for controlling the tip speed of an articulated flexible endoscope is provided. The method includes: generating a command to move the tip of the elongate member of the articulated flexible endoscope at an expected speed; receiving sensor data acquired by a sensor disposed at a distal tip portion of the elongate member to calculate the tip speed; calculating a difference between the expected speed and the tip speed; and controlling the tip speed based on the difference.

Description

[0001] Citation

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 324,746, filed Mar. 29, 2022, and U.S. Provisional Patent Application No. 63 / 480,502, filed Jan. 18, 2023, each of which is incorporated herein by reference in its entirety. BACKGROUND OF THE INVENTION

[0003] Endoscopic procedures use an endoscope to examine the interior of a hollow organ or body cavity. Unlike many other medical imaging techniques, the endoscope is inserted directly into the organ. Flexible endoscopes that rely on intuitive manipulation and control can be delivered for the diagnosis and treatment of diseases accessible through any natural orifice in the body. Depending on the clinical indication, the endoscope can be designated as a bronchoscope, ureteroscope, colonoscope, gastroscope, otolaryngoscope, and various other endoscopes. For example, a flexible bronchoscope can be used for the diagnosis and / or surgical treatment of lung cancer. However, one challenge of bronchoscopy is reaching the upper lobes of the lungs while navigating through the airways. In another example, flexible endoscopy has been used to examine and treat gastrointestinal (GI) tract disorders without creating an opening in the patient. The endoscope is introduced into the upper or lower GI tract via the mouth or anus, respectively. A miniature camera at the distal end captures images of the GI wall, which helps the clinician diagnose GI diseases. Simple surgical procedures (such as polypectomy and biopsy) can be performed by introducing a flexible tool through the working channel to reach the site of interest at the distal end.

[0004] Traditionally, endoscopes are reusable and may need to be thoroughly cleaned, disinfected, and / or sterilized after each use. In most cases, cleaning, disinfection, and sterilization can be aggressive processes to kill germs and / or bacteria. Such procedures can also be harsh on the endoscope itself. Therefore, the design of such reusable endoscopes is often complex, especially to ensure that the endoscope can withstand such harsh cleaning, disinfection, and sterilization protocols. Regular maintenance and repair of such reusable endoscopes may typically be required.

[0005] For instruments that are difficult to clean properly, low-cost disposable medical devices designated for single use have become popular. Single-use disposable devices can be packaged in sterile wrappers to avoid the risk of pathogenic cross-contamination of diseases such as HIV, hepatitis, and other pathogens. Hospitals generally readily accept the convenience of single-use disposable products because they no longer have to worry about product aging, overuse, breakage, malfunction, and sterilization. Traditional endoscopes typically include a handle that an operator uses to manipulate the endoscope. For single-use endoscopes, the handle usually encapsulates a camera, expensive electronics, and mechanical structures at the proximal end to transmit video and allow the user to manipulate the endoscope via a user interface. This can result in a high cost for the handle of single-use endoscopes.

[0006] Continuous robots and endoscopes are typically long and flexible. The shaft of the endoscope may be limited by bending or kinking when inserting the device into an anatomical structure. The passive deformation of the endoscope shaft may be the result of the insertion force and contact with the anatomical structure, and this deformation is difficult to model. Prolapse or kinking may cause potential damage because it may expose the sharp edges of the kinked slender device and complicate the surgical procedure. In addition, a bent or kinked slender device may cause the system to lose position / shape control of the device and may block the passage of instruments. Summary of the Invention

[0007] This document recognizes the need for a robotic endoscope that allows surgical or diagnostic procedures to be performed with improved performance and cost efficiency. This document also recognizes devices and systems that include an endoscope that can be disposable and may not require extensive cleaning procedures. The present disclosure provides low-cost, single-use articulating endoscopes for use in a variety of applications such as bronchoscopy, urology, gynecology, arthroscopy, orthopedics, otolaryngology, gastrointestinal endoscopy, neurosurgery, colonoscopy, and various other applications for diagnosis and treatment. In some embodiments, the present disclosure provides a single-use, robotically controlled disposable bronchoscope for use with a robotic system to enable diagnostic evaluation of injuries at any location in the pulmonary anatomy. It should be noted that the provided endoscope system can be used in a variety of minimally invasive surgical procedures, therapeutic or diagnostic procedures involving various types of tissues including heart, bladder, and lung tissues, and can be used in other anatomical regions of the patient's body such as the digestive system including but not limited to the esophagus, liver, stomach, colon, urethra or the respiratory system including but not limited to the bronchi, lungs, and various other regions.

[0008] In one aspect of the present disclosure, a method for controlling the tip speed of an articulating flexible endoscope is provided. The method includes: generating a command to move the tip of the elongate member of the articulating flexible endoscope at an expected speed; receiving sensor data acquired by a sensor disposed at the distal tip portion of the elongate member to calculate the tip speed; calculating the difference between the expected speed and the tip speed; and controlling the tip speed based on the difference.

[0009] In one aspect, a method for controlling the tip movement of an articulating flexible endoscope is provided. The method includes: generating a command to drive the elongate member of the articulating flexible endoscope along an anatomical path via an instrument drive mechanism (IDM); receiving sensor data acquired by a position sensor disposed at the distal tip portion of the elongate member; when determining that the distal tip portion is at a preselected position within the anatomical path, setting the movement of the distal tip to zero and calculating the movement of the distal tip portion within a time window; calculating the difference between the movement of the distal tip portion and the movement of the IDM within the same time window; and detecting a buckling event by comparing the difference with a threshold.

[0010] In some embodiments, the preselected position is the main carina. In some embodiments, the method further includes determining that the distal tip portion is at the preselected position based at least in part on the sensor data and a 3D model of the anatomical path. In some cases, the position sensor includes an electromagnetic sensor.

[0011] In some embodiments, the method further includes determining that the distal tip portion is at the preselected position based at least in part on image data acquired by a camera located at the distal tip portion. In some embodiments, the threshold is dynamically determined based on the target anatomical region towards which the articulating flexible endoscope is moving. In some embodiments, the threshold is a function of the tortuosity of the anatomical path.

[0012] In some embodiments, the size of the time window is between 4 seconds and 8 seconds. In some cases, the size of the time window is determined based on empirical data. In some embodiments, the movement of the distal tip portion within the time window is the accumulation of the distances traveled at each time step.

[0013] In some embodiments, the method further includes setting the insertion force applied by the IDM to zero when determining that the distal tip portion is at a preselected position within the anatomical path, and comparing the insertion force with a force threshold. In some embodiments, the method further includes generating and displaying a warning message on a user interface when determining that the insertion force is higher than the force threshold.

[0014] In some embodiments, the method further includes controlling the speed of the distal tip portion of the elongate member based on a difference between an expected speed and a measured tip speed. In some cases, the measured tip speed is calculated as the filtered time derivative of sensor data projected in the forward direction. In some cases, the expected speed is based on an input command. In some cases, controlling the speed of the distal tip portion of the elongate member includes closed-loop control. For example, the measured tip speed is processed by a low-pass filter to be used as a feedback signal for closed-loop control.

[0015] In some embodiments, the distal tip portion includes a structure for housing an imaging device, a position sensor, and an illumination device. In some embodiments, the proximal end of the elongate member of the articulated flexible endoscope is connected to an IDM for applying a force to one or more pull wires to articulate the distal tip portion of the elongate member, insert, or retract the articulated flexible endoscope. In some embodiments, the method further includes displaying, on a user interface, a message indicating a buckling event and a suggestion for taking action in response to the buckling event.

[0016] In a related but independent aspect, a system for controlling the tip movement of an articulated flexible endoscope is provided. The system includes: a memory storing computer-executable instructions; and one or more processors in communication with the articulated flexible endoscope and configured to execute the computer-executable instructions to: generate a command to drive the elongate member of the articulated flexible endoscope along an anatomical path via an instrument drive mechanism (IDM); receive sensor data acquired by a position sensor disposed at a distal tip portion of the elongate member; when determining that the distal tip portion is located at a preselected position within the anatomical path, set the movement of the distal tip to zero and calculate the movement of the distal tip portion within a time window; calculate a difference between the movement of the distal tip portion and the movement of the IDM within the same time window; and detect a buckling event by comparing the difference with a threshold.

[0017] It should be noted that the various components of the provided modular endoscope assembly and device can be used in a variety of minimally invasive surgical procedures, therapeutic or diagnostic procedures involving various types of tissues including cardiac, bladder, and lung tissues, and can be used in other anatomical regions of the patient's body, such as the digestive system including but not limited to the esophagus, liver, stomach, colon, urethra, or the respiratory system including but not limited to the bronchi, lungs, and various other regions.

[0018] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in the art from the following detailed description, which illustrates and describes only illustrative embodiments of the present disclosure. As will be recognized, the present disclosure is capable of other and different embodiments, and its several details are capable of modification in various obvious aspects, all without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0019] Incorporated by reference

[0020] All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. In the event of a conflict between the incorporated publications and patents or patent applications and the disclosure contained herein, the specification is intended to supersede and / or take precedence over any such conflicting material. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments of the invention, in which the principles of the invention are utilized, and the accompanying drawings (also referred to herein as "FIGURES"):

[0022] Figure 1 An example of a lumen network of a patient is schematically shown.

[0023] Figure 2 An example of a flexible elongate member undergoing extension and flexion in an airway / channel is shown.

[0024] Figure 3 An example of a dead zone occurring during retraction of a controlled endoscope is shown.

[0025] Figure 4 An exemplary algorithm for controlling the speed of an endoscope tip during a safety check operation according to some embodiments of the present disclosure is shown.

[0026] Figure 5 An exemplary warning algorithm is shown.

[0027] Figure 6 Examples of IDM insertion distance and endoscope tip insertion distance at different time points are shown.

[0028] Figure 7 An example of a dynamic tracking error threshold (DTET) as a function of insertion distance is shown.

[0029] Figure 8 An example of a user interface module is shown.

[0030] Figure 9 Shows an example of a robotic endoscope (e.g., bronchoscopy) system according to some embodiments of the present invention.

[0031] Figure 10 Illustrates an example of a flexible endoscope according to some embodiments of the present disclosure.

[0032] Figure 11 Shows an example of a robotic bronchoscope including a handle portion and a flexible elongate member.

[0033] Figure 12A Shows an example of an instrument drive mechanism (IDM) that provides a mechanical interface for the handle portion of a robotic bronchoscope.

[0034] Figure 12B Shows an example of a disposable endoscope detachably coupled to the IDM.

[0035] Figure 13 Shows an example of the distal tip of an endoscope.

[0036] Figure 14 Shows an example of a distal portion of a catheter having an integrated imaging device and a lighting device.

[0037] Figure 15 Shows an example of a distal portion of a catheter with an integrated imaging device and a lighting device.

[0038] Figure 16 Shows an example of a "zero position".

[0039] Figure 17 Shows an example of calculating a motion difference or motion mismatch within a predetermined time window.

[0040] Figure 18 Shows an exemplary algorithm for controlling the movement of an endoscope tip.

[0041] Figure 19 Shows an example of a threshold that varies based on an anatomical region.

[0042] Figure 20 and Figure 21 Shows an example of a GUI that displays a warning message. Detailed Description

[0043] Although various embodiments of the present invention have been shown and described herein, it will be readily apparent to those skilled in the art that such embodiments are provided by way of example only. Many variations, changes, and substitutions can be contemplated by those skilled in the art without departing from the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed.

[0044] The embodiments disclosed herein can be combined in one or more of many ways to provide improved diagnosis and treatment for a patient. The disclosed embodiments can be combined with existing methods and devices to provide improved treatment, such as in combination with known pulmonary diagnostic and surgical methods and surgical methods for other tissues and organs. It should be understood that any one or more of the structures and steps described herein can be combined with any one or more additional structures and steps of the methods and devices described herein, and the drawings and supporting text provide a description according to the embodiments.

[0045] Although the exemplary embodiments will be directed primarily to devices or systems for bronchoscopy, those skilled in the art will understand that this is not intended to be limiting, and the devices described herein can be used in other therapeutic or diagnostic procedures and various anatomical regions of the patient's body. The provided device or system can be used in urology, gynecology, rhinology, otology, laryngoscopy, gastroenterology using an endoscope, a combined device including an endoscope and instruments, an endoscope with positioning function, and those skilled in the art will understand that this is not intended to be limiting, and the devices described herein can be used in other therapeutic or diagnostic procedures and other anatomical regions of the patient's body, such as the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testicles, bladder, ears, nose, mouth, soft tissues (such as bone marrow, adipose tissue, muscle, glandular and mucosal tissues, spinal cord and nerve tissues, cartilage), hard biological tissues (such as teeth, bones, etc.), and body cavities and ducts (such as sinuses, ureters, colon, esophagus, pulmonary tract, blood vessels and throat), and various other tissues, in the form of: neuroendoscope, encephaloscope, ophthalmoscope, otoscope, rhinoscope, laryngoscope, gastroscope, esophagoscope, bronchoscope, thoracoscope, pleuroscope, angioscope, mediastinoscope, nephroscope, gastroscope, duodenoscope, cholangioscope, choledochoscope, laparoscope, amnioscope, ureteroscope, hysteroscope, cystoscope, proctoscope, colonoscope, arthroscope, sialoendoscope, orthopedic endoscope, etc., in combination with various tools or instruments.

[0046] The systems and devices herein can be combined in one or more of many ways to provide improved diagnosis and treatment for a patient. The systems and devices provided herein can be combined with existing methods and devices to provide improved treatment, such as in combination with known pulmonary diagnostic and surgical methods and surgical methods for other tissues and organs. It should be understood that any one or more of the structures and steps described herein can be combined with any one or more additional structures and steps of the methods and devices described herein, and the drawings and supporting text provide a description according to the embodiments.

[0047] Whenever the terms "at least", "greater than", or "greater than or equal to" precede the first value in a series of two or more numerical values, the terms "at least", "greater than", or "greater than or equal to" apply to each value in the series. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0048] Whenever the terms "not more than", "less than", or "less than or equal to" precede the first value in a series of two or more numerical values, the terms "not more than", "less than", or "less than or equal to" apply to each value in the series. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0049] As used herein, the terms distal and proximal generally may refer to positions referenced from the device and may be opposite to an anatomical reference. For example, the distal position of a main shaft or a main duct may correspond to the proximal position of an elongate member of a patient, and the proximal position of a main sheath or a main duct may correspond to the distal position of an elongate member of a patient.

[0050] Response insertion and retraction control

[0051] As described above, one challenge in navigating an intraluminal device (e.g., bronchoscopy) is reaching difficult-to-access regions (e.g., the upper lobe of the lung) when navigating through the airway. Figure 1 An example of a lumen network 100 of a patient is schematically shown. In the illustrated embodiment, the lumen network 100 is a bronchial network of the airways (i.e., lumens, branches) of a patient's lung. Although the illustrated intraluminal network 100 is a bronchial network of the airways within a patient's lung, the present disclosure is not limited to the illustrated example. The systems and methods described herein can be used to navigate any type of intraluminal network, such as a bronchial network, a renal network, a cardiovascular network (e.g., arteries and veins), the gastrointestinal tract, the urinary tract, as described elsewhere herein.

[0052] Reaching the periphery of the right upper lobe 101 of the lung through the apical segment (Ap) has always been a challenge for traditional bronchoscopes. In Figure 1In the illustrated Example 110, the combination of the 180-degree sharp angle between the trachea and the Ap and the large space at the junction of the intermediate bronchus, the right main bronchus, and the right upper lobe bronchus can cause the bronchoscope to kink downward and / or prolapse through the intermediate bronchus. When a section of the elongate device flexes as it moves through one or more passageways, the flexed section is compressed. This compression causes the elongate device to bend along its length and contact the wall of one or more passageways. In an area where there is one or more large passageways around the elongate device, when the distal end of the device encounters tissue resistance, the elongate device may bend within the large airway and move toward an area of the anatomy other than in the intended direction ("prolapse"). The likelihood of prolapse is greater when the tissue resistance is high (e.g., due to a too-small airway, incomplete inflation of the lung, tissue lesions, blind insertion, etc.). Example 110 shows that the deflection / kinking of the mirror axis can even deflect into the contralateral bronchus (e.g., when the tip cannot move forward but the operator continues to try to drive the mirror further).

[0053] Figure 2 Examples of prolapse 210 and flexion 220 during the insertion of a flexible device are shown. When the flexible endoscope is pushed at the proximal end, the flexible endoscope can deform 210 as it passes through turns and flexions 220 during the insertion of the flexible device into the anatomy. The deformation can occur during insertion because the flexible device can assume a minimum energy shape, which can be the shaft "hugging" the tissue. Flexion can occur when the distal portion of the shaft encounters resistance.

[0054] During the retraction of the flexible device, when the system actuation direction is reversed, the "lost due to flexion" distance of the shaft can become slack. This phenomenon will result in a perceivable dead zone or system delay. For example, an input where the user commands the tip of the robotic endoscope to move will not directly map to the movement of the tip of the robotic endoscope. Figure 3 An example of the dead zone for controlling the retraction of the endoscope is shown. When the endoscope is flexed, the retraction of the endoscope can cause robotic actuation while there is little translational movement of the tip of the endoscope.

[0055] Prolapse or kinking can cause potential damage as it can expose sharp edges of the kinked elongate device and complicate the surgical procedure. Additionally, a bent or kinked elongate device can cause the system to lose position / shape control of the device during insertion and retraction and can block the passage of instruments. Further, a prolapsed or kinked device may not provide sufficient access to the target anatomy to perform the intended task. Current approaches to address kinking / prolapse issues can include detecting prolapse, using shape sensing, force sensing, medical imaging of the shape and position of the device (comparing the detected position and the expected position) to detect when prolapse can occur. However, such approaches require additional imaging or sensing methods to determine shape and deformation, which may not provide real-time control of the movement of the endoscopic tip and may further complicate the device or increase cost.

[0056] In one aspect of the present disclosure, methods and systems for responsive insertion and retraction speed control of a flexible endoscope are provided. The speed control methods herein can automatically correct for the motion differences between the endoscopic tip and the speed command (e.g., instrument drive mechanism (IDM) command). The method for controlling the tip movement of a flexible endoscope can have an integrated safety check for detecting buckling / deformation.

[0057] Unlike conventional methods based on buckling detection from endoscopic device position differences (e.g., expected position and measured tip position), the methods and systems herein can automatically correct for buckling / deformation during insertion and retraction based on the speed measured at the endoscopic tip and the speed control command. This facilitates avoiding shape sensing or using additional imaging methods to determine the shape or position of the endoscopic device. Additionally, speed-based insertion and retraction control allows for more responsive buckling automatic correction with minimal latency.

[0058] Figure 4 An exemplary algorithm 400 for controlling the speed of an endoscopic tip with an integrated safety check in accordance with some embodiments of the present invention is shown. During insertion of the endoscopic device 401, the endoscopic device can receive a speed command. The speed command can be provided by a user input 403. For example, a user can provide control instructions via a control interface of the endoscopic device indicating the desired / expected speed of the endoscopic tip.

[0059] The control interface of the endoscopic device can be part of a user interface module for an operator or user to interact with the endoscope (e.g., a bronchoscope) during a surgical procedure. Figure 8An example of the user interface module 933 is shown. The user interface module can be a handheld controller. In some cases, the user interface module can include a proprietary user input device and one or more additional elements detachably coupled to an existing user device to improve the user input experience. For example, a physical trackball or roller can replace or supplement the function of at least one of the virtual graphical elements (e.g., the navigation arrows displayed on a touchpad) displayed on a graphical user interface (GUI) by giving it a function similar to that of the replaced graphical element. Examples of user devices can include, but are not limited to, mobile devices, smart phones / cellular phones, tablet computers, personal digital assistants (PDAs), laptop or notebook computers, desktop computers, media content players, etc.

[0060] The user interface can include various devices, such as a touch screen display, a joystick, a keyboard, and other interaction devices. The user can use the user input device to navigate and / or control the movement of the robotic arm 911 and the movement of the catheter 920 (e.g., tip speed). The user input device can have any type of user interaction component, such as buttons, a mouse, a joystick, a trackball, a touchpad, a stylus, an image capture device, a motion capture device, a microphone, a touch screen, a handheld wrist gimbal, an exoskeleton glove, or other user interaction systems, such as a virtual reality system, an augmented reality system, etc.

[0061] User input for commanding the speed of the endoscope tip can be received via the input device. For example, pressing the joystick can be mapped to an analog value indicating the rate / speed of the tip. For example, pressing the joystick halfway can be mapped to 3 mm / s, pressing it fully can be mapped to 6 mm / s, and not pressing it can be mapped to 0 mm / s. When the joystick is in various positions, the input speed can be any value (e.g., continuous numbers).

[0062] Velocity input can be provided via any suitable user input device. In some cases, the user input device can be a haptic stylus device that physically contacts a touch display, and the user can control the robotic system by moving the haptic stylus device across the display. For example, one or more physical user input devices or add-on elements (e.g., trackball, joystick, or roller) can be coupled via haptic sensing or Bluetooth to a graphical user interface (GUI) provided on the user device. For example, a trackball, joystick, or roller can replace or supplement the functionality of at least one of the virtual graphical elements (e.g., navigation arrows, speed) displayed on the graphical user interface (GUI) by imparting it with a functionality similar to the replaced graphical element. The add-on element can be coupled to the GUI via physical contact in the touch screen, via an IO port, wired or wireless communication, such that user input received via the add-on element can be mapped to the input received by the virtual graphical elements presented on the GUI. Examples of user devices can include but are not limited to mobile devices, smart phones / cellular phones, tablet computers, personal digital assistants (PDAs), laptop or notebook computers, desktop computers, media content players, etc.

[0063] Return reference Figure 4 , based on a particular user input device, the user input can be processed to convert it into a desired / commanded speed 405 of the tip of the catheter / endoscope. Next, a tip speed error 407 is calculated. The tip speed error is the difference between the desired / commanded tip speed and the endoscope tip speed. In some embodiments, the speed of the endoscope tip can be measured based on sensor data. In some cases, the sensor data can include position and orientation information of the distal tip of the endoscope.

[0064] In some cases, the sensor signal can be acquired by a positioning sensor. For example, the sensor signal can be acquired by an electromagnetic coil located at the distal end, which is used together with an electromagnetic tracking system to detect the position and orientation of the distal end of the endoscope 417. For example, a positioning sensor (e.g., an electromagnetic (EM) sensor) can be embedded in the distal tip of the catheter, and the EM field generator can be placed beside the patient's torso during the procedure. The EM field generator can localize the position of the EM sensor in 3D space, or can localize the position and orientation of the EM sensor in 5D or 6D space. The endoscope tip position p e = EM sensor (tip) position, which can be represented in the field generator frame.

[0065] Next, the linear velocity of the endoscopic tip can be calculated based on the measured position data 419. The linear velocity can be calculated using a time derivative method (e.g., backward Euler derivative). In some cases, the endoscopic tip velocity can be calculated as the filtered time derivative of the tip position projected in the forward direction 423 (e.g., measured by the EM sensor represented in the field generator frame). This projection may be required because the user-provided velocity command is based on the integrated position change occurring in the direction of n. The EM sensor data can be obtained at a specific frequency (e.g., 10 - 60 Hz). In some cases, a low-pass filter can be applied to generate the filtered time derivative data 421.

[0066] The endoscopic tip velocity can be calculated as the filtered time derivative of the tip position projected in the forward direction using the projection matrix v n = nn T filt(dp e / dt) (e.g., measured by the EM sensor represented in the field generator frame) of the tip position projected in the forward direction of n, where n is the unit vector indicating the forward direction of the endoscopic tip represented in the field generator frame. There may be a mechanical offset between the EM sensor and the endoscopic tip, and the mechanical offset can be calibrated for each endoscopic device. dp e / dt represents the time derivative of the endoscopic tip, and nn T is the projection matrix that maps the above velocity to the forward direction of the endoscopic tip (i.e., ignores the velocity not in the forward direction). The velocity v n can be unaffected by the joints because the endoscopic tip translates due to the joints being orthogonal to n.

[0067] After calculating the endoscopic tip velocity, the endoscopic tip velocity error 407 can be calculated and further processed for safety checks 409. In some cases, the safety checks 409 can include multiple checks. For example, the multiple checks can include determining whether the endoscopic tip is stationary (e.g., the tip velocity is approximately zero) while the insertion distance of the handle portion of the endoscope (e.g., IDM) exceeds a distance threshold. In another example, the multiple checks can include determining whether the endoscopic tip retracts (e.g., negative tip velocity error) when the handle portion of the endoscope (e.g., IDM) is inserted, and if so, prolapse may occur. In another example, the multiple checks can include determining whether the insertion force exceeds a force threshold. The term "insertion distance" used herein can refer to the distance along the navigation path.

[0068] Method 400 can include closed-loop control of the tip velocity to reduce the tip velocity error 411. The tip velocity of the endoscope can be controlled based on the tip velocity error calculated at operation 407, which is used as a feedback signal. Figure 8Closed-loop control of the tip speed is schematically shown by the movement of the IDM. The user speed input can be mapped to commands to control the movement of the IDM (e.g., the speed of moving the IDM along the insertion axis). The command can be a control signal to control the motors of the robotic arm, thereby controlling the movement of the IDM (i.e., the proximal end of the endoscope). The endoscope tip speed can be calculated based on the inverse kinematics of the robotic arm. Then, using the endoscope tip speed, the tip speed in the forward direction is calculated by projecting in the forward direction of n using the projection matrix as described above. In some cases, the feedback signal can be the projection of the tip speed that is processed by a low-pass filter to filter out noise data.

[0069] Return Figure 4 , based on the control algorithm, generate a motion command to activate the robotic arm 413, thereby affecting the tip speed of the endoscope 415. Due to the tortuosity, buckling, and / or prolapse of the navigation path (as described above), the effect on the IDM movement (e.g., insertion speed, insertion distance) and the effect on the movement of the endoscope tip (e.g., tip speed) may not match perfectly.

[0070] In some embodiments, multiple safety checks can be combined, sorted, and selected according to a warning algorithm to trigger a warning or pause the operation. Figure 5 An exemplary warning algorithm 500 is shown. The warning algorithm can trigger an operation such as a warning or pause based on one or more of the multiple safety checks. In the method shown, a warning can be triggered based on a comparison between the endoscope tip insertion distance and the IDM insertion distance.

[0071] The process can start with obtaining EM sensor data (e.g., EM position and orientation signals) 501. The endoscope tip insertion distance can be calculated based on the obtained EM sensor data (e.g., EM position and orientation signals) 503. For example, the endoscope tip insertion distance can be calculated based on the sensor orientation information, which is used to distinguish lateral movement, joint movement, and movement generated by robot insertion. Robot joint position data 511 can also be obtained. As described later herein, the endoscope device can be attached to the robotic arm via the IDM. The position and orientation of the IDM can be calculated based on the joint position data using forward kinematics 513. The IDM insertion distance is calculated at a series of time points 515 and compared with the tip insertion distance 505.

[0072] Figure 6 Shows the IDM insertion distance X IDM (t) at time t 601 and the IDM insertion distance X IDM (t + 1) at time t + 1. The tip insertion distance is also calculated as Xtip. The IDM insertion distance X IDMThe difference between the tip insertion distance can be a deformation loss. The deformation loss can be caused by buckling, prolapse, kinking of the flexible catheter, and / or tortuosity of the navigation path. For example, a tortuous path can result in a greater deformation loss than navigating through a straight path.

[0073] Return reference Figure 5 , the difference / divergence can be compared to a threshold to determine the presence of buckling. The threshold can be dynamic based on one or more factors (such as the anatomical region and / or tortuosity of the navigation path).

[0074] In some cases, the dynamic threshold can be referred to as a dynamic tracking error threshold (DTET), which can trigger warnings and / or system safety operations when the tracking error (TE) exceeds the DTET. The tracking error can be the deformation loss as described above. As described above, the DTET can be dynamic at least in part based on the anatomical region through which the flexible catheter passes. In some cases, the DTET can be a function of the navigation path, insertion distance, or tortuosity of the path. The tortuosity of the path can be calculated, for example, as the ratio of the length of a segment of the path to the distance between its two ends.

[0075] Figure 7 An example of the dynamic tracking error threshold (DTET) as a function of the insertion distance is shown. Example A shows the DTET for a path with less tortuosity, so the total value of the DTET is a function of the insertion distance and the maximum value is less than 4 cm. The tracking error (TE) has been compared to the DTET at various insertion distances, and a warning can be triggered when the TE exceeds the DTET for a given insertion distance, for example, TE1. Example B shows the DTET for a path with greater tortuosity, so the total value of the DTET is a function of the insertion distance and the value is greater than 4 cm at a specific insertion distance (or specific anatomical region).

[0076] During retraction of the medical device, the previously determined deformation loss during insertion can be used as a feedforward term to compensate for retraction control in reverse. This reduces the deformation loss that occurred during insertion before the tip movement during retraction.

[0077] The dynamic tracking error threshold can be determined using any suitable method, such as based on historical data, computer tomography (CT) scans of the anatomy, 3D models of the navigation path, etc. For example, the tortuosity of the path can be calculated for various anatomical regions, segments, insertion lengths along the path, and the DTET can vary based on the anatomical region.

[0078] In some cases, in addition to the anatomical region, DTET can also consider the effect of the insertion force applied within a safety range or an acceptable deformation range. The safety range describes the range within which buckling will not damage the contacting tissues or organs within the patient's body. This safety range can be determined based on historical endoscopic data and patient data from previous similar procedures. Any suitable method can be used to determine such a safety range. For example, machine learning algorithms can be used to train a model to determine the safety range. Historical endoscopic data and patient data can be used to create a training dataset for training the model. As new sensor data is collected, the model can be continuously updated and improved.

[0079] Return reference Figure 5 , the dynamic tracking error threshold can be used to determine whether the difference triggers the tip tracking warning 507. Next, the process can continue to determine whether the tip moves backward when the IDM moves forward, i.e., inserts 509. If so, there may be a prolapse, and the prolapse warning 510 can be triggered.

[0080] The warning algorithm 500 is for illustrative purposes only. It should be noted that fewer or more safety checks can be performed. For example, in addition to prolapse and buckling detection, the algorithm can also include checking whether the insertion force exceeds a threshold. The insertion force can be detected by one or more force sensors coupled to the robotic arm of the surgical robot system. When the insertion force is close to or approaching the insertion force threshold within a predefined range, the surgical robot system sends visual, audio, and / or tactile feedback to the user via the system GUI. For example, a warning indicating that the insertion force is close to or approaching the insertion force threshold can be displayed on the GUI or conveyed to the user using a suitable UI function.

[0081] In some embodiments, instead of or in addition to detecting the speed difference, the method herein can also track the difference between the IDM insertion movement and the measured tip movement within a time window. For example, the movement of the endoscopic tip and the movement of the IDM can be added within a time window to determine the difference. Figure 17 An example of calculating the movement difference or movement mismatch within a predetermined time window is shown. For example, the distances traveled in each time step in the forward direction (e.g., d1 between T0 and T1, d2 between T1 and T2, etc.) can be added up for seven seconds and used as the total travel distances of the endoscopic tip and the IDM within the time window, respectively. The movement of the endoscopic tip can be measured by the EM sensor described elsewhere herein. The EM sensor data can be obtained at a specific frequency (e.g., 10 - 60 Hz).

[0082] The motion mismatch tracked within a time window can be compared with a threshold to determine the presence of a flexion event. In some cases, the time window is a window of four seconds, five seconds, six seconds, seven seconds, eight seconds, a number below five seconds, a number above eight seconds, or any number between integers. The length of the time window (e.g., seven seconds) can be selected such that the motion mismatch tracked within the time window is not overly sensitive while allowing for real-time detection of flexion events. In some cases, the size of the time window can be determined based on empirical data. In some cases, the length (window size) of the time window can be adjusted based on the application used (e.g., body region), the desired level of sensitivity, etc. The length of the time window can remain constant throughout the process. Alternatively, the length of the time window can vary as the endoscope navigates to different regions in the body.

[0083] In some cases, when inserting the endoscope using a controller command, only the forward motion of the IDM and the endoscope tip can be tracked. In some cases, if the amount of endoscope retraction commanded by the user exceeds a retraction threshold, the tracking of the motion mismatch between the IDM and the endoscope tip can be reset to zero.

[0084] As described above, the threshold of force and / or the threshold of endoscope tip motion can be based on the tortuosity of the path or anatomical region. Figure 19 Examples of thresholds that vary based on the anatomical region are shown. In some cases, the dynamic tracking error threshold, the motion mismatch threshold, and / or the force threshold can be determined based on the tortuosity of the path. The magnitude of the force exerted by the endoscope tip on the anatomical structure can be determined at least in part based on the force exerted by the IDM in the insertion direction and the friction between the endoscope and the path to the tip. The higher the tortuosity, the greater the resulting frictional force, and thus the smaller the force exerted by the endoscope tip on the anatomical structure, with the magnitude of the force sensed at the IDM being fixed. Figure 19 Examples of different thresholds selected according to which part of the lung the endoscope is driven within are shown. For example, due to the high tortuosity in the upper right region, a higher force threshold and a higher dynamic tracking error threshold / motion mismatch threshold can be selected; due to the medium tortuosity in the upper left region, a medium force threshold and a medium dynamic tracking error threshold can be selected; due to the lower tortuosity in the lower lobe region, a lower force threshold and a lower dynamic tracking error threshold can be selected. In some cases, the threshold is a function of the tortuosity of the path, and the function can be determined at least in part based on the anatomical model of the subject.

[0085] Detection using thresholds (e.g., force threshold and dynamic tracking error threshold) can be challenging because the amount of motion mismatch and the end effector force reaching the main carina of the lung can be variables affected by the initial settings. For example, as Figure 16As shown, the amount of motion mismatch and the end effector force reaching the main carina point 1600 can depend on variables in the setup, such as friction within the endotracheal tube, the mechanical interface with the bronchial adapter, the angle of the robotic arm after retraction, the endoscope stiffness curve, and / or different lubricant doses applied to the endoscope. The algorithms and methods herein can beneficially avoid noise or uncertainty introduced by initial setup conditions by setting the force and / or motion mismatch values to zero as the endoscope tip moves along the endotracheal tube 1601 to the main carina 1600. It should be noted that the zero position can be chosen as other suitable positions, such as a position within a certain distance from the main carina or a position depending on the anatomical structure.

[0086] In some cases, the main carina position 1600 can be preselected as the zero position. For example, when the endoscope is inserted into the trachea and reaches the main carina point 1600, the system can initiate tracking of the motion mismatch, starting from zero and comparing it with the motion mismatch (or dynamic tracking error threshold), as described elsewhere herein. Similarly, when the tip of the endoscope reaches the main carina position 1600, the force can also be set to zero, and the system can start tracking the force and comparing it with a force threshold to detect safety events.

[0087] In some cases, when it is determined that the endoscope tip is at the main carina 1600, the system can automatically set the force and / or motion mismatch between the IDM and the endoscope tip motion to zero. The systems and methods herein can use various suitable methods to detect the zero position at the main carina. In some cases, the system can utilize navigation data (e.g., EM sensor data) to determine whether the endoscope tip has reached the main carina. For example, by tracking the EM sensor data of the tip position within the CT scan model of the patient's lung, the system can automatically detect whether the endoscope tip has reached the main carina. In some cases, the system can employ an image recognition algorithm to detect the exit of the endoscope tip from the endotracheal tube 1601 into the trachea. For example, the method can use an image recognition algorithm to process the camera image (e.g., Figure 20 or Figure 21 the intraluminal view shown in ) to identify the bifurcation at the main carina. Based on the detection of the bifurcation, it can be determined that the endoscope tip is at the main carina 1600. Various other methods can also be used to automatically detect the zero position. For example, after the IDM is aligned with the bronchial adapter and the automatic retraction starting position, it can be determined based on human factors that the endoscope tip is at the main carina position 1600 at a fixed insertion distance. The above methods for detecting the zero position can be used alone or in any combination.

[0088] Figure 18An exemplary algorithm 1800 for controlling the movement of an endoscope tip is shown. Algorithm 1800 can start from detecting a zero position (e.g., the main carina 1801). The zero position can be detected using navigation data (e.g., EM sensor data), image recognition methods, and / or other methods as described above. After detecting the zero position, the force and / or movement mismatch of the endoscope tip can be set to zero 1803. The algorithm can also identify which lobe of the lung the endoscope is being driven towards. Based on the target area, force thresholds and movement thresholds (or DTET) can be determined 1805. In some cases, the algorithm can start tracking the movement mismatch within a sliding window (e.g., a window size of seven seconds) and continuously compare it with a movement mismatch threshold. For example, the sliding window can have a window size of seven seconds, and the sliding size can be one second, such that the comparison can be made every second. In some cases, the sliding size can be adjusted to one second, two seconds, three seconds, etc. to adjust the detection frequency.

[0089] When it is detected that the movement mismatch is greater than the movement mismatch threshold, a warning message can be displayed on the GUI to notify the user that a kinking or buckling event may have occurred. The user can choose to follow the suggestions in the message or continue to drive the endoscope. In some cases, when it is detected that the force is greater than the force threshold, a warning message can be displayed on the GUI and the robotic arm can stop moving. In some cases, the warning message can provide the user with instructions to remove the endoscope and restore the system (e.g., the system can automatically retract the IDM) 1807. In some cases, when it is detected that the endoscope is being driven towards a different anatomical area (e.g., lobe) 1809, the algorithm can repeat the operation of setting the movement mismatch and force to zero 1803 and repeat the process. In the example shown, during IDM / endoscope retraction, the algorithm can track the IDM retraction distance and use the IDM retraction distance as the endoscope tip retraction distance.

[0090] Detected buckling, prolapse, and large insertion forces can trigger warning messages and / or control signals to the system. Warning or feedback messages can be communicated to the user in various forms. In some cases, the warning or feedback message can indicate the type of problem, such as buckling, prolapse, or an unsafe insertion force. For example, a message or warning can be provided to be displayed on a graphical user interface (GUI) presented on a display that the operator uses to control the operation. In some cases, the system can also generate suggestions for the user to take action in response to the detected buckling event. For example, the system can generate suggestions for the user to move the endoscope backward, adjust the movement of the tip, reduce / adjust the insertion force provided by the IDM, etc. Figure 20 An example of a GUI that displays a warning message is shown. When endoscope buckling is detected, a notification or warning message can be displayed. In the example shown, the message can also include suggestions, such as retracting the endoscope or taking a fluoroscopy image. Figure 21An example of a warning message triggered when a high insertion force is detected is shown. In this example, when the insertion force is detected to exceed the force threshold, a warning message can be displayed on the GUI. The warning message can include information that the robot movement has stopped and a suggestion to the user, such as "Remove the endoscope and press Resume". In some cases, when the user clicks the "Resume" button, the system can automatically retract the IDM.

[0091] In some cases, a control signal can be generated to modify the operation of the system, robotic arm, IDM, and / or endoscope. For example, a control signal can be automatically generated based on the detection of buckling, prolapse, or high insertion force to adjust the movement of the endoscope, stop the movement of the endoscope, etc. For example, the control signal can automatically command the actuators of the IDM and / or robotic arm to move the tip of the endoscope to reduce buckling or insertion force. An algorithm based on sensor data (e.g., EM sensor data) can be used to automatically determine the speed, movement, and trajectory path of moving the tip of the endoscope.

[0092] Flexible endoscope system and device

[0093] In some embodiments, the endoscope tip speed control methods and systems herein can be used to improve the reliability and stability of flexible endoscopes. The provided endoscope tip speed control mechanism can be used by any device or apparatus. In one aspect of the invention, a flexible endoscope with improved performance (e.g., improved reliability) is provided.

[0094] Figure 9 Examples of robotic endoscope (e.g., bronchoscope) systems 900, 930 according to some embodiments of the present invention are shown. As Figure 9As shown, a robotic endoscope (e.g., a bronchoscope) system 900 may include a steerable catheter assembly 920 and a robotic support system 910 for supporting or carrying the steerable catheter assembly. The steerable catheter assembly may be a bronchoscope. The steerable catheter assembly may be the same as the above-described endoscope device. In some embodiments, the steerable catheter assembly may be a disposable robotic bronchoscope. In some embodiments, the robotic endoscope (e.g., a bronchoscope) system 900 may include an instrument drive mechanism (IDM) 913 attached to the arm of the robotic support system. The instrument drive mechanism may be provided by any suitable controller device (e.g., a hand-held controller), which may or may not include the robotic system. The instrument drive mechanism may provide a mechanical and electrical interface for the steerable catheter assembly 920. The mechanical interface may allow the steerable catheter assembly 920 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the steerable catheter assembly may be attached to the instrument drive mechanism via a quick-mount / release device (e.g., a magnet, a spring-loaded level, etc.). In some cases, the steerable catheter assembly may be manually coupled to or released from the instrument drive mechanism without using tools.

[0095] The steerable catheter assembly 920 may include a handle portion 923, which may include components configured to process image data, provide power, or establish communication with other external devices. For example, the handle portion 923 may include circuitry and communication elements that enable electrical communication between the steerable catheter assembly 920 and the instrument drive mechanism 913 and any other external systems or devices. In another example, the handle portion 223 may include circuit elements, such as a power source for powering the electronics of the endoscope (e.g., a camera and LED lights). In some cases, the handle portion may communicate electrically with the instrument drive mechanism 913 via an electrical interface (e.g., a printed circuit board) such that image / video data and / or sensor data may be received by the communication module of the instrument drive mechanism and may be transmitted to other external devices / systems. As an alternative or in addition, the instrument drive mechanism 913 may provide only a mechanical interface. The handle portion may communicate electrically with a modular wireless communication device or any other user device (e.g., a portable / hand-held device or controller) to transmit sensor data and / or receive control signals.

[0096] The steerable catheter assembly 920 may include a flexible elongate member 911 that is coupled to a handle portion. In some embodiments, the flexible elongate member may include a shaft, a steerable tip, and a steerable section. The steerable catheter assembly may be a disposable robotic bronchoscope. In some cases, only the elongate member may be disposable. In some cases, at least a portion of the elongate member (e.g., the shaft, the steerable tip, etc.) may be disposable. In some cases, the entire steerable catheter assembly 920, including the handle portion and the elongate member, may be disposable. The flexible elongate member and the handle portion are designed such that the entire steerable catheter assembly can be provided at low cost.

[0097] In some embodiments, the provided bronchoscope system may further include a user interface. As shown in the example system 930, the bronchoscope system may include a treatment interface module 931 (on the user console side) and / or a treatment control module 933 (on the patient and robotic side). The treatment interface module may allow an operator or user to interact with the bronchoscope during a procedure. In some embodiments, the treatment control module 933 may be a handheld controller. The treatment control module 933 may allow a user to control the speed of the bronchoscope tip, as described elsewhere herein. In some cases, the treatment control module may include a proprietary user input device and one or more additional elements that are detachably coupled to an existing user device to improve the user input experience. For example, a physical trackball or roller may replace or supplement the function of at least one of the virtual graphical elements (e.g., the navigation arrows shown on a touchpad) displayed on a graphical user interface (GUI) by imparting a function similar to that of the replaced graphical element. Examples of user devices may include, but are not limited to, mobile devices, smart phones / cellular phones, tablet computers, personal digital assistants (PDAs), laptop or notebook computers, desktop computers, media content players, and the like. Details regarding the user interface device and the user console will be described later herein.

[0098] Figure 10 An example of a flexible endoscope 1000 in accordance with some embodiments of the present disclosure is illustrated. As Figure 10As shown, the flexible endoscope 1000 may include a handle / proximal portion 1009 and a flexible elongate member to be inserted into an object. The flexible elongate member may be the same as the flexible elongate member described above. In some embodiments, the flexible elongate member may include a proximal shaft (e.g., insertion shaft 1001), a steerable tip (e.g., tip 1005), and a steerable section (active bending section 1003). The active bending section and the proximal shaft section may be the same as the active bending section, the anti-prolapse passive section, and the proximal shaft section described elsewhere herein. The endoscope 100 may also be referred to as a steerable catheter assembly, as described elsewhere herein. In some cases, the endoscope 100 may be a single-use robotic endoscope. In some cases, the entire catheter assembly may be disposable. In some cases, at least a portion of the catheter assembly may be disposable. In some cases, the entire endoscope may be released from the instrument drive mechanism and discarded. In some embodiments, the endoscope may include different levels of stiffness along its axis to improve functional operation.

[0099] The endoscope or steerable catheter assembly 1000 may include a handle portion 1009, which may include one or more components configured to process image data, provide power, or establish communication with other external devices. For example, the handle portion may include circuitry and communication elements that enable electrical communication between the steerable catheter assembly 1000 and an instrument drive mechanism (not shown) and any other external system or device. In another example, the handle portion 1009 may include circuit elements, such as a power source for powering the electronics of the endoscope (e.g., camera, electromagnetic sensor, and LED lights).

[0100] One or more components located at the handle may be optimized such that expensive and complex components may be allocated to the robotic support system, the hand-held controller, or the instrument drive mechanism, thereby reducing costs and simplifying the design of the disposable endoscope. The handle portion or proximal portion may provide electrical and mechanical interfaces to allow for electrical and mechanical communication with the instrument drive mechanism. The instrument drive mechanism may include a set of motors that are actuated to rotationally drive a set of wire ropes of the catheter. The handle portion of the catheter assembly may be fitted onto the instrument drive mechanism such that its pulley / winch assembly is driven by the set of motors. The number of pulleys may vary based on the wire rope configuration. In some cases, one, two, three, four, or more wire ropes may be used to articulate the flexible endoscope or catheter.

[0101] The handle portion can be designed to allow the robotic bronchoscope to be disposable at a reduced cost. For example, a classic manual bronchoscope and a robotic bronchoscope can have a cable at the proximal end of the bronchoscope handle. This cable typically includes illumination optical fibers, a camera video cable, and other sensor optical fibers or cables (such as electromagnetic (EM) sensors or shape-sensing optical fibers). Such a complex cable can be expensive, increasing the cost of the bronchoscope. The provided robotic bronchoscope can have an optimized design such that a simplified structure and components can be employed while retaining mechanical and electrical functions. In some cases, the handle portion of the robotic bronchoscope can employ a cableless design while providing a mechanical / electrical interface for the catheter.

[0102] The electrical interface (e.g., a printed circuit board) can allow image / video data and / or sensor data to be received by the communication module of the instrument drive mechanism and can be transmitted to other external devices / systems. In some cases, the electrical interface can establish electrical communication without cables or wires. For example, the interface can include pins soldered to an electronic board such as a printed circuit board (PCB). For example, a socket connector (e.g., a female connector) is provided on the instrument drive mechanism as a mating interface. This can beneficially allow the endoscope to be quickly inserted into the instrument drive mechanism or the robotic support without using additional cables. This type of electrical interface can also be used as a mechanical interface such that when the handle portion is inserted into the instrument drive mechanism, both a mechanical coupling and an electrical coupling are established. As an alternative or in addition, the instrument drive mechanism can provide only a mechanical interface. The handle portion can communicate electrically with a modular wireless communication device or any other user device (e.g., a portable / handheld device or a controller) for transmitting sensor data and / or receiving control signals.

[0103] In some cases, the handle portion 1009 can include one or more mechanical control modules, such as a Luer interface 1011, for connecting a lavage system / suction system. In some cases, the handle portion can include a lever / knob for articulation control. Alternatively, the articulation control can be located at a separate controller attached to the handle portion via the instrument drive mechanism.

[0104] An endoscope can be attached to a robotic support system or a handheld controller via an instrument drive mechanism. The instrument drive mechanism can be provided by any suitable controller device (e.g., a handheld controller), which may or may not include a robotic system. The instrument drive mechanism can provide a mechanical and electrical interface to the steerable catheter assembly 1000. The mechanical interface can allow the steerable catheter assembly 1000 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the steerable catheter assembly can be attached to the instrument drive mechanism via a quick-mount / detach tool (such as a magnet, a spring-loaded level, etc.). In some cases, the steerable catheter assembly can be manually coupled to or released from the instrument drive mechanism without using a tool.

[0105] In the illustrated example, the distal tip of the catheter or endoscope shaft is configured to articulate / bend in two or more degrees of freedom to provide a desired camera view or to control the orientation of the endoscope. As illustrated in this example, the imaging device (e.g., a camera), the position sensor (e.g., an electromagnetic sensor) 1007 is located at the tip of the catheter or endoscope shaft 1005. For example, the line of sight of the camera can be controlled by controlling the articulation of the active bending section 1003. In some cases, the angle of the camera can be adjustable such that the line of sight can be adjusted without articulating the distal tip of the catheter or endoscope shaft or in addition to articulating the distal tip of the catheter or endoscope shaft. For example, the camera can be oriented (e.g., tilted) at an angle relative to the axial direction of the endoscope tip by means of an optical component.

[0106] The distal tip 1005 can be a rigid component that allows sensors (such as electromagnetic (EM) sensors), imaging devices (e.g., cameras), and other electronic components (e.g., LED light sources) to be positioned and embedded at the distal tip.

[0107] In real-time EM tracking, an EM sensor consisting of one or more sensor coils located in one or more positions and orientations (e.g., the tip of an endoscopic tool) embedded in a medical device measures the changes in the EM field generated by one or more static EM field generators positioned at locations close to the patient. The position information detected by the EM sensor is stored as EM data. The EM field generator (or transmitter) can be placed close to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensors. This magnetic field induces small currents in the sensor coils of the EM sensor, and the small currents can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. For example, the EM field generator can be positioned close to the patient's torso during surgery to localize the position of the EM sensor in 3D space or to localize the position and orientation of the EM sensor in 5D or 6D space. This can provide visual guidance to the operator when driving a bronchoscope towards a target site.

[0108] The endoscope may have a unique design in the elongate member. In some cases, the active bending section 1003 and the proximal shaft of the endoscope may consist of a single tube that includes a series of incisions (e.g., notches, slits, etc.) along its length to allow for improved flexibility, desired stiffness, and anti-prolapse features (e.g., features for defining a minimum bending radius).

[0109] As described above, the active bending section 1003 can be designed to allow bending (e.g., articulation) in two or more degrees of freedom. A greater degree of bending, such as 180 degrees and 270 degrees (or other articulation parameters for clinical indications), can be achieved through the unique structure of the active bending section. In some cases, a variable minimum bending radius can be provided along the axial axis of the elongate member such that the active bending section or the passive section can include two or more different minimum bending radii.

[0110] The articulation of the endoscope can be controlled by applying a force via one or more pull wires to the distal end of the endoscope. One or more pull wires can be attached to the distal end of the endoscope. In the case of multiple pull wires, pulling one wire at a time can change the orientation of the distal tip to tilt it up, down, left, right, or in any desired direction. In some cases, the pull wires can be anchored at the distal tip of the endoscope, travel through the bending section, and enter the handle where the pull wires are coupled to a drive assembly (e.g., a pulley). This handle pulley can interact with the output shaft from a robotic system.

[0111] In some embodiments, the proximal or near-side portion of one or more pull wires can be operatively coupled to respective mechanisms (e.g., gears, pulleys, winches, etc.) in the handle portion of the catheter assembly. The pull wires can be metal wires, cables, or filaments, or they can be polymer wires, cables, or filaments. The pull wires can also be made of natural or organic materials or fibers. The pull wires can be any type of suitable wire, cable, or filament capable of supporting various loads without deforming, significantly deforming, or breaking. The distal / far-side portion of one or more pull wires can be anchored or integrated into the distal portion of the catheter such that the operation of the pull wires by the control unit can apply a force or tension to the distal portion that can manipulate or articulate (e.g., up, down, pitch, yaw, or in any direction therebetween) at least the distal portion (e.g., the flexible section) of the catheter.

[0112] The pull wire can be made of any suitable material (such as stainless steel (e.g., SS316), metal, alloy, polymer, nylon, or biocompatible material). The pull wire can be a wire, cable, or filament. In some embodiments, different pull wires can be made of different materials to vary the load-bearing capacity of the pull wire. In some embodiments, different segments of the pull wire can be made of different materials to vary the stiffness and / or load-bearing capacity along the pull wire. In some embodiments, the pull wire can be used for the transmission of electrical signals.

[0113] The proximal design can improve the reliability of the device without introducing additional costs, thereby allowing for low-cost single-use endoscopes. In another aspect of the present invention, a single-use robotic endoscope is provided. The robotic endoscope can be a bronchoscope and can be the same as the steerable catheter assembly described elsewhere herein. Conventional endoscopes can be complex in design and are typically designed for reuse after surgery, which requires thorough cleaning, disinfection, or sterilization after each surgery. Existing endoscopes are typically designed with complex structures to ensure that the endoscope can withstand the cleaning, disinfection, and sterilization processes. The provided robotic bronchoscope can be a single-use endoscope, which can beneficially reduce cross-contamination between patients and infected individuals. In some cases, the robotic bronchoscope can be delivered to a medical practitioner in a pre-sterilized package and is intended to be discarded after single use.

[0114] As Figure 11 shown, the robotic bronchoscope 1120 can include a handle portion 1113 and a flexible elongate member 1111. In some embodiments, the flexible elongate member 1111 can include a shaft, a steerable tip, and a steerable / actively bendable section. The robotic bronchoscope 1120 can be the same as the steerable catheter assembly described in Figure 10 . The robotic bronchoscope can be a single-use robotic endoscope. In some cases, only the catheter can be disposable. In some cases, at least a portion of the catheter can be disposable. In some cases, the entire robotic bronchoscope can be released from the instrument drive mechanism and discarded. In some cases, the bronchoscope can include different levels of stiffness along its axis to improve functional operation. In some cases, the minimum bending radius along the axis can vary.

[0115] A robotic bronchoscope can be releasably coupled to an instrument drive mechanism 1120. The instrument drive mechanism 1120 can be mounted to an arm of a robotic support system or can be mounted to any actuated support system as described elsewhere herein. The instrument drive mechanism can provide a mechanical and electrical interface to the robotic bronchoscope 1110. The mechanical interface can allow the robotic bronchoscope 1110 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the robotic bronchoscope can be attached to the instrument drive mechanism via a quick mount / dismount tool (such as a magnet and spring-loaded level). In some cases, the robotic bronchoscope can be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0116] Figure 12A An example of an instrument drive mechanism (IDM) 1220 is shown that provides a mechanical interface to the handle portion 1213 of a robotic bronchoscope. As shown in this example, the instrument drive mechanism 1220 can include a set of motors that are actuated to rotationally drive a set of wires of a flexible endoscope or catheter. The handle portion 1213 of the catheter assembly can be mounted onto the instrument drive mechanism so that its pulley assembly or winch is driven by the set of motors. The number of pulleys can vary based on the wire configuration. In some cases, one, two, three, four, or more wires can be used to articulate the flexible endoscope or catheter.

[0117] The handle portion can be designed to allow the robotic bronchoscope to be disposable at a reduced cost. For example, classic manual bronchoscopes and robotic bronchoscopes can have a cable at the proximal end of the bronchoscope handle. This cable typically includes illumination fibers, a camera video cable, and other sensor fibers or cables (such as electromagnetic (EM) sensors or shape-sensing fibers). Such complex cables can be expensive, increasing the cost of the bronchoscope. The provided robotic bronchoscope can have an optimized design such that a simplified structure and components can be employed while retaining mechanical and electrical functionality. In some cases, the handle portion of the robotic bronchoscope can employ a cableless design while providing a mechanical / electrical interface to the catheter.

[0118] Figure 12BAnother example of a disposable endoscope 1200 detachably coupled to an IDM 1201 is shown. One or more components located at the handle 1209 can be optimized such that expensive and complex components can be allocated to the robotic support system 1203, the handheld controller, or the instrument drive mechanism 1201, thereby reducing costs and simplifying the design of the disposable endoscope. The handle portion or proximal portion 1209 can provide electrical and mechanical interfaces to allow electrical and mechanical communication with the instrument drive mechanism 1201. The instrument drive mechanism 1201 can include a set of motors that are actuated to rotationally drive a set of wire ropes of the catheter. The handle portion 1209 of the catheter assembly can be mounted to the instrument drive mechanism 1201 such that its pulley / winch assembly is driven by the set of motors. For example, the handle pulley can interact with the output shaft 1203 of the IDM supported by the robotic system. The number of pulleys can vary based on the wire rope configuration. In some cases, one, two, three, four, or more wire ropes can be used to articulate the flexible endoscope or catheter.

[0119] Figure 13 An example of the distal tip 1300 of the endoscope is shown. In some cases, the distal portion or tip of the catheter 1300 can be substantially flexible such that it can be maneuvered in one or more directions (e.g., pitch, yaw). The catheter can include a tip portion, a bending section, and an insertion shaft. In some embodiments, the catheter can have a variable bending stiffness along the longitudinal axis. For example, the catheter can include multiple segments having different bending stiffnesses (e.g., flexible, semi-rigid, and rigid). The bending stiffness can vary by selecting materials with different stiffness / rigidity, changing the structure (e.g., cuts, patterns) in different sections, adding additional support components, or any combination of the above. In some embodiments, the catheter can have a variable minimum bending radius along the longitudinal axis. Selecting different minimum bending radii at different positions along the catheter can beneficially provide anti-prolapse capabilities while still allowing the catheter to reach difficult-to-reach areas. In some cases, the proximal end of the catheter does not require high bending, so the proximal portion of the catheter can be reinforced with additional mechanical structures (e.g., additional material layers) to achieve greater bending stiffness. This design can provide support and stability to the catheter. In some cases, the variable bending stiffness can be achieved by using different materials during the extrusion of the catheter. This can advantageously allow different stiffness levels along the axis of the catheter during the extrusion manufacturing process without additional fastening or assembly of different materials.

[0120] The distal portion of the catheter can be manipulated by one or more pull wires 1305. The distal portion of the catheter can be made of any suitable material (such as a copolymer, polymer, metal, or alloy) such that it can be bent by the pull wires. In some embodiments, the proximal end or terminus of one or more pull wires 1305 can be coupled to a drive mechanism (e.g., gears, pulleys, winches, etc.) via the anchoring mechanism described above.

[0121] The pull wires 1305 can be metal wires, cables, or filaments, or they can be polymer wires, cables, or filaments. The pull wires 1305 can also be made of natural or organic materials or fibers. The pull wires 1305 can be any type of suitable wire, cable, or filament capable of supporting various loads without deforming, significantly deforming, or breaking. The distal end or distal portion of one or more pull wires 1305 can be anchored or integrated into the distal portion of the catheter such that operation of the pull wires by the control unit can apply a force or tension to the distal portion that can manipulate or articulate (e.g., up, down, pitch, yaw, or in any direction therebetween) at least the distal portion (e.g., the flexible section) of the catheter.

[0122] The catheter can be sized such that one or more electronic components can be integrated into the catheter. For example, the outer diameter of the distal tip can be about 4 to 4.4 millimeters (mm), and the diameter of the working channel 1303 can be about 2 mm such that one or more electronic components can be embedded in the wall of the catheter. However, it should be noted that based on different applications, the outer diameter can be in any range less than 4 mm or greater than 4.4 mm, and the diameter of the working channel can be in any range depending on the tool size or specific application.

[0123] One or more electronic components can include an imaging device, a lighting device, or a sensor. In some embodiments, the imaging device can be a video camera 1313. The imaging device can include optical elements and an image sensor for capturing image data. The image sensor can be configured to generate image data in response to the wavelength of light. Various image sensors (such as complementary metal oxide semiconductor (CMOS) or charge-coupled device (CCD)) can be employed to capture image data. The imaging device can be a low-cost camera. In some cases, the image sensor can be provided on a circuit board. The circuit board can be an imaging printed circuit board (PCB). The PCB can include multiple electronic components for processing the image signal. For example, the circuit for a CCD sensor can include an A / D converter and an amplifier to amplify and convert the analog signal provided by the CCD sensor. Optionally, the image sensor can be integrated with an amplifier and a converter to convert the analog signal into a digital signal, such that a circuit board may not be required. In some cases, the output of the image sensor or the circuit board can be image data (a digital signal) that can be further processed by the camera circuit or the processor of the camera. In some cases, the image sensor can include an array of optical sensors.

[0124] The lighting device can include one or more light sources 1311 located at the distal tip. The light source can be a light-emitting diode (LED), an organic LED (OLED), a quantum dot, or any other suitable light source. In some cases, the light source can be a small LED or a dual-tone flash LED lighting for a compact design.

[0125] The imaging device and the lighting device can be integrated into the catheter. For example, the distal portion of the catheter can include a suitable structure that matches at least one dimension of the imaging device and the lighting device. The imaging device and the lighting device can be embedded in the catheter. Figure 14 An example distal portion of a catheter with an integrated imaging device and lighting device is shown. The camera can be located at the distal portion. The distal tip can have a structure for accommodating the camera, the lighting device, and / or a position sensor. For example, the camera can be embedded in a cavity 1410 at the distal tip of the catheter. The cavity 1410 can be integrally formed with the distal portion of the cavity and can have dimensions that match the length / width of the camera, such that the camera can not move relative to the catheter. The camera can be adjacent to the working channel 1420 of the catheter to provide a near-field view of the tissue or organ. In some cases, the attitude or orientation of the imaging device can be controlled by controlling the rotational movement (e.g., rolling) of the catheter.

[0126] The power for the camera can be provided through a wired cable. In some cases, the cable can be in a harness that provides power to the camera and to the illumination element or other circuitry at the distal tip of the catheter. The camera and / or light source can be powered by a power source located at the handle portion via a wire, copper wire, or any other suitable means traveling through the length of the catheter. In some cases, real-time images or videos of the tissue or organ can be transmitted wirelessly to an external user interface or display. The wireless communication can be WiFi, Bluetooth, RF communication, or other forms of communication. In some cases, the images or videos captured by the camera can be broadcast to multiple devices or systems. In some cases, the image and / or video data from the camera can be transmitted along the length of the catheter via a wire, copper wire, or any other suitable means to a processor located in the handle portion. The image or video data can be transmitted to an external device / system via a wireless communication component in the handle portion. In some cases, the system can be designed such that the wire is not visible to the operator or the wire is not exposed to the operator.

[0127] In conventional endoscopy, the illumination light can be provided by an optical fiber cable that transmits the light from a light source located at the proximal end of the endoscope to the distal end of the robotic endoscope. In some embodiments of the present disclosure, small LED lights can be employed and embedded in the distal portion of the catheter to reduce design complexity. In some cases, the distal portion can include structure 1430 that has dimensions matching the size of the small LED light source. As shown in the illustrated example, two cavities 1430 can be formed integrally with the catheter to accommodate two LED light sources. For example, the outer diameter of the distal tip can be about 4 to 4.4 millimeters (mm), and the diameter of the working channel of the catheter can be about 2 mm, such that the two LED light sources can be embedded at the distal end. The outer diameter can be in any range less than 4 mm or greater than 4.4 mm, and the diameter of the working channel can be in any range depending on the size of the tool or the specific application. Any number of light sources can be included. The internal structure of the distal portion can be designed to accommodate any number of light sources.

[0128] In some cases, each LED can be connected to a power cord that can extend to the proximal handle. In some embodiments, the LED can be soldered to separate power cords that are subsequently bundled together to form a single strand. In some embodiments, the LED can be soldered to a pull cord that supplies power. In other embodiments, the LED can be crimped or directly connected to a single pair of power cords. In some cases, a protective layer such as a thin layer of biocompatible glue can be applied to the front surface of the LED to provide protection while allowing light to be emitted. In some cases, an additional cover 1431 can be placed at the forward end face of the distal tip to provide precise positioning of the LED and sufficient space for glue. The cover 1431 can be composed of a transparent material that matches the refractive index of the glue so that the illumination light can be unblocked.

[0129] It should be noted that the illustrated distal end design is for illustration purposes only. There may be other suitable designs for integrating one or more components into the distal tip. Figure 15 Another example of a distal portion 1500 of a catheter with an integrated imaging device and an illumination device is shown. As shown in example 1500, the distal tip may have a structure 1501 for accommodating a camera, a structure 1503 for accommodating an illumination device and / or a position sensor. The camera may be embedded in a cavity 1501 at the distal tip of the catheter. Cavity 1501 may be integrally formed with the distal portion of the cavity, and may have a size that matches the length / width of the camera so that the camera cannot move relative to the catheter. The camera may be adjacent to the working channel 1507 of the catheter to provide a near-field view of a tissue or organ. In some cases, the attitude or direction of the imaging device may be controlled by controlling the rotational movement (e.g., rolling) of the catheter. As shown in example 1500, cavity 1503 may be integrally formed with the catheter to accommodate an LED light source.

[0130] In one aspect, a system for controlling the tip motion of an articulated flexible endoscope is provided. The system includes: a memory storing computer executable instructions; one or more processors communicating with the articulated flexible endoscope and configured to execute the computer executable instructions to: generate a command to drive an elongated member of the articulated flexible endoscope along an anatomical path via an instrument drive mechanism (IDM); receive sensor data acquired by a position sensor disposed at a distal tip portion of the elongated member; after determining that the distal tip portion is located at a preselected position within the anatomical path, set the motion of the distal tip to zero and calculate the motion of the distal tip portion within a time window; calculate the difference between the motion of the distal tip portion and the motion of the IDM within the same time window; and detect a buckling event by comparing the difference with a threshold.

[0131] As used herein, a processor includes one or more processors, such as a single processor, or multiple processors such as a distributed processing system. The controller or processor described herein typically includes a tangible medium for storing instructions for implementing process steps, and the processor may include, for example, one or more of a central processing unit, programmable array logic, gate array logic, or a field programmable gate array. In some cases, one or more processors may be programmable processors (e.g., a central processing unit (CPU) or a microcontroller), digital signal processors (DSPs), field programmable gate arrays (FPGAs), and / or one or more advanced RISC machines (ARM) processors. In some cases, one or more processors may be operably coupled to a non-transitory computer-readable medium. The non-transitory computer-readable medium may store logic, code, and / or program instructions executable by one or more processor units to perform one or more steps. The non-transitory computer-readable medium may include one or more memory units (e.g., removable media or external memory, such as an SD card or random access memory (RAM)). One or more methods or operations disclosed herein may be implemented in hardware components or a combination of hardware and software, such as an ASIC, a dedicated computer, or a general-purpose computer.

[0132] One or more processors may communicate with the endoscope. One or more processors may be located remote from the endoscope system or on the endoscope system (e.g., at the handle, at a user device for controlling the endoscope).

[0133] Although the preferred embodiments of the present invention have been shown and described herein, it will be readily apparent to those skilled in the art that such embodiments are provided by way of example only. Many variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. The appended claims are intended to define the scope of the invention and thereby cover the methods and structures within the scope of these claims and their equivalents.

Claims

1. A system for controlling the tip movement of an articulated flexible endoscope, the system comprising: a memory storing computer executable instructions; one or more processors in communication with the articulated flexible endoscope and configured to execute the computer executable instructions to: generating commands to drive an elongated member of the articulated flexible endoscope along an anatomical path via an instrument drive mechanism (IDM); receiving sensor data acquired by a position sensor disposed at a distal tip portion of the elongated member; upon determining that the distal tip portion is at a preselected position within the anatomical path, setting the motion of the distal tip portion to zero and calculating the motion of the distal tip portion within a time window; calculating a difference between the motion of the distal tip portion projected in the advancement direction and the motion of the IDM along the insertion direction within the same time window; determining a dynamic threshold along the insertion direction, wherein the dynamic threshold varies based at least in part on an insertion distance; and Buckling events are detected by comparing the difference to the dynamic threshold.

2. The system of claim 1, wherein the preselected location comprises a primary carina location.

3. A system according to claim 1, wherein the one or more processors are further configured to determine that the distal tip portion is located at the preselected position based at least in part on the sensor data and the 3D model of the anatomical path.

4. The system of claim 1 , wherein the one or more processors are further configured to determine that the distal tip portion is located at the preselected position based at least in part on image data acquired by a camera located at the distal tip portion. The system according to claim 1 , wherein the size of the time window is between 4 seconds and 8 seconds. The system of claim 5 , wherein the size of the time window is determined based on empirical data.

7. The system of claim 1, wherein the movement of the distal tip portion within the time window is an accumulation of distance traveled at each time step.

8. The system of claim 1 , wherein the one or more processors are further configured to set the insertion force applied by the IDM to zero upon determining that the distal tip portion is at the preselected position within the anatomical path and compare the insertion force to a force threshold. 9 . The system of claim 8 , wherein the one or more processors are further configured to generate and display a warning message on a user interface when it is determined that the insertion force is above the force threshold.

10. The system of claim 1, wherein the difference in motion comprises a difference between an expected velocity and a tip velocity of the distal tip portion measured based on the sensor data.

11. The system of claim 10, wherein the tip velocity is calculated as a filtered time derivative of the sensor data projected in the forward direction.

12. The system of claim 11, wherein the tip velocity is processed by a low pass filter to be used as a feedback signal for closed loop control.

13. The system according to claim 11, wherein the sensor data is acquired in a frequency range of 10 - 60 Hz.

14. The system according to claim 10, wherein the expected speed is based on an input command.

15. The system according to claim 1, wherein the distal tip portion includes a structure for accommodating an imaging device, the position sensor, and an illumination device.

16. The system according to claim 1, wherein a proximal end of the elongate member of the articulated flexible endoscope is connected to an IDM for applying a force to one or more pull wires to articulate the distal tip portion of the elongate member, insert, or retract the articulated flexible endoscope.

17. The system according to claim 1, wherein the one or more processors are further configured to display, on a user interface, a message indicating the buckling event and a suggestion for taking an action in response to the buckling event.

18. The system according to claim 17, wherein the action in response to the buckling event includes taking a fluoroscopic image of at least a portion of the articulated flexible endoscope.

Images