Steerable endoscope with motion alignment

Abstract

Steerable endoscopes are provided with active steering control. The endoscope includes a flexible tubular body having a first articulating segment and a second articulating segment and a camera. In embodiments, the endoscope includes a direction sensor. A controller for the endoscope automatically analyzes the alignment between the motion axis of the endoscope and the viewing axis of the camera and actively steers the endoscope to improve the alignment.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Application No. 62 / 888,906, filed August 19, 2019, and U.S. Provisional Application No. 63 / 012,741, filed April 20, 2020, the disclosure of which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] This disclosure relates generally to medical devices, and more specifically to steerable endoscopes with active motion alignment, and related methods and systems. Background Technology

[0004] Medical endoscopes are long, flexible instruments that can be introduced into a patient's cavity during medical procedures in a variety of situations to facilitate visualization and / or medical procedures within the cavity. For example, one type of endoscope is one with a camera at its distal end. Endoscopes can be inserted into a patient's mouth, throat, or other cavities to help visualize anatomical structures or facilitate procedures such as biopsies or ablation. Endoscopes may include a steerable distal tip that can be actively controlled to bend or twist in a desired direction to obtain a desired view or navigate through anatomical structures. However, these steerable endoscopes can be difficult to manipulate to the desired location and orientation within the patient's anatomy. Summary of the Invention

[0005] The following summarizes certain embodiments that are proportionate to the originally claimed subject matter. These embodiments are not intended to limit the scope of this disclosure. In fact, this disclosure may include various forms that may be similar to or different from the embodiments set forth below.

[0006] In one embodiment, the computer-controlled endoscope system includes an endoscope and a controller. The endoscope has a flexible tubular body with a first hinged section at its distal end and a second hinged section connected to the proximal end of the first hinged section. The first hinged section includes a camera having a field of view along a camera axis and a direction sensor sensitive to movement along a motion axis. The controller communicates with the endoscope and has a hardware memory storing instructions for analyzing alignment between the motion axis and the camera axis. During endoscope movement, the controller steers the first and second hinged sections of the endoscope to improve alignment.

[0007] In one embodiment, a method for computer-assisted endoscope steering includes receiving user input via a touchscreen display to move the observation axis of the endoscope. The endoscope has a first independent hinge segment and a second independent hinge segment, a camera having a field of view along the observation axis, and a orientation sensor. In response to the user input, the method includes hinged the first hinge segment of the endoscope to move the observation axis. The method also includes receiving a motion signal from the orientation sensor indicative of movement of the endoscope along a motion axis, comparing the motion axis with the observation axis at a processing chip, and generating a steering signal that controls the hinge of the first and second hinge segments to reduce the difference between the motion axis and the observation axis.

[0008] In one embodiment, a computer-implemented method for automatic endoscope steering includes receiving user input via a graphical user interface, specifying the direction of an observation axis that includes the movement of the endoscope. The endoscope has a first independent hinge segment and a second independent hinge segment, a camera having a field of view along the observation axis, and a direction sensor. The method includes generating a first steering signal having instructions for bending the first hinge segment of the endoscope in the direction indicated by the user input. The method also includes receiving a motion signal from the direction sensor indicating forward movement of the endoscope, and, in the absence of steering input from the user, generating a second steering signal having instructions for bending the second hinge segment during forward movement of the endoscope.

[0009] In one embodiment, the computer-controlled endoscope system includes an endoscope comprising a flexible tubular body having a first hinged segment at a distal end of the body and a second hinged segment proximal to the first hinged segment, wherein the first hinged segment includes a camera and a orientation sensor. The system also includes a controller in communication with the endoscope, which receives user steering input and motion signals from the orientation sensor. The controller includes a steering controller that controls independent hinges of the first and second hinged segments to hinge the first hinged segment, thereby presenting the orientation of the camera axis of the camera according to the user steering input, and maintaining the camera axis in that orientation by hinged first and second hinged segments during forward movement of the endoscope.

[0010] A feature in one aspect or embodiment can be applied as a feature in any other aspect or embodiment in any suitable combination. For example, any one of a system, laryngoscope, handle, controller, endoscope, or method feature can be applied as any one or more of a system, laryngoscope, controller, endoscope, or method feature. Attached Figure Description

[0011] The advantages of the disclosed technology will become apparent from the following detailed description and with reference to the accompanying drawings, in which:

[0012] Figure 1AThis is a cross-sectional view of an endoscope that moves distally through the patient's cavity.

[0013] Figure 1B This is a cross-sectional view of an endoscope that moves distally through the patient's cavity.

[0014] Figure 2A This is a cross-sectional view of an articulated endoscope with active motion alignment according to an embodiment of the present disclosure.

[0015] Figure 2B This is a cross-sectional view of an articulated endoscope with active motion alignment according to an embodiment of the present disclosure.

[0016] Figure 2C This is a cross-sectional view of an articulated endoscope with active motion alignment according to an embodiment of the present disclosure.

[0017] Figure 2D This is a cross-sectional view of an articulated endoscope with active motion alignment according to an embodiment of the present disclosure.

[0018] Figure 3 This is a cross-sectional view of two articulated endoscopes moved distally through patient tissue according to an embodiment of the present disclosure to illustrate motion alignment and steering.

[0019] Figure 4 This is a front view of a graphical user interface according to an embodiment of the present disclosure.

[0020] Figure 5A This is a schematic diagram of an optical flow technique for motion alignment according to an embodiment of the present disclosure.

[0021] Figure 5B This is a schematic diagram of an optical flow technique for motion alignment according to an embodiment of the present disclosure.

[0022] Figure 5C This is a schematic diagram of an optical flow technique for motion alignment according to an embodiment of the present disclosure.

[0023] Figure 6 This is a perspective view of the controller and endoscope according to an embodiment of the present disclosure.

[0024] Figure 7 This is a block diagram of a controller and endoscope according to embodiments of the present disclosure.

[0025] Figure 8 This is a flowchart depicting a method for computer-assisted endoscope steering according to embodiments of the present disclosure.

[0026] Figure 9 This is a flowchart depicting a method for computer-assisted endoscope steering according to embodiments of the present disclosure. Detailed Implementation

[0027] The medical endoscopes or endoscopes described herein are thin, elongated, flexible instruments that can be inserted into body cavities for exploration, imaging, biopsy, or other clinical treatments, including catheters, narrow tubular instruments, or other types of endoscopes or probes. Endoscopes can be navigated into body cavities (such as a patient's airway, gastrointestinal tract, oral cavity, or nasal cavity, or other cavities or openings) and guided by the user by advancing the distal end to the desired position and, in some embodiments, by biomimetic movement of the endoscope. The endoscope may be tubular in shape.

[0028] Long, flexible medical devices are typically advanced into a patient's cavity by forces transmitted from the proximal portion of the device (outside the patient's cavity), causing the distal tip to advance within the cavity. For example, a physician or other caregiver holding the proximal portion of the medical device (such as the handle) outside the patient's cavity pushes it down or forward, and the resulting motion is transmitted to the distal tip, causing the tip to move forward within the cavity. Similarly, a pulling force applied by a caregiver to the proximal portion can cause the distal tip to retract or move in the opposite direction away from the patient's cavity. However, because patient cavities are not regularly shaped or of a specific size, the endoscope moves along a tortuous path, and forces transmitted from the proximal end in a pushing or pulling motion may not result in predictable movement at the distal tip.

[0029] Figures 1A to 1B An example of unintended movement is shown, illustrating a cross-sectional view of an endoscope 10 moved distally through a patient cavity. In this example, the patient cavity is the nasal cavity and oral cavity, leading to the esophagus 12 and trachea 14, and the operator intends to guide the endoscope 10 into the trachea 14. Also labeled for reference are the patient's tongue 16, jaw 18, vocal cords 28, and palate 26. Figure 1A During the procedure, the operator moves the endoscope 10 forward through the nasal cavity to the area behind the epiglottis 22 and groove 24. At this point, the operator stops and bends the distal tip 29 of the endoscope upward toward the trachea 14 (in the figure), as indicated by the dashed line of the distal tip of the endoscope 12. The distal tip 29 can be hinged in this upward bend to allow the operator to control the view of the camera at the distal tip 29 and to view around the patient cavity.

[0030] The trachea is above the esophagus (forward, towards the patient's chest), and therefore the endoscope must be navigated in a forward direction to avoid entering the esophagus. Figure 1BThe operator pushes endoscope 10 forward in a distal direction, as indicated by arrow D. However, this forward movement of the endoscope moves it into the esophagus 12, not the trachea 14. This movement of the endoscope is undesirable if the operator intends to intubate the patient (insert an endotracheal tube into the patient's trachea). In fact, if the operator is unaware that the endoscope has moved into the esophagus instead of the trachea, the operator may inadvertently perform esophageal intubation (insert an endotracheal tube into the esophagus instead of the trachea), which could create a medical emergency for the patient (because breathing air is subsequently delivered to the gastrointestinal system instead of the lungs).

[0031] Figures 1A to 1B This demonstrates that thrust at the proximal end of the endoscope (outside the patient) may be insufficient to cause the distal tip 29 to turn in the desired direction within the patient's cavity. Smoothly navigating the length of the endoscope 10 through curved or irregular sections of the patient's cavity can be particularly challenging.

[0032] This document provides an articulated endoscope with computer-controlled or automatically controlled steering, which aligns the movement of the endoscope with its viewing direction. This alignment can be performed to correct, improve, or enhance a user-provided steering input that provides coarse guidance regarding a desired position at the distal end. According to an embodiment, the endoscope system includes an endoscope having a flexible tubular body comprising a first articulated segment and a second articulated segment at its distal end. The first articulated segment includes a camera having a field of view along a camera axis and a direction sensor sensitive to movement along a motion axis. The system also includes a controller in communication with the endoscope, which performs automatic analysis of the alignment between the motion axis and the camera axis. The controller actively steers the first and second segments of the endoscope to improve alignment. Although embodiments are disclosed in the context of the first and second articulated segments, it should be understood that the endoscope system may include an endoscope having additional articulated segments (e.g., third, fourth) as provided herein.

[0033] Figures 2A to 2D A cross-sectional view of an endoscope 220 positioned within and moving through a patient cavity according to an embodiment of the present disclosure is shown. The endoscope 220 includes a camera 230 at its distal tip 229. The depicted motion patterns include hinges to different (e.g., first, second) actuable portions of the endoscope 220 to produce desired movement through the patient's trachea. The nasal cavity and oral cavity, as well as the trachea, are shown by way of example, and in other embodiments, the endoscope may enter other patient cavities and traverse other variations of the anatomical structure.

[0034] Figures 2A to 2DA rectangular cross-sectional view of an endoscope 220 moving through a patient's anatomy is shown, along with a circular field of view 230V showing the view from a camera 230 of the endoscope 220. The endoscope 220 includes two steerable segments 232, 234 in a distal region of the endoscope 220. The two steerable segments are connected to each other, with the first segment 232 distal to the second segment 234. Each segment 232, 234 can be hinged independently of the other segments. In embodiments, segments 232, 234 can be directly adjacent to each other or can be separated by a central connecting portion of the endoscope 220. In embodiments, each segment 232, 234 can be bent and flexed in three dimensions (not just in a single plane, such as up / down or right / left), bending in all directions until the limits of its range of motion are reached. For example, in an embodiment, each segment can be bent up to 90 degrees in any direction, enabling it to move within a hemisphere with a radius equal to the length of the segment. Each segment is operated by an actuation system comprising one or more actuators (such as a sleeve-type drawwire or other actuators described below) that move to bend the segment into a curved shape or straighten the segment out of a curved shape. Each segment 232, 234 may be controlled by a central actuation system that controls all articulated segments, or may be coupled to a dedicated actuation system for each articulated segment.

[0035] exist Figure 2A In this configuration, the two steerable segments 232 and 234 are in their stationary default positions, where they do not actively bend. The endoscope 220 has been inserted through the patient's nasal cavity into the pharynx and is pointed towards the patient's esophagus 12. The camera 230 is pointed along the camera axis CA, as indicated by the dashed line. In this configuration, segments 232 and 234 are straight, and axis CA points towards the patient's esophagus 12. The camera's view 230V shows a view of the esophagus 12, with the vocal cords 28 visible at the top of the view.

[0036] Still referencing Figure 2A Caregivers can provide input (e.g., user input) to orient the camera upward toward the vocal cords 28. For example, a caregiver can tap the vocal cords 28 on the field of view 230V displayed on the touchscreen to command the endoscope to bend upward toward that view. The user's touch input indicates the direction CA2 as where the user wants the camera to be pointed. In response, the endoscope bends upward at the first segment 230, as... Figure 2A The second dashed line CA2 is shown in the diagram.

[0037] After the turn, the endoscope now bends along segment 232 to point along axis CA2, as... Figure 2BAs shown. At this point, if the endoscope moves forward in the distal direction without any further steering, it may impinge on the tracheal wall 29, or pass beneath the tracheal wall and enter the esophagus 12. Therefore, in this embodiment, the endoscope 220 actively steers itself to align its movement with the camera axis. This automatic motion alignment steering... Figure 2B As shown in D. In Figure 2B In this configuration, the user pushes the endoscope 220 distally, and the endoscope is computer-controlled to automatically hinge the second segment 234, thereby aligning the movement of the endoscope with the observation axis of the camera. Therefore, when the user... Figure 2B As the endoscope is pushed forward (in the distal direction), it detects the movement and hinges the second segment 234 to compensate. The hinge of the second segment 234 can occur during the forward movement, allowing active steering to occur as the endoscope 220 moves. The second segment 234 bends to align itself with the camera axis CA. This active steering causes the distal end of the endoscope 220 to bend upward toward the trachea 14, with the camera axis CA pointing toward the trachea. Figure 2B In the middle, the field of view 230V now points above the vocal cords 28, and segments 232 and 234 both bend upward (in the forward direction, toward the patient's chest).

[0038] At this point, the user can turn the camera backward and downward to point the camera's view 230V towards the vocal cords and into the trachea, such as... Figure 2C As shown. For example, a user can tap the vocal cords 28 on the touchscreen display, and the endoscope responds by bending the first segment 232 downwards to point towards the vocal cords 28. At this time, the first segment 232 bends downwards (in a rearward direction, toward the patient's back), while the second segment 234 remains bent upwards (forwards), as... Figure 2C As shown.

[0039] From here, if the user pushes the endoscope 220 further forward into the patient's body (in the distal direction), the endoscope 220 will again actively rotate itself to align its movement with the camera's axis CA, as... Figure 2D As shown. In Figure 2D In this process, the user has pushed the endoscope 220 forward through the vocal cords 28. The endoscope 220 detects the forward movement and bends a second segment 234 in the forward direction to align the movement with the camera's line of sight CA. At this point, both segments 232 and 234 bend in the backward direction, and the field of view 230V now observes the tracheal wall passing through the vocal cords 28.

[0040] In one embodiment, automatic motion alignment steering is applied to the first segment 232 and the second segment 234. In this case, the system allows the user to steer the first segment 232 (to point towards the camera axis CA) when the endoscope 220 is stationary or not moving, and to automatically steer both segments as the endoscope moves. In another embodiment, even during movement, automatic motion alignment steering allows the user to provide input to steer the first segment 232, and the system interprets the user input and the motion signal to steer the first segment 232. That is, the system allows the distal tip 29 to be steered via the hinge of the first segment 232 and / or the second segment 234 during translation of the endoscope 220. In one embodiment, the user steering input is used only to directly steer the first segment 232, while automatic or active steering is used to control both segments 232 and 234. That is, the user steering input results in a direct movement of the first segment 232 to reorient the camera 230. When the camera 230 is in the desired orientation, automatic steering controls the hinge of segments 232 and 234 to maintain the camera field of view 230V along the camera axis CA during movement.

[0041] Figure 3 A schematic diagram of two different endoscopes moving through internal passages within a patient's body is shown. In row 300, endoscope 10 includes only one steerable segment at its distal end. The user can instruct the endoscope to steer this distal segment to point the endoscope and its camera where the user wants it to go (configuration 300A), but the endoscope cannot actively align its movement with the camera's view by steerable a single steerable segment at its distal end. Therefore, as the user pushes endoscope 10 forward (distally), the endoscope pushes and rubs along the patient's tissues (structures 300B and 300C) as it moves forward. For example, as the endoscope moves through the bronchial passage in the lungs, the anterior guide edge of endoscope 10 rubs against the bronchial wall as it moves distally. This direct contact can irritate the tissue and blur the camera's view (by pointing it at the tissue or covering it with secretions). Here, the combination of single-segment steerable and translational movements of endoscope 10 provides undesirable positioning of endoscope 10.

[0042] Conversely, in line 302, the endoscope 20 according to an embodiment of the present disclosure includes two independently steerable segments at its distal end. In this embodiment, the endoscope 20 is computer-controlled to actively steer the two segments, thereby aligning the distal movement of the endoscope with the observation axis of the camera. Therefore, the endoscope 20 bends away from the tissue wall, reducing contact between the patient's tissue and the endoscope's insertion edge.

[0043] In this embodiment, during the endoscope's forward (distal) movement, rather than during its backward (proximal) movement, the endoscope actively steers both distal articulated segments to align their axes of motion with its camera axis. During the backward (proximal) movement, the user can steer the first (farthest) articulated segment to control the camera's view, but the second articulated segment (proximal to the first articulated segment) remains passive (non-actively articulated).

[0044] Figure 3 Line 302 also shows an enlarged cross-sectional view of the distal end of endoscope 20 to illustrate the arrangement of camera 30 and orientation sensor 56. This example shows camera 30 positioned at the distal end of endoscope 20 to obtain a clear forward view. Orientation sensor 56 is located directly behind camera 30. In an embodiment, orientation sensor 56 is adjacent to camera 30. In an embodiment, orientation sensor 56 is mounted on a printed circuit assembly (e.g., flexible circuit) behind camera 30. In an embodiment, orientation sensor 56 is mounted on the same printed circuit assembly as camera 30, although the orientation sensor and camera do not need to communicate on a shared printed circuit assembly. In an embodiment, orientation sensor has a size between 1-2 mm in each dimension.

[0045] Orientation sensor 56 is an electronic component that senses the orientation (such as orientation relative to gravity) and / or movement (acceleration) of the distal endoscope. Orientation sensor 56 generates motion signals indicating orientation and / or movement. Orientation sensor 56 is composed of sensors or combinations of sensors to achieve this, such as accelerometers, magnetometers, and gyroscopes. Orientation sensor 56 may be an inertial measurement unit (IMU) or a magnetic, angular rate, and gravity (MARG) sensor that allows yaw measurements. Orientation sensor 56 detects the static orientation and dynamic movement of the distal tip of the endoscope and provides signals indicating changes in endoscope orientation and / or endoscope movement. Orientation sensor 56 sends this signal to a controller. Orientation sensor 56 is located within the tubular housing of endoscope 20. Figure 3 As shown, in one embodiment, the orientation sensor 56 is located very close to the distal end of the endoscope 20, such as behind the camera 30, so that the orientation sensor 56 can capture most of the distal tip and the entire range of motion of the camera 30. In another embodiment, the orientation sensor 56 is arranged at the distal end of the first steerable portion, away from the proximal end of the steerable portion, to position the orientation sensor away from the pivot point of motion.

[0046] Figure 3Line 302 also illustrates how the hinge of the first and second segments aligns the motion axis with the camera axis. In configuration 302A, endoscope 20 is pointed toward the channel, as indicated by the camera axis CA. The portion of endoscope 20 shown does not contact the sidewalls of the patient cavity or channel. If the user pushes forward to advance endoscope 20, endoscope 20 will move forward along the motion axis MA, which deviates from the camera axis CA. The controller detects this offset and responds by bending one or more distal segments to compensate. For example, in configuration 302B, endoscope 20 actively bends the second hinge segment to reduce the offset between CA and MA. By bending, the segment converts motion along MA into motion along CA. As the user continues to advance endoscope 20, it eventually contacts the patient tissue, as shown in configuration 302C. The point of contact with the tissue causes endoscope 20 to deflect in the desired direction so that any further push from the user will move the endoscope forward along the camera axis CA. Therefore, the rest of endoscope 20 does not need to be actively steerable. The rest of the mirror (the proximal side of the two hinged sections) should be flexible so that it can passively follow the hinged sections, bending to turn along the path through which the channel passes, following a tortuous path.

[0047] The hinge of the first and second hinge segments can be performed in parallel (i.e., simultaneously) or in a continuous or alternating (e.g., rapidly alternating) manner. In this example, the hinges are alternately driven by rapidly and continuously driving one motor at a time. Furthermore, the hinges of the first and second hinge segments can be in opposite directions, such that one segment rotates in the opposite direction to the other.

[0048] Figure 3 Line 302 also illustrates a shallow contact angle between the second steerable segment and the patient's tissue. In an embodiment, the second segment is long enough to lift the first segment away from the tissue wall while maintaining a shallow contact angle between the second segment and the tissue. In an embodiment, this contact angle is about 40 degrees or less. Therefore, the second segment is long enough to advance the first segment away from the patient's tissue without the second segment having to bend more than about 40 degrees. In an embodiment, the angle is about 50 degrees or less, or about 30 degrees or less. The shallow angle also helps protect the patient's tissue by reducing irritation by creating a smooth curve rather than a sharper curve. The shallow angle also allows the endoscope to slide across the tissue with less force from the user. In an embodiment, the second segment is longer than the first segment. In an embodiment, the first segment has a length of about 35 mm, and the second segment has a length of about 50 mm. In an embodiment, the first segment has a length of about 20-40 mm, and the second segment has a longer length in the range of about 30-50 mm.

[0049] In one embodiment, the endoscope uses signals from orientation sensor 56 to identify the direction of gravity (downward) and then bends upward in the opposite direction (opposite gravity) to elevate the first segment and the camera over the patient's tissue. The direction of gravity can also be used as input to determine a specific portion of the patient's tissue. If the endoscope pushes against the tissue, the location of the push point or fulcrum in absolute space can be identified. Position information can be used to scale the sensitivity to user input. The further into the airway, the smaller the structure appears. If the relative position of nearby structures is being inferred, this can help to scale down, so similar input gestures produce similar movements in video transmission when moving along them. Similarly, if all reference points are far away, a more exaggerated articulation is generated from relatively similar inputs.

[0050] Figure 4 This is a front view of a graphical user interface (GUI) 400 according to an embodiment of the present disclosure. The GUI 400 is displayed on the display screen 412 of the controller 410. Figure 4 The controller is a handheld stick 416. In this embodiment, the display screen 412 includes a touchscreen 414. The GUI 400 receives user input by detecting the user's touch on the screen 414. The user touches the screen to indicate where the user wants to point a camera (such as camera 230 of the endoscope 220). The GUI 400 sends the touch input to a processor (described more fully below), which generates an instruction to bend a first distal segment (such as segment 232) to point the camera axis in the direction the user touched. In this particular instance, the user can hold the stick 416 with his or her left hand and touch the touchscreen 414 with his or her left thumb, freeing his or her right hand to hold and advance the endoscope. The user can turn the endoscope camera (such as camera 230) by tapping the screen 414 with his or her thumb. Figure 4 (as shown), and then the endoscope 420 can be advanced by pushing it forward with his or her right hand (or removed by pulling it back).

[0051] The controller 410 is shown as a rod 416, and the endoscope 420 is removably and directly connected to the rod 416 for transmitting control signals from the rod to the endoscope and video signals from the endoscope to the rod. In other embodiments, the controller 410 may have other forms or structures. For example, the controller 410 may be a video laryngoscope, a desktop display, a tablet computer, a laptop computer, a handheld cursor, or other form factor.

[0052] In an embodiment, GUI 400 includes a touchscreen responsive to tapping, touching, or proximity gestures from a user. For example, a user can input touch gestures (such as tapping, double-tapping, tapping and holding, swiping, highlighting, or swiping) to identify a target point or orientation within an image on the screen. This gesture identifies the location where the user intends to orient the endoscope, and the controller translates this into a real-world steering direction and a corresponding instruction for operating the steering system to move the distal steerable segment of the endoscope in that direction. The user can swipe on touchscreen 414 in the desired direction to reorient the distal end of the endoscope. The desired orientation or movement of the camera can be interpreted from the direction and length of the swipe movement on touchscreen 414. In an embodiment, steering input can be additionally or alternatively provided by the user selecting from a menu, selecting soft keys, pressing buttons, operating a joystick, etc. In an embodiment, the user can circle or otherwise highlight the portion of the image in which the distal end should be oriented.

[0053] The controller 410, which includes the endoscope 420, operates as a two-part endoscope, serving as a handle, display, and user input for the endoscope 420. In this embodiment, the controller 410 is reusable, and the endoscope 420 is for individual use and single-use to prevent cross-contamination between patients or caregivers. The controller 410 itself does not require contact with the patient, and it can be wiped and cleaned, ready for use with a new sterile endoscope 420 for the next patient.

[0054] In an embodiment, endoscope 420 (e.g., endoscope 220, see FIG. 2) automatically and proactively hinges the first and second segments (e.g., segments 232, 234, see FIG. 2, or alternatively, only the second segment 234) in response to detected motion, without requiring steering input from the user. The user provides two inputs: the direction of the camera axis CA (which the user can achieve by tapping). Figure 4 The user inputs the direction (using screen 414) and the translation of the endoscope proximally or distally. No additional input is required for segments 232 and 234 to be turned in the desired direction. Instead, the endoscope 220 will automatically turn to attempt to align its movement with the camera axis. This automatic turning frees the user from focusing on screen 414 (…). Figure 4 The endoscope displays the anatomical structures and the places the user wants to go, without having to figure out how to manually manipulate the endoscope to move in that direction.

[0055] Steering control systems can use computer vision technology to identify changes in camera orientation and / or predict the desired user navigation direction. Figure 5A Figures C to C are schematic diagrams of optical flow techniques for motion alignment according to embodiments of the present disclosure. The figures illustrate exemplary methods for aligning motion with camera axes. Figure 5AFigure C shows a view from an endoscopic camera displayed on a screen 512 of a controller 510, in this case, the controller being a video laryngoscope 518. The screen 512 shows a field of view 530V from the endoscopic camera within the patient's cavity. Figure 5A In the center, the field of view 530V points along the patient's trachea, and this field of view includes a continuous tracheal ring 514A.

[0056] The diverging arrows PF represent the stream of pixels across the screen as the endoscope moves forward into the trachea. As the endoscope moves forward, the individual objects in the field of view will move along these arrows. Therefore, arrows PF indicate the direction of movement of objects in the image as the endoscope is advanced by those objects (as described above, continuing). Specifically, the endoscope's axis of motion is directed toward the point where these objects appear to diverge. This point can also be called the vanishing point VP, which is where arrows PF diverge. As objects in the image appear to move along arrows PF, the endoscope moves toward point VP.

[0057] exist Figure 5A In this model, the vanishing point VP is near the center of the field of view 530V. This indicates good alignment between the motion axis and the camera axis. That is, the camera's field of view points to the vanishing point, which is the expected direction of endoscope movement.

[0058] exist Figure 5B In this view, the vanishing point VP is offset to the right of the camera's field of view 530V. This view can be produced when the camera turns to the left (in the direction shown in Figure 5) while the endoscope continues to move in its previously pointed direction. The flow of objects (along arrow PF) now becomes more parallel instead of diverging from the center of the field of view.

[0059] exist Figure 5C In the middle, the vanishing point VP is not visible. When the camera turns further to the left (in... Figure 5A This view can be generated when pointing in the direction of C. The arrow PF becomes more parallel instead of diverging. This view indicates that the motion axis and camera axis are not aligned.

[0060] Pixel flow, vanishing point, or pixel divergence analysis can be used to actively control the endoscope to improve motion and camera alignment. Pixel groups can be identified as objects in an image, and pixel flow can refer to the movement of objects to different pixels on the camera / display. In an embodiment, the endoscope controller performs automated analysis to generate alignment metrics indicating the degree of alignment between the endoscope's camera axis and motion axis. The controller generates steering signals to hinge a first and / or second hinge segment of the endoscope, thereby improving the alignment metrics.

[0061] In embodiments, pixel characteristics such as pixel brightness, pixel velocity, and pixel depth can be used to track motion. For example, pixel brightness can be used to estimate proximity to the camera (brightness indicates proximity—that is, brighter pixels are more likely to be closer to the camera than dimmer pixels, and dimmer pixels are more likely to be farther away from the camera), and changes in pixel brightness during motion can be used to track local changes in camera orientation.

[0062] In this embodiment, the alignment metric is the deviation of an object (within the field of view) from the center of the field of view. The controller identifies objects (such as vocal cords, bronchial passages, tumors, or other anatomical structures) near the center of the field of view and tracks objects within the field of view. If the object remains near the center, the endoscope may move in its pointing direction. If the object deviates from the center, the endoscope may no longer move in that direction, and the controller hinges the endoscope to compensate. In this way, the camera axis can be actively steered to lock onto a specific anatomical feature. In this embodiment, the controller identifies the passage walls (tissue) in the image data and automatically steers the camera axis to be centered in the passage (pointing between the walls, rather than directly on the walls) and pointing in a forward direction of movement down the passage.

[0063] In this embodiment, the alignment metric is the degree of dispersion (divergence) of pixels moving within the field of view.

[0064] In this embodiment, the alignment metric is the percentage of convergence of optical streamlines in the field of view.

[0065] In one embodiment, the alignment metric is the proximity of a point in the field of view to the center of the field of view. This proximity is an indicator of whether the endoscope is moving toward that point. In one embodiment, the point is the vanishing point (of the pixels moving in the field of view), and the proximity of the vanishing point to the center indicates whether the endoscope is moving in the direction the camera is pointing. In another embodiment, the point is a possible target within the field of view (such as an anatomical feature), and the proximity of the target to the center indicates whether the endoscope is moving toward the target. Anatomical targets can also be used in a negative feedback loop to calculate errors and adjust—for example, if the target moves away from the center of the view, the system turns the endoscope in the opposite direction.

[0066] In this embodiment, the alignment metric is the amount of alignment or discrepancy between the orientation of the distal endoscope and the movement of the endoscope. These two signals—orientation and acceleration—can be obtained from an orientation sensor. If the endoscope moves to the position pointed to by the camera, the orientation and acceleration signals will align.

[0067] In one embodiment, the controller uses local and global orientation information of the endoscope to maintain the camera axis in a desired orientation during endoscope movement and navigation within the patient passage. Local orientation can be extracted, at least in part, from image data captured by the camera. Local orientation may include identifying the presence and location of anatomical features and determining the camera's position and orientation relative to those features. Global information can be extracted from motion signals from an orientation sensor and may include the endoscope's orientation relative to gravity and endoscope movement caused by patient movement or user manipulation. In combination, the local and global information can be used to provide steering control commands to steer the first and / or second articulated segments.

[0068] Figure 6 A perspective view of a controller 610, including a handle or grip 640 and a screen 614, is shown. In an embodiment, the controller 610 is a laryngoscope having a camera 650 and being coupled to a laryngoscope viewing lens 652. The controller 610 is connected to an endoscope 620, which is transmitted through an endotracheal tube 642 (with an inflatable cuff 644). The endoscope 620 is connected to the controller 610 at its proximal end 620P. At its opposite distal end 620D, the endoscope includes two articulated sections 632, 634 and a camera 630. In an exemplary use case, the controller 610 and endoscope 620 are used during a patient intubation procedure. The proximal end 620P of the endoscope is connected to the controller, and an image from the camera 630 is displayed on the screen 614. Using one hand (such as the left), the user taps screen 614 to turn the endoscope camera 630, and using the other hand (such as the right), the user pushes endoscope 620 forward into the patient's cavity. When the endoscope is in place (in the case of intubation, the endoscope passes through the patient's vocal cords into the trachea), the proximal end 620P disconnects from the controller 610, and the endotracheal tube 642 passes through the endoscope. Once the proximal end 620P emerges from the endotracheal tube 642, the endoscope can be reconnected to the controller 610. The endotracheal tube 642 then passes through the endoscope into the trachea, and the endoscope can then be withdrawn from the patient, retracted through the tube 642.

[0069] In an embodiment, the disclosed endoscopic steering technique can be used as part of awakening the intubation, where the user faces the patient and the patient can sit upright. The endoscope 620 can essentially be "flipped" from a first orientation (where the patient's chest is downward on the user's screen) (initially, when the endoscope 620 is inserted into the patient's nose) to a second, opposite orientation (where the patient's chest is upward on the user's screen) (after the endoscope 620 has passed through the nasal passage). The camera axis is maintained by automatic steering performed in the background by the controller 610 and without user input, allowing the user to orient the camera towards specific features of the captured image.

[0070] Each articulated segment at the distal end of the endoscope is actuated by a steering system that operates an actuator coupled to that segment to bend or straighten it. The steering system may include one or more shape-memory metal components (e.g., memory wire, nitinol wire) that change shape based on electrical input, piezoelectric actuators (such as the SQUIGGLE motor from New Scale Technologies of Victor, New York), retractable sheaths (retractable to release pre-formed bent components, such as spring steel that returns to its bent shape when released from the sheath), mechanical control lines (pull cables), hydraulic actuators, servo motors, or other means for bending, rotating, or turning the distal end or components at the distal end of the endoscope.

[0071] Complex motion patterns can be achieved by actuators attached to two separate articulated segments at the distal end of the endoscope. For example, an "S" shape can be produced when the two segments are actuated in different directions (such as one bending upwards and the other downwards). The endoscope includes a flexible housing to allow manipulation of the endoscope within the patient's cavity.

[0072] Furthermore, because the articulation of the segments can change the direction of rotation at the distal end, the distal bending and movement of the endoscope are accomplished independently of the orientation, position, or movement of the proximal end of the endoscope. Therefore, compared to embodiments where steering relies on torsional force transmission, the endoscope structure can have lower torsional stiffness. In one embodiment, the endoscope is an extruded structure with low torsional stiffness (so low that torsional rotation will not translate from the proximal end to the distal end). In another embodiment, the endoscope is a non-woven structure, such as an extruded polymer. In yet another embodiment, the endoscope is an extruded structure without torsional reinforcement, such as braided thread or a braided structure.

[0073] Figure 7 A block diagram including an endoscope 720 and a controller 710 is shown. The connection between them can be wired (in which case they each have an electrical connector) or wireless (in which case they each include a wireless transceiver). The endoscope 720 includes a camera 730 and a orientation sensor 756 at the distal end of the endoscope. The orientation sensor can be an inertial measurement unit (INIU), an accelerometer, a gyroscope, or other suitable sensor. The endoscope 720 also includes a light source 762 and an actuator 760 coupled to a distally steerable segment to bend or straighten it, as described herein.

[0074] The controller 710 includes a processor 766 or chip (such as a chip, processing chip, processing board, chipset, microprocessor, or similar device), hardware memory 768, a display screen 712 (such as a touch screen), and a steering control system 770, which may include a motor or other drive for operating actuators. The controller 710 may also include other types of user inputs (buttons, switches) and power sources (such as onboard removable and / or rechargeable batteries).

[0075] The controller 710 may also include a power source (e.g., an integrated or removable battery) for supplying power to one or more components of the endoscope, and communication circuitry for facilitating wired or wireless communication with other devices. In one embodiment, the communication circuitry may include a transceiver that facilitates handshake communication with a telemedicine device or full-screen monitor. The communication circuitry may provide received images to an attached monitor in real time.

[0076] Figure 8 This is a flowchart illustrating a method 800 for computer-assisted endoscope steering according to an embodiment. The method includes receiving user input to move an observation axis of the endoscope (801), and in response to the user input, hinged the endoscope (such as a first distal hinge segment) to move the observation axis (802). The method also includes receiving a motion signal (803) indicative of movement of the endoscope along a motion axis, such as a motion signal from a orientation sensor, and dynamically comparing the motion axis with the observation axis (804). In an embodiment, comparing the motion axis with the observation axis includes generating an alignment metric indicating the degree of alignment between the two axes. The method also includes generating control signals controlling the hinge of the first and second hinge segments to reduce the discrepancy between the motion axis and the observation axis (805). The control signals include instructions for hinged first and second segments to improve the alignment metric. These steps may be performed by a processor or chip that is part of a controller for the endoscope.

[0077] Figure 9This is a flowchart illustrating a method (900) for computer-assisted endoscopic steering according to an embodiment. The method includes receiving user input via a graphical user interface to move the field of view of the endoscope (901), and hinged a first articulated segment of the endoscope to move the field of view in the direction indicated by the user input (902). The method then includes receiving a motion signal instructing the endoscope to move forward (903), and actively steering a second articulated segment during the forward movement of the endoscope in the absence of user steering input from the user (904). The active steering is performed by a controller that generates a steering signal based on a comparison of the direction of motion and the direction of the field of view to steering the second articulated segment, as described above. As the user pushes and advances the endoscope forward within a patient's cavity, the controller automatically steering the first and / or second articulated segments to align the direction of motion with the direction of observation. This active steering is performed without any further steering input from the user; at this point, the user can simply advance the endoscope forward, and the controller will automatically steering the first and / or second articulated segments. This automatic steering, which requires no steering input from the user, allows the user to focus on the view from the endoscope camera and the forward movement of the endoscope, without having to work simultaneously to manually steer the articulated part of the endoscope.

[0078] Based on this method, user input is limited to pointing at the camera and advancing the endoscope, without bending the articulated segments to navigate through the patient's anatomy. By pointing the camera where the user wants to go and then advancing the endoscope, the controller automatically bends the first and second articulated segments to align the axis of motion with the direction the user wants to go. The controller bending these segments is equivalent to a virtual gimbal behind the camera, rotating the endoscope behind the camera to keep it moving in the direction the camera is pointing. In this way, the user is prompted to provide more intuitive input, which typically indicates the desired camera orientation while controlling the endoscope's forward movement. The user provides coarse steering guidance, for example, via a touchscreen, and the controller generates instructions for finer or more precise steering control based on this coarse guidance. Furthermore, based on the user's steering input or steering locked onto a specific anatomical feature, the controller can predict or estimate future steering instructions. For example, the desired orientation within the passage can be predicted based on the absolute or relative position of the distal end of the patient and / or features identified in the image. This prediction or interpretation of the user's intent can be used to maintain the desired orientation of the camera's field of view, for example, at the center of the passage or to keep the anatomical feature at the center of the passage. The user's forward steering movements at the proximal endoscope can vary from user to user based on their preferences. However, the controller corrects for these variations by automatically steering based on the desired orientation of the camera axis and maintaining that desired orientation, which corrects for user variations in the way the proximal endoscope is manipulated. Given an image where only local information is relevant to the user and overall information from the orientation sensor is hidden from the user, the algorithm searches for specific features or potential targets. The user's touch coordinates, the speed and magnitude of the gesture can indicate which potential target the user is aiming at, for example, using filtering or a Long Short-Term Memory (LSTM) network. In the case where the user's thumb is on the screen, the gesture is parsed from the time series.

[0079] In addition to illustrating the movement of the endoscope operated by the user, this technology also provides correction or adjustment for patient movement during endoscopic procedures. During certain procedures, the patient may move independently or be repositioned by a caregiver; for example, the patient may sit up, turn over, etc. These patient movements are reflected in motion signals from a direction sensor, which can provide the endoscope's orientation relative to gravity or its absolute orientation. Changes in absolute orientation relative to the desired camera axis can be analyzed, allowing the controller to automatically adjust the camera position to account for the patient's movement, thereby returning the camera axis to its desired orientation. In one example, an endoscope positioned behind the patient is used, where the patient's anterior side corresponds to an absolutely upward position and the posterior side corresponds to a gravity-down position. In this orientation, the camera is also oriented in the caregiver's gravity direction and absolute orientation. In cases where the patient is flipped to be positioned on the patient's side or stomach, the controller can redirect the image and / or indicate these changes in orientation relative to gravity via a graphical user interface, showing the camera's reference frame rotating from its original orientation, and can convert steering commands from the camera axis's reference frame to the endoscope's reference frame. In this way, anatomical structures are presented in a way familiar to the user. In this embodiment, the user can switch between gravity orientation and patient orientation. If the endoscope is inside the patient during rotation, orientation signals and camera transmissions can be coordinated to indicate that the patient is being repositioned. If the patient is already in a non-supine position when the endoscope is introduced, the image can be redirected.

[0080] Processor (e.g., processor 766, see...) Figure 7 The memory may include one or more application-specific integrated circuits (ASICs), one or more general-purpose processors, one or more controllers, FPGAs, GPUs, TPUs, one or more programmable circuits, or any combination thereof. For example, the processor may also include or relate to control circuitry for the display screen. The memory may include volatile memory, such as random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM). Image data captured by an endoscopic camera and / or a laryngoscope camera (if present) may be stored in the memory and / or may be directly provided to the processor. Furthermore, image data from each patient procedure may be stored and collected for later examinations. The memory (e.g., hardware memory 768, see...) Figure 7 This may include stored instructions, code, logic, and / or algorithms that can be read and executed by a processor to perform the techniques disclosed herein.

[0081] Although this technique is discussed in the context of endotracheal intubation, it should be understood that the disclosed technique can also be used for other types of airway management or clinical procedures. For example, the disclosed technique can be used in conjunction with the placement of other devices within the airway, removal of secretions from the airway, arthroscopic surgery, visualization of the bronchi through the vocal cords (bronchoscopy), tube exchange, lung biopsy, nasal or transnasal endotracheal intubation, etc. In some embodiments, the disclosed visualization instrument can be used for visualization of anatomical structures (such as the pharynx, larynx, trachea, bronchi, stomach, esophagus, upper and lower airways, ear, nose and throat, vocal cords), or biopsy of tumors, masses, or tissues. The disclosed visualization instrument can also be used for, or in conjunction with, visualization of tissue aspiration, drug delivery, ablation, or other treatments, and can also be used in conjunction with endoscopes, probes, guides, scopes, or probes.

[0082] While this disclosure is open to various modifications and alternatives, specific embodiments are illustrated by way of example in the accompanying drawings and are described in detail herein. However, it should be understood that the embodiments provided herein are not intended to limit one to the specific forms disclosed. Rather, various embodiments may cover all modifications, equivalents, and alternatives falling within the spirit and scope of this disclosure as defined by the appended claims.

Claims

1. A computer-controlled endoscope system comprising: an endoscope comprising: a flexible tubular body comprising a first articulating segment at a distal end of the flexible tubular body, and a second articulating segment coupled to a proximal end of the first articulating segment, wherein the first articulating segment comprises a camera having a field of view along a camera axis and an orientation sensor sensitive to movement along a motion axis; and a controller in communication with the endoscope and comprising a hardware memory storing instructions for analyzing an alignment between the motion axis and the camera axis and steering the first articulating segment and the second articulating segment of the endoscope during endoscope motion to improve the alignment by reducing a difference between the motion axis and the camera axis by steering the endoscope to align forward motion of the endoscope with the camera axis, wherein the orientation sensor is configured to provide motion signals to the controller indicative of an orientation and forward motion of the endoscope.

2. The endoscope system of claim 1, wherein, the endoscope is detachable from the controller and is disposable, and wherein the controller is reusable.

3. The endoscope system of claim 1, further comprising a graphical user interface responsive to user commands to move the camera axis.

4. The endoscope system of claim 1, wherein, the instructions for analyzing the alignment comprise analyzing an optical flow of pixels from the camera during the endoscope motion.

5. The endoscope system of claim 1, wherein, the instructions for analyzing the alignment comprise generating an alignment metric.

6. The endoscope system of claim 1, wherein, the controller comprises a graphical user input comprising a touchscreen display, and the controller further comprises instructions for steering the first articulating segment in response to input from a user on the touchscreen display.

7. The endoscope system of claim 1, wherein the camera is located at a distal tip of the first articulating segment and the orientation sensor is located proximal to the camera.

8. The endoscope system of claim 1, wherein, communication between the endoscope and the controller is through a direct wired connection.

9. The endoscope system of claim 5, wherein, the instructions for analyzing the alignment comprise identifying an object near a center of the field of view and tracking movement of the object within the field of view, and wherein the alignment metric comprises a deviation of the object from the center of the field of view.

10. The endoscope system of claim 5, wherein, the alignment metric comprises a degree of dispersion of pixels moving in the field of view.

11. The endoscope system of claim 5, wherein, the alignment metric comprises a degree of convergence or divergence of optical flow lines in the field of view.

12. The endoscope system of claim 5, wherein, the alignment metric comprises a proximity of a vanishing point in the field of view to a center of the field of view.

Images