Graphical user interface for steerable endoscope

Abstract

An endoscope system is provided with a graphical orientation indicator. The endoscope includes a flexible tubular body having a steerable distal end with a camera. A controller for the endoscope presents a user interface that displays a view from the camera and a graphical orientation indicator that visually represents the orientation of the distal end of the endoscope.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Application 62 / 928854, filed October 31, 2020, the disclosure of which is incorporated herein by reference in its entirety for all purposes. Background Technology

[0003] This disclosure relates generally to medical devices, and more specifically to steerable guides having graphical orientation indicators (e.g., displayed orientation indicators), such as flexible endoscopes, as well as related methods and systems.

[0004] Medical endoscopes are long, flexible instruments that can be introduced into a patient's cavities during medical procedures in a variety of situations to facilitate visualization and / or procedures within those cavities. For example, one type of endoscope is one with a camera at its distal end. Endoscopes can be inserted into a patient's mouth, throat, or other cavities to help visualize anatomical structures or facilitate procedures such as biopsies or ablation. Endoscopes may include a steerable distal tip that can be actively controlled to bend or rotate in the desired direction to obtain the desired view or navigate through anatomical structures. However, it can be difficult to maneuver these steerable endoscopes to the desired location and orientation within the patient's anatomy. Summary of the Invention

[0005] The following outlines some embodiments whose scope is commensurate with the subject matter of the original claims. These embodiments are not intended to limit the scope of this disclosure. In fact, this disclosure may cover a variety of forms that may be similar to or different from the embodiments set forth below.

[0006] In one embodiment, the endoscope system includes an endoscope having a steerable distal endpiece with a camera and an orientation sensor. The endoscope system also includes a handheld controller in communication with the endoscope. The handheld controller includes a display screen and hardware memory. The hardware memory stores instructions for: displaying on the display screen a view from the camera and a graphic orientation indicator representing the orientation of the steerable distal endpiece; detecting rotational movement of the steerable distal endpiece; maintaining the rotational orientation of the view from the camera displayed on the display screen; and modifying the graphic orientation indicator to represent the detected rotational movement of the steerable distal endpiece.

[0007] In another embodiment, the endoscope system includes an endoscope having a steerable distal end, the steerable distal end having a camera and an orientation sensor. The endoscope system also includes a controller that communicates with the endoscope and includes a display screen. The endoscope system also includes a user interface (UI) presented on the display screen. The UI includes: a window displaying a view from the camera; an orientation icon visually representing a first orientation of the steerable distal end; and a touch interface for receiving touch input from a user. The controller includes hardware memory storing instructions for: detecting a change in a second orientation of the steerable distal end of the endoscope, and modifying the orientation icon to visually represent the second orientation.

[0008] In another embodiment, a method for operating a UI for an endoscope system includes the following steps: displaying a view from the endoscope and a graphic orientation indicator on a touchscreen display, the endoscope including a rotatable distal end with a camera; receiving user input via the touchscreen display to steer the distal end of the endoscope; steer the endoscope to the changed orientation of the distal end in response to the user input; and modifying the graphic orientation indicator to visually represent the changed orientation of the distal end.

[0009] A feature of one aspect or implementation may be applied as a feature of any other aspect or implementation in any suitable combination. For example, any one of the features of a system, laryngoscope, handle, controller, endoscope, or method may be applied as any one or more of the features of a system, laryngoscope, controller, endoscope, or method. Attached Figure Description

[0010] The advantages of the disclosed technology will become apparent from the following detailed description and with reference to the accompanying drawings, in which:

[0011] Figure 1 This is a front view of a user interface (UI) for an endoscope system according to an embodiment of this disclosure.

[0012] Figures 2A-2E The implementation schemes according to this disclosure include Figure 1 A series of views of the endoscopic system in the UI.

[0013] Figure 3 and Figures 3A-3C It is a series of views of an endoscope system and a UI according to an embodiment of this disclosure.

[0014] Figure 4 This is a front view of a UI for an endoscope system according to an embodiment of this disclosure.

[0015] Figures 5A-5EThe implementation schemes according to this disclosure include Figure 4 A series of views of the endoscopic system in the UI.

[0016] Figure 6 This is a front view of a UI for an endoscope system according to an embodiment of this disclosure.

[0017] Figures 7A-7D It is a series of views of an endoscope system and a UI according to an embodiment of this disclosure.

[0018] Figure 8 This is a block diagram of an endoscope system according to an embodiment of the present disclosure.

[0019] Figure 9 This is a flowchart of a method for using a UI in conjunction with an endoscope system. Detailed Implementation

[0020] The medical endoscopes or endoscopes described herein are thin, elongated, flexible instruments that can be inserted into body cavities for exploration, imaging, biopsy, or other clinical treatments, including catheters, narrow tubular instruments, or other types of endoscopes or probes. Endoscopes can be navigated into body cavities (such as a patient's airway, gastrointestinal tract, oral cavity, nasal cavity, or other cavities or openings) and are steered by the user via advancing the distal end to the desired position, and in some embodiments, through biomimetic motion. The endoscope may be tubular in shape.

[0021] Advancing a long, flexible medical device into a patient's cavity typically involves a force transmitted from the proximal portion of the device (outside the cavity), which causes the distal tip to advance within the cavity. For example, a physician or other caregiver may grasp the proximal portion (such as a handle) of the medical device outside the cavity and push it down or forward, and the resulting motion is transmitted to the distal tip, causing it to move forward within the cavity. Similarly, a pull applied by a caregiver to the proximal portion may cause the distal tip to retract, or in the opposite direction, remove it from the cavity. However, because patient cavities are not regular in shape or size, the endoscope moves through a tortuous path, and the transmitted force from the pushing or pulling motion of the proximal end may not result in predictable movement at the distal tip. Furthermore, navigating the endoscope through a winding passage may involve multiple rotations or turns of the distal end, and the user may have difficulty tracking the changing orientation of the distal tip during and after navigation. Subsequently, the user may inadvertently bend or rotate the endoscope in the wrong direction. For example, a user might intend to turn the endoscope to their right, but because the endoscope starts rotating from its default position, the command results in the endoscope bending to the user's left. Therefore, users may find it difficult to predict how the endoscope will respond to turning commands during medical procedures.

[0022] This document provides a graphical user interface (UI) for a steerable endoscope, wherein one or more graphical orientation indicators indicate the current orientation or articulation of the endoscope. According to one embodiment, the endoscope system includes: an endoscope having a steerable distal end and a camera; and a control rod having a display screen and user input elements (such as a touchscreen). The display screen shows a view from the camera, as well as graphical indicators showing how the endoscope has been rotated or turned, to assist the user in manipulating the endoscope during medical procedures.

[0023] Figure 1 and Figures 2A-2E An endoscope system 100 according to one embodiment is shown. The system includes an endoscope 12 and a controller 14 having a display screen 16. The endoscope 12 includes a steerable distal end 24 carrying a camera and a directional sensor. The controller 14 is coupled to the endoscope 12 to provide power and / or control commands to the steerable endoscope 12, and to receive data from the camera and the directional sensor. In this embodiment, the controller 14 is implemented as a video laryngoscope.

[0024] A user interface (UI) 18, such as a graphical user interface (GUI), is presented on the display screen 16. The UI 18 includes a view 20 from the endoscope (such as a video feed or still image from an endoscopic camera), a touchscreen interface, and three graphical orientation indicators 30, 32, and 34. The endoscopic view 20 is presented in a circular shape within a circular outline 22. In the example shown, the view points towards the user's vocal cords and trachea 26. Figures 2A-2E This includes five different presentations of UI 18 across five views. The diagram also shows an endoscope 12 connected to a laryngoscope 14, such as by connecting the proximal end of the endoscope 12 to a port on the back of the laryngoscope screen 16. The opposite distal end 24 of the endoscope is inserted into the patient. The distal end 24 is capable of active swivel and includes a camera and orientation sensors, as described below.

[0025] It should be understood that the disclosed technology can support visualization and navigation for orientation of blind-steering guides, i.e., excluding cameras or slender, flexible instruments where cameras are not activated.

[0026] The three directional indicators 30, 32, and 34 are in Figure 1 Shown in a close-up view of UI 18. Orientation indicator 30 is a scroll indicator that graphically depicts the current scroll orientation of the distal end 24 of endoscope 12. Figure 1In one embodiment, the scroll indicator 30 includes a horizon indicator represented by two markers 30a and 30b, placed 180 degrees apart on or near the circular outline 22. Markers 30a and 30b are shown as small solid squares, but can be other shapes and sizes. For example, in another embodiment, the horizon indicator is a dashed line running from marker 30a to marker 30b, shown in view 20. When markers 30a and 30b are horizontal on the screen, the distal end 24 is in its default or stationary position, meaning it has not been actively turned to rotate.

[0027] As the distal endoscope 24 rotates within the patient's cavity, the scroll indicator 30 moves. This scrolling movement... Figure 2A and Figure 2D As shown in the view. Figure 2A and Figure 2D In the view, the distal end 24 has been rotated approximately 20 degrees clockwise relative to its default position. To indicate this rotation, markers 30a and 30b have been moved proportionally by approximately 20 degrees around image view 20. It is worth noting that in... Figures 2A-2E In this embodiment, the image view 20 itself has not been rotated. The endoscopic system includes an image correction or adjustment feature that reverses the rotation of the image to make it upright (with the patient's posterior orientation facing the patient's chest, maintained in an "upward" direction on screen 16), even when the endoscope is actively turned and rotated. This type of image adjustment can be achieved by reversing the rotation of the image proportionally based on the amount of rotation detected by a directional sensor at the distal end of the endoscope, as more fully described in U.S. Provisional Application 62 / 812678, filed March 1, 2019, and U.S. Patent Application 16 / 802242, Publication 20200275824, the entire contents of which are hereby incorporated by reference.

[0028] Refer again Figures 2A-2E While the image in image view 20 remains stable, the user can actively steer the endoscope, such as rotating the distal end 24. In this situation, the user may lose track of the steering input provided to the endoscope and may have difficulty remembering how the endoscope is currently rotated or positioned within the patient's cavity. The scroll indicator 30 provides the user with a visible graphical representation of the endoscope's current rotational orientation, making it easy for the user to track this orientation.

[0029] exist Figure 1 and Figures 2A-2EIn the image 20, the scroll indicator 30 also includes a working channel indicator 30W. The working channel indicator is shown as a transparent shaded area overlaid on the image 20. This indicator shows the rotational position of the working channel on the endoscope 12. The working channel can be an open lumen formed longitudinally within the endoscope to allow the passage of tools or fluids, such as biopsy tools, nebulizing catheters, saline, and fluids to be drained from the body. The working channel can be formed offset from the central axis of the endoscope, in which case it will rotate as the endoscope rotates. On UI 18, indicator 30W visually indicates the current location of the working channel, allowing the user to anticipate where tools, fluids, or other instruments will enter the image 20 when it emerges from the working channel. If the working channel is not aligned with the anatomical area to be accessed (for a particular clinical procedure), the user can steer the endoscope to move the working channel to the desired position.

[0030] The second orientation indicator 32 is a pitch / yaw indicator that indicates the amount and direction in which the endoscope has been bent, folded, or rotated. In the illustrated embodiment, the pitch / yaw indicator 32 is depicted by brackets 32a, 32b, 32c, and 32d formed around the contour 22 of the image 20. The brackets expand and contract as the endoscope rotates left or right (yaw) or up or down (pitch), as... Figure 2B , 2C The views are shown in 2D and 2E. Figure 2B In the view, the distal end 24 has rotated to the right, and brackets 32a and 32d have grown larger, while brackets 32b and 32c have become smaller, indicating that the endoscope has rotated to the right. The growing lines (32a, 32d) also have arrows at their ends in the direction of rotation of the endoscope. Figure 2B In the view, the endoscope has rotated to the right portion of its full range of motion. Figure 2C In the view, the endoscope has rotated all the way to the limit of its range of motion, and therefore lines 32a and 32d have come all the way to the vicinity of image view 20, while lines 32b and 32c are either completely removed or only shown as small dashed lines or small markers. With these changes, the yaw / pitch indicator 32 shows the user which direction the endoscope has turned and how far it has gone within its range of motion. The user can then see how the endoscope has rotated or bent and how much further it can rotate by referring to UI 18.

[0031] Looking up and down Figure 2EThe view indicates this. In this view, lines 32c and 32d on the bottom side of image view 20 have grown, and lines 32a and 32b on the top have contracted, causing these lines to show the user that the endoscope is bent upwards. Lines 32c and 32d have arrows pointing upwards in the same direction to the top of the screen to further reinforce the direction of the endoscope's rotation.

[0032] The third orientation indicator 34 is a graphic representation of the distal end 24 of the endoscope 12. For example, this indicator 34 could be a cartoon image or icon representing the tip of the endoscope. Figures 2A-2E As shown in the view, the icon bends and rotates in coordination with the distal end 24 of the endoscope. This indicator 34 provides another way to visually represent the current orientation of the endoscope on the screen 16.

[0033] Figures 2A-2E The view in the image shows the UI 18 in various states based on the active movement of the endoscope 12. Figure 2A In the view, the endoscope has been rotated approximately 20 degrees clockwise, as indicated by the clockwise rotation of the scroll indicator 30 (marked 30a, 30b and working channel indicator 30W). Figure 2B In the view, the endoscope is rolled back to the center position and rotated to the right midway, as indicated by indicator 32 (brackets) and indicator 34 (pointed icon). Figure 2B The thumbprint in the view shows an example of user input to turn the endoscope to the right; in this case, the user taps or drags / swipes their thumb to the right to guide the endoscope to bend to the right. Figure 2C In the view, the user further taps or drags / swipes to the right (as shown by a thumbprint) to guide the endoscope to continue bending to the right. In this example, in Figure 2C In the view, the endoscope reaches the limit of its rightward movement. Figure 2D In the view, the endoscope rolls to the right (clockwise). In one implementation, the rotational movement of the endoscope is achieved by the user manually twisting the endoscope. Figure 2E In the view, based on user input, the endoscope tilts upward to move the view up (e.g., as...). Figure 2E The thumbprint in the view shows the thumb moving upwards. Although these views are shown as snapshots, indicators 30, 32, and 34 are dynamic and change in real time with the position of the distal end 24. These indicators can move between discrete configurations, or can continuously change shape, color, or form, or can be animated.

[0034] Orientation indicators can be presented as follows: within the clinical view from the endoscopic camera, outside the clinical view on the display screen, or by overlapping the clinical view on the screen with other areas. Figure 3 and Figures 3A-3C Another endoscope system 300 according to one embodiment is shown. In this embodiment, endoscope 312 is connected to a video laryngoscope 314 having a display screen 316. A graphical user interface (UI) 318 is presented on the display screen. UI 318 includes two orientation indicators 330, 336. The first orientation indicator 330 includes a scroll indicator that indicates the rotational position of the distal end 324 of endoscope 312. The scroll indicator includes two triangular horizon indicators 330a, 330b and a working channel indicator 330W. In this embodiment, the working channel indicator 330W overlaps with the circular outline 322 of the camera view 320, thereby passing through the circular outline. The working channel indicator 330W is shown as two lines generally perpendicular to the circular outline, partly inside and partly outside the circular camera view 320. Indicators 330a, 330b, 330W move about the circle 322 in directional synchronization with the distal end 324 of endoscope 312. For example, in Figure 3A In the view, the endoscope has been rotated approximately 45 degrees clockwise, and the indicators 330a, 330b, and 330W have rotated around the circle by the corresponding amount.

[0035] The second graphic indicator 336 includes a gravity indicator that indicates the orientation of the distal tip 324 relative to gravity. The gravity indicator can be presented in various styles or shapes. In this embodiment, it is shown as a pin or marker with its tip pointing downwards towards the bottom of the screen 316. The pin is located within a circle having a line or plane indicating the horizontal horizon. The view of the pin changes as the endoscope tip tilts up or down. Figure 3C As shown in the view, the gravity indicator is tilted to indicate that the endoscope tip 324 is tilted upwards. When the tip 324 is tilted upwards, the plane of the probe is also tilted upwards. When the endoscope is in the neutral position (neither tilted upwards nor downwards), the gravity indicator is horizontal, as shown in the view. Figure 3A and 3B The view is shown.

[0036] UI 318 also includes a status indicator or icon 340 above the camera view 320. In the illustrated embodiment, the status indicator 340 is a lung icon. This area of ​​the UI can be used to indicate what type of procedure is being performed or other types of status information, such as distance relative to the vocal cords or distance relative to a user-defined landmark. The UI implementation disclosed herein can be displayed on a controller communicating with the endoscope. The controller can be a handheld device with a screen, such as a video laryngoscope, rod, calibrator, tablet, or other type of display. Figure 4 and Figures 5A-5EAn embodiment where the controller is implemented as a handheld stick is shown. Endoscopic system 400 includes an endoscope 412, which is connected to a handheld stick 413 having a display screen 416. A UI 418 is presented on the screen 416. In this embodiment, UI 418 includes a scroll indicator 430 (including two horizon markers 430a and 430b spaced 180 degrees apart and having a dashed horizon line between them, and a working channel indicator 430W). Figure 5A As shown in the view, the scroll indicator 430 rotates on UI 418 to indicate the degree and direction of rotation of the distal end 424 of endoscope 412. As described above, when the endoscope rotates, in one embodiment, the view from the endoscope camera is adjusted on screen 416 to keep the direction of gravity downwards towards the bottom of screen 416. Therefore, in Figure 5A In the view, the scroll indicator 430 rotates, but the image 420 does not rotate.

[0037] UI 418 also includes a pitch / yaw indicator 432, implemented in this case as a crosshair or bubble 440. Bubble 440 indicates the direction of yaw movement (left or right) or pitch movement (up or down) of the endoscope 412. Figure 5B In the view, the user turns the endoscope to the right (as shown by tapping the touchscreen to the right with a thumbprint), and the bubble 440 also moves to the right within the camera viewing window 420 on the screen. Figure 5C In the view, the user tilts the endoscope upwards, and the bubble 440 moves upwards. In one embodiment, the distance the bubble 440 moves within the window 420 corresponds to the amount the endoscope has rotated. In one embodiment, the bubble 440 reaches a circular profile 422 when the endoscope reaches its limit of motion. This is depicted in... Figure 5D In the view, the endoscope is tilted up (bent upwards) to its maximum extent, and the bubble 440 is in contact with the top edge of the circle 422. In one embodiment, UI 418 also provides additional limit indicators, such as a light 442, when the endoscope reaches its movement limit. The light 442 appears around the circular view 420 when the endoscope reaches its movement limit, and then disappears as the endoscope moves away from the corresponding limit and back in another direction. The controller 414 can also provide a tactile response when the movement limit is reached, such as providing a short pulse, a buzzing sound, or a vibration from a tactile generator. The tactile feedback can be proportional to the magnitude of the signal; the greater the feedback, the stronger its correlation with the endoscope's movement being at or beyond a certain range of motion.

[0038] like Figure 4As shown in Figure 5, UI 418 also includes a gravity indicator 436, which indicates the direction of gravity relative to the endoscope's orientation. When the endoscope is level with gravity (neither tilted up nor down), the gravity indicator 436 shows a horizontal line below the tip of the probe. When the endoscope tip is tilted up (as shown in Figure 5), the gravity indicator 436 indicates the direction of gravity relative to the endoscope's orientation. Figure 5C When the endoscope tip reaches its upward limit (as in the view), the gravity indicator also tilts upward, causing the view to move upward beyond the tip of the probe. Figure 5D In the view (420), the gravity indicator has moved all the way to the top, past the pointed tip, to indicate that the direction of gravity relative to the endoscope is directly below view 420. This is just one way to indicate the direction of gravity relative to the tip, and other gravity indicators can be used.

[0039] exist Figure 4 and Figures 5A-5E In this embodiment, the controller is a handheld rod 413, not a video laryngoscope (shown in other figures). An endoscope 412 is directly connected to the rod 413, for example by inserting the proximal end of the endoscope 412 into a port 415 on the rod 413, for transmitting control signals from the rod to the endoscope and for transmitting video signals from the endoscope to the rod. In one embodiment, the endoscope 412 is removable from the rod 413. The endoscope 412 may be for single use only and discarded (or recycled) after use, while the controller (such as the rod 413) may be reusable. In other embodiments, the controller may have other forms or structures. For example, the controller may be a video laryngoscope, a desktop display, a tablet computer, a laptop computer, a calibrator, or other form factor. The controller with the endoscope operates as a two-component endoscope, where the controller serves as the handle, display, and user input for the endoscope. In one embodiment, the controller is reusable, while the endoscope is for single use and discarded after use to prevent cross-contamination between patients or caregivers. The controller itself does not require contact with the patient and can be wiped and cleaned to be ready for the next patient along with a new sterile endoscope.

[0040] Figure 6 and Figures 7A-7DAn embodiment of a controller implemented as a video laryngoscope is shown. In this embodiment, two image views are simultaneously displayed on the display screen, one from a camera on the video laryngoscope and one from a camera on the endoscope. As shown, the endoscope system 600 includes an endoscope 612 connected to a video laryngoscope 614 having a display screen 616. The display screen presents a UI 618, which displays a first view 644 from the video laryngoscope camera and a second view 620 from the endoscope camera. The two image views are displayed simultaneously on the UI, such as one image overlapping the other. In one embodiment, the video laryngoscope image 644 is shown within a first contour shape (such as a rectangular corner or contour), and the endoscope image 620 is shown within a second, different contour shape (such as a rounded corner or contour 622) to help distinguish the two images. Figure 6 In the image, the distal end 624 of the endoscope 612 is visible in the first image view 644 from the video laryngoscope camera.

[0041] UI 618 includes a scroll indicator 630 and a pitch / yaw indicator 632. The scroll indicator 630 may be located on circle 622 to correspond to the position of the endoscope's working channel. Figures 7A-7D As shown in the view, the roll indicator 630 rotates to indicate the roll of the distal end 624, and the pitch / yaw indicator line 632 moves around the profile 622 to indicate the left / right yaw or pitch / pitch of the distal end 624. Figure 7A In the view, the user uses their right hand to twist or rotate the endoscope to the left (counterclockwise), and the scroll indicator 630 moves the corresponding amount along the circle 622. The user can see the movement of the distal end 624 within the video laryngoscope image 644. Figure 7B In the view, the user guides the endoscope to yaw to the left, such as by tapping the screen 616 in a leftward direction with their left thumb. The yaw / pitch indicator line 632 moves to the left around a circle to indicate changes in the endoscope's position. The video laryngoscope image 644 shows the distal end 624 bent to the left. Figure 7C In the view, the user guides the endoscope to tilt upwards, such as by tapping the screen upwards. The yaw / pitch indicator line 632 moves upwards by a corresponding amount, and the video laryngoscope image 644 shows the distal end 624 curving upwards. Figure 7D In the view, the user advances the endoscope through the patient's vocal cords and into the trachea 626, and guides the endoscope downwards, as indicated by tapping the screen with the thumb and the yaw / pitch indicator 632. Endoscopic image 620 now shows a view of the patient's tracheal wall.

[0042] In an exemplary use case, the endoscope system 600 is used for intubation of a patient. The user holds the video laryngoscope 614 with one hand (such as the left hand) and taps the screen 616 to orient the endoscope 612, while the user uses the other hand (such as the right hand) to push the endoscope 612 forward into the patient's cavity. When the endoscope is in the appropriate position (for intubation), the endoscope passes through the patient's vocal cords into the trachea, as... Figure 7D (As shown in the view), the proximal end of endoscope 612 is disconnected from laryngoscope 614, and the endotracheal tube passes through the endoscope. Once the proximal end exits from the endotracheal tube, endoscope 612 can be reconnected to laryngoscope 614. The endotracheal tube is then passed along the endoscope into the trachea, and endoscope 612 can then be withdrawn from the patient, retracted through the tube, and left in place with the endotracheal tube in place.

[0043] An endoscope includes one or more steerable segments at its distal end for actively bending, rotating, or flexing the distal end of the endoscope. In one embodiment, the steerable segments can be bent and flexed in three dimensions (not just in a single plane, such as up / down or right / left), thus flexing in all directions until the limits of their range of motion are reached. For example, in one embodiment, each segment can be bent up to 90 degrees in any direction, enabling it to move within a hemisphere with a radius equal to the segment length. Each segment is manipulated by its own actuation system, including one or more actuators (such as sheath-type drawstrings or other actuators described below) that move to bend the segment into a curved shape or straighten it from that curved shape. In one embodiment, the endoscope includes two steerable segments at its distal end. The two steerable segments are connected to each other, and each segment can articulate independently of the other segment.

[0044] Each articulated segment at the distal end of the endoscope is manipulated by a steering system that operates an actuator coupled to that segment to bend or straighten it. The steering system may include one or more shape-memory metal components (e.g., shape-memory guidewires, nitinol guidewires) that change shape based on electrical input, piezoelectric actuators (such as SQUIGGLE motors from New Scale Technologies, Victor Ny), retractable sheaths (retractable to release pre-formed bent components, such as spring steel, which return to their bent shape when released from the sheath), mechanical control lines (pull wires), hydraulic actuators, servo motors, or other means for bending, rotating, or turning the distal end of the endoscope or components at the distal end.

[0045] In one embodiment, the camera is located at the distal end or tip of the most distal segment of the endoscope to provide a clear anterior view of the patient's anatomy. In one embodiment, the orientation sensor is also located at the distal end, just proximal to the camera. In one embodiment, the orientation sensor is adjacent to the camera. In one embodiment, the orientation sensor is mounted on flexible circuitry behind the camera. In one embodiment, the orientation sensor is mounted on the same flexible circuitry as the camera, but the orientation sensor and the camera do not necessarily communicate on the shared flexible circuitry. In one embodiment, the orientation sensor has a size between 1 mm and 2 mm in each dimension.

[0046] An orientation sensor is an electronic component that senses the orientation (such as orientation relative to gravity) and / or movement (acceleration) of the distal end of an endoscope. To achieve this, the orientation sensor comprises one or a combination of sensors, such as an accelerometer, magnetometer, and gyroscope. The orientation sensor may be an inertial measurement unit (IMU). The orientation sensor detects the static orientation and dynamic movement of the distal tip of the endoscope and provides an indication signal of changes in the orientation and / or movement of the endoscope. The orientation sensor sends this signal to a controller. The orientation sensor is located inside the tubular housing of the endoscope. In one embodiment, the orientation sensor is located very close to the distal end of the endoscope, such as behind a camera, so that the orientation sensor can capture most of the range of motion of the distal tip and the camera. In one embodiment, the orientation sensor is positioned at the distal end of a first steerable portion, away from the proximal end of that steerable portion, to position the orientation sensor away from the pivot point of movement.

[0047] Figure 8 A block diagram of an endoscope system 800 is shown, which includes an endoscope 812 and a controller 814. The connection between them can be wired (in which case they each have an electrical connector) or wireless (in which case they each include a wireless transceiver). The endoscope 812 includes a camera and an orientation sensor located at the distal end of the endoscope. The orientation sensor can be an inertial measurement unit (IMU), an accelerometer, a gyroscope, or other suitable sensor. The endoscope 812 also includes a light source and an actuator coupled to a steerable section at the distal end to bend or straighten them, as described herein.

[0048] The controller 814 includes a processor or chip (such as a chip, processing chip, processing board, chipset, microprocessor, or similar device), hardware memory, a display screen (such as a touch screen), and a steering control system, which may include a motor or other drive for operating actuators. The controller 814 may also include other types of user input elements (buttons, switches) and power sources (such as onboard removable and / or rechargeable batteries).

[0049] The controller 814 may also include a power supply (e.g., an integrated or removable battery) that powers one or more components of the endoscope and a communication circuitry system to facilitate wired or wireless communication with other devices. In one embodiment, the communication circuitry system may include a transceiver that facilitates handshake communication with a telemedicine device or full-screen monitor. The communication circuitry system can provide received images to another monitor in real time.

[0050] The processor may include one or more application-specific integrated circuits (ASICs), one or more general-purpose processors, one or more controllers, FPGAs, GPUs, TPUs, one or more programmable circuits, or any combination thereof. For example, the processor may also include or refer to a control circuitry system for the display screen. Memory may include volatile memory, such as random access memory (RAM); and / or non-volatile memory, such as read-only memory (ROM). Image data may be stored in memory and / or may be directly provided to the processor. Furthermore, image data for each patient procedure may be stored and collected for later review. The memory may include stored instructions, code, logic, and / or algorithms that can be read and executed by the processor to perform the techniques disclosed herein.

[0051] Figure 9 The flowchart illustrates a method 900 for operating a UI of an endoscope system. The method includes: at 901, displaying a view from the endoscope and a graphical orientation indicator. In one embodiment, the endoscope includes a steerable distal end having a camera, and the display is positioned on a touchscreen display. The method also includes: at 902, receiving user input via the touchscreen display to steer the distal end of the endoscope. In response to the user input, the method includes: at 903, steer the endoscope to the changed orientation of the distal end. The method includes: at 904, modifying the graphical orientation indicator to visually represent the changed orientation of the distal end. In one embodiment, the graphical orientation indicator includes one or more of a scroll marker, a pitch marker, or a yaw marker, and modifying it includes: moving the marker around or along the camera view to represent the changed orientation of the distal end of the endoscope.

[0052] In one implementation, the UI is implemented on a touchscreen that responds to taps, touches, or proximity gestures from the user. The UI detects the user's touch and sends this touch input to a processor, which generates instructions to operate the steering system to bend, rotate, or move the endoscope. For example, a user can input a touch gesture (such as a tap, double tap, tap and hold, swipe, or slide) to identify a target point or direction within an image on the screen. This gesture identifies where the user wants to turn the endoscope, and the controller translates this into a real-world steering direction and corresponding instructions to operate the steering system to move the steerable segment of the distal end of the endoscope in that direction. The user can swipe the touchscreen in the desired direction to redirect the distal end of the endoscope. The desired orientation or movement of the camera can be interpreted from the direction and length of the swipe movement on the touchscreen. In one example, the direction of movement of the distal end of the endoscope is the same as the direction of movement input by the user. Swiping or sliding to the right turns the endoscope to the right, bringing the anatomical area to the right into view when the endoscope camera is turned to the right. Therefore, in such implementations, a left swipe turns left, an up swipe turns up, and a down swipe turns down. In another example, the direction of movement of the distal endoscope is opposite to the direction of movement input by the user. This turning is similar to turning on a map or digital paper, and a right swipe or slide drags the entire image to the right, and when the endoscope camera turns left, the anatomical region on the left side is brought into view. In this implementation, a left swipe turns right, an up swipe turns down, and a down swipe turns up.

[0053] In one implementation, directional input may be provided additionally or alternatively via user selection from a menu, softkey selection, button pressing, joystick operation, etc. In one implementation, the user holds the controller with their left hand and touches the screen with their left thumb, thereby freeing their right hand to hold, rotate, and advance the endoscope. The user can turn the endoscope camera by tapping the screen with their thumb and then advance the endoscope by pushing forward (or pulling back) with their right hand.

[0054] The steering response to user input correlates the steering speed of the distal endoscope with the speed of the user input. Therefore, a fast swipe motion results in a faster movement of the distal endoscope compared to a slower swipe. Users can steer by selecting (tapping, swiping) a portion of the endoscope's display view as the steering destination. The steering speed for this selection process is related to the distance of the selected image portion relative to the distal endoscope. For example, steer to a point farther from the distal end than to a closer point results in a faster steering.

[0055] Although this technique is discussed in the context of endotracheal intubation, it should be understood that the disclosed technique can also be used in other types of airway management procedures or clinical procedures. For example, the disclosed technique can be used in conjunction with: placement of other devices within the airway, removal of secretions from the airway, arthroscopic surgery, visualization of the bronchus through the vocal cords (bronchoscopy), tube replacement, lung biopsy, nasal or nasotracheal intubation, etc. In some embodiments, the disclosed visualization instrument can be used to visualize anatomical structures (such as the pharynx, larynx, trachea, bronchi, stomach, esophagus, upper and lower airways, ear, nose and throat, vocal cords), or biopsies of tumors, masses, or tissues. The disclosed visualization instrument can also be used for aspiration, drug delivery, ablation, or other visualized tissue treatments, or in conjunction with aspiration, drug delivery, ablation, or other visualized tissue treatments, and can also be used in conjunction with endoscopes, probes, guides, scopes, or probes.

[0056] The description and accompanying diagrams illustrate various icons, graphic indicators, lines, outlines, shadows, marks, and visual indicators to represent the status or position of an endoscope. These different types of graphic indicators can be mixed and matched into various combinations, groupings, and styles. A shadow or line style shown by an indicator can be used with different indicators, and so on. Various combinations can be made to provide the desired indicators or animations.

[0057] While this disclosure allows for various modifications and alternatives, the accompanying drawings have illustrated specific embodiments by way of example and have been described in detail herein. However, it should be understood that the embodiments provided herein are not intended to be limited to the specific forms disclosed. Rather, various embodiments may encompass all modifications, equivalents, and alternatives falling within the spirit and scope of this disclosure as defined by the following appended claims.

Claims

1. An endoscope system comprising: an endoscope including a working channel and a steerable distal end having a camera and an orientation sensor; and a handheld controller in communication with the endoscope and including a display screen and hardware memory storing instructions for: displaying, on the display screen, a view from the camera, a working channel indicator representing a rotational orientation of the working channel of the endoscope, and a graphical orientation indicator representing an orientation of the steerable distal end of the endoscope; detecting, by the orientation sensor, rotational movement of the steerable distal end of the endoscope; maintaining a rotational orientation of the view from the camera displayed on the display screen; and modifying the graphical orientation indicator and the working channel indicator to represent the detected rotational movement of the steerable distal end.

2. The endoscope system of claim 1, wherein the graphical orientation indicator includes a roll indicator, and wherein the stored instructions further include instructions for displaying a second graphical orientation indicator including a pitch or yaw indicator.

3. The endoscope system of claim 1 or 2, wherein the graphical orientation indicator includes a horizon marker.

4. The endoscope system of claim 1 or 2, wherein the view from the camera is displayed within a circle, and wherein the graphical orientation indicator includes a visual marker along the circle.

5. The endoscope system of claim 1 or 2, wherein the graphical orientation indicator includes a gravity indicator.

6. The endoscope system of claim 1 or 2, wherein maintaining the rotational orientation of the view from the camera comprises: counter-rotating the view in proportion to the detected rotational movement of the steerable distal end of the endoscope.

7. The endoscope system of claim 1 or 2, wherein the endoscope is removable from the controller and is disposable, and wherein the controller is reusable.

8. The endoscope system of claim 1 or 2, wherein the display screen includes a touchscreen, and wherein the controller further includes instructions for steering the distal end in response to input from a user on the touchscreen.

9. The endoscope system of claim 1 or 2, wherein the handheld controller includes a video laryngoscope or a wand.

10. An endoscope system comprising: an endoscope including a working channel and a steerable distal end having a camera and an orientation sensor; a controller in communication with the endoscope and including a display screen; and a user interface rendered on the display screen, wherein the user interface includes: a window displaying a view from the camera; an orientation icon visually representing a first orientation of the steerable distal end; a working channel indicator representing a rotational orientation of the working channel of the endoscope; and a touch interface for receiving touch input from a user; wherein the controller includes hardware memory storing instructions to: detect a change in a second orientation of the steerable distal end of the endoscope, and modify the orientation icon and the working channel indicator to visually represent the second orientation.

11. The endoscope system of claim 10, wherein the orientation icon includes one or more of a roll indicator, a yaw indicator, and a pitch indicator.

12. The endoscope system of claim 10 or 11, wherein the orientation icon includes a horizon marker.

13. The endoscopic system of claim 10 or 11, wherein the steerable distal end is actively steerable within a range of motion having limits, and wherein modifying the directional icon comprises: visually changing the orientation icon when the limit is reached.

14. The endoscopic system of claim 13, wherein the controller further comprises a haptic generator, and wherein the stored instructions further comprise: providing a haptic response when the limit is reached.

Images