Multifunctional visualization instrument with orientation control

Abstract

A multifunction laryngoscope is provided that includes a handle including a proximal end and a distal end and a display screen on the handle. The laryngoscope includes a laryngoscope camera at the distal end of the handle and is connected to a guide that includes an orientation sensor at a distal end of the guide. The laryngoscope includes a processor programmed to execute instructions for receiving a steering command in a first frame of reference from a steering input and mapping the steering command to a second frame of reference oriented to the distal end of the guide based on an orientation signal from the orientation sensor.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 62 / 812,678, filed March 1, 2019, the disclosure of which is incorporated by reference in its entirety for all purposes. BACKGROUND

[0003] The present disclosure relates generally to medical devices, and more particularly to a method of controlling a steerable introducer, such as a flexible endoscope.

[0004] Introducers are long, flexible instruments that can be introduced into a patient’s cavity in a medical procedure in various situations. For example, one type of introducer is a flexible endoscope with a camera at a distal end. The endoscope can be inserted into a patient’s mouth or throat or other cavity to help visualize the anatomy, or to help perform a procedure such as a biopsy or ablation. Another type of introducer is a blind stylet (without a camera) that can be inserted and then used to guide another device, such as an endotracheal tube, into place. These and other introducers can include a steerable distal tip that can be actively controlled to bend or turn the distal tip in a desired direction, to obtain a desired view or to navigate through the anatomy. However, these steerable introducers can be difficult to maneuver to a desired position and orientation within a patient’s anatomy. SUMMARY

[0005] Certain aspects or embodiments commensurate in scope with the originally claimed subject matter are summarized as follows. These aspects or embodiments are not intended to limit the scope of the present disclosure. Indeed, the present disclosure can encompass a variety of forms that can be similar or different from the aspects set forth below.

[0006] In one aspect or embodiment, a steerable introducer system including a laryngoscope and an introducer is provided. The laryngoscope includes a handle including a proximal end and a distal end, a display screen on the handle, and a laryngoscope camera at the distal end of the handle. The laryngoscope also includes a steering input for steering the introducer, the steering input being located on the handle or the display screen. The introducer is coupled to the handle and has an orientation sensor at a distal end of the introducer. The laryngoscope also includes a processor within the laryngoscope programmed to execute instructions for receiving a steering command in a first frame of reference from the steering input and mapping the steering command from the first frame of reference to a second frame of reference oriented to the distal end of the introducer based on an orientation signal from the orientation sensor.

[0007] The processor can be further programmed to execute instructions for generating a control signal for steering the guide in accordance with the mapped steering command. The second reference frame can be defined by an angular offset from the first reference frame. Mapping the steering command to the second reference frame can include adjusting the steering command by the angular offset. The first reference frame can be defined by user input. The first reference frame can be defined by automatic image recognition. The processor can be programmed to receive an image from the laryngoscope camera to identify features of the image to perform the automatic image recognition. The processor can be programmed to receive an image from a guide camera at the distal end of the guide to identify features of the image to perform the automatic image recognition. The display screen can display the image from the laryngoscope camera in the first reference frame. The display screen can display the image from the guide camera at the distal end of the guide in the first reference frame.

[0008] In yet another aspect or embodiment, which can be provided independently, an endoscope controller is provided that includes a handle, a display screen on the handle, an endoscope port located on the handle or the display screen, and a user input located on the handle or the display screen. A processor within the controller is programmed to execute instructions for receiving a steering command in a user reference frame from the user input, receiving an orientation signal from an orientation sensor at a distal end of an endoscope coupled to the endoscope port from the endoscope, and converting the steering command in accordance with the orientation signal.

[0009] The processor can be further programmed to execute instructions for steering the endoscope in accordance with the converted steering command. The controller can further include an endoscope coupled to the endoscope port, where the endoscope can include an orientation sensor that generates the orientation signal.

[0010] In another aspect or embodiment, which can be provided independently, a method for controlling a steerable guide includes receiving, at a processor, an orientation signal from an orientation sensor located at a distal end of a steerable guide. The orientation signal defines an angular orientation of the distal end of the guide. The method further includes receiving, at the processor, a steering command that includes a steering direction in a user reference frame; converting the steering command from the user reference frame to the angular orientation of the distal end of the guide; and steering the distal end of the guide in accordance with the converted steering command.

[0011] The user reference frame can be defined with reference to an anatomical feature of a patient. The user reference frame can be defined by user input. The method can further include receiving, at the processor, an image from a camera at the distal end of the guide; rotating the image into the user reference frame; and displaying the rotated image at a display screen.

[0012] In another independently provided aspect or embodiment, a method for controlling a steerable introducer includes receiving, at a processor, a steering command from a user input and an orientation signal from an orientation sensor of the steerable introducer. The method also includes converting, at the processor, the steering command as a function of the orientation signal, and steering the introducer as a function of the converted steering command.

[0013] In yet another independently provided aspect or embodiment, a method for controlling a steerable introducer includes receiving, at a processor, a steering command from a user input and an orientation input from an orientation sensor. The method also includes generating, at the processor, a variable steering signal comprising steering instructions that vary as a function of both the steering command and the orientation input, and steering the introducer as a function of the variable steering signal.

[0014] In another independently provided aspect or embodiment, a method includes receiving, at a processor, a laryngoscope image from a laryngoscope camera; receiving, at the processor, an endoscope image from an endoscope camera at a distal end of an endoscope and an orientation signal from an orientation sensor at the distal end of the endoscope; receiving a user input to establish a reference frame for the distal end; receiving an update signal from the orientation sensor indicating that the distal end has rotated away from the reference frame; and rotating an updated endoscope image into the reference frame based on the updated signal.

[0015] Features in one aspect or embodiment can be applied as features in any other aspect or embodiment in any suitable combination. For example, any of the system, laryngoscope, controller, introducer, or method features can be applied as any one or more of the system, laryngoscope, controller, introducer, or method features. BRIEF DESCRIPTION OF DRAWINGS

[0016] Advantages of the disclosed technology can become apparent to those skilled in the art with the benefit of the following description and drawings. It is to be understood that the disclosed technology is not limited to the specific embodiments there described.

[0017] Figure 1 is a perspective view of a multifunctional controller and steerable introducer of a steerable introducer system according to certain embodiments of the present disclosure.

[0018] Figure 2 is a perspective view of a visualization wand and steerable introducer of a steerable introducer system according to certain embodiments of the present disclosure.

[0019] FIG. 3A is a schematic diagram of an image frame associated with a first introducer orientation according to certain embodiments of the present disclosure.

[0020] FIG. 3B is a schematic diagram of an image frame associated with a second introducer orientation according to certain embodiments of the present disclosure.

[0021] FIG. 3B is a schematic diagram of an image frame associated with a second introducer orientation according to certain embodiments of the present disclosure.

[0021] Figure 4 is a system diagram of a controller and a steerable introducer according to certain embodiments of the present disclosure.

[0022] Figure 5 is a cross-sectional top view of a distal end of a steerable introducer according to certain embodiments of the present disclosure.

[0023] Figure 6A is a diagram of an image frame associated with a first introducer orientation according to certain embodiments of the present disclosure.

[0024] Figure 6B is a diagram of an image frame associated with a second introducer orientation according to certain embodiments of the present disclosure.

[0025] Figure 6C is a diagram of an image frame associated with a third introducer orientation according to certain embodiments of the present disclosure.

[0026] Figure 7 is a flowchart of a method for steering an introducer according to certain embodiments of the present disclosure.

[0027] Figure 8 is a flowchart of a method for steering an introducer according to certain embodiments of the present disclosure.

[0028] Figure 9 is a flowchart of a method for adjusting an introducer orientation to a reference frame according to certain embodiments of the present disclosure. DETAILED DESCRIPTION

[0029] One or more specific embodiments of the present technology will be described below. According to embodiments, a system for navigating a patient’s anatomy with a steerable introducer and for adjusting steering commands according to the orientation of the introducer is provided. As the introducer is navigated into the patient, a user can rotate or turn the distal tip of the introducer in order to maneuver through the patient’s anatomy or to obtain a desired view. When the introducer is rotated or turned multiple times during a procedure, it is difficult for the user to keep track of the changing orientation of the distal end of the introducer. Subsequently, the user can inadvertently bend or turn the introducer in the wrong direction. For example, the user can want to turn the introducer to the user’s right, but because the introducer is rotated from its default position, the result of this command is to bend the introducer to the user’s left.

[0030] The disclosed embodiments use the guide's orientation information to illustrate the difference between the orientation of the guide's distal end and the user's own reference frame. Therefore, the guide steering system using orientation information provides a more intuitive view of the image captured by the guide and / or a more intuitive view of the guide's distal end's orientation within the handle. Furthermore, because the orientation information is not obtained from a handheld device manipulated by the operator, changes in the position or angle of the handheld device during use do not result in inaccurate orientation information.

[0031] Therefore, in embodiments, the guide steering system translates steering commands from user reference frames into guide orientation to maintain the user's intention to steer the guide. Embodiments of the steerable guide system are as follows: Figure 1 The system is shown in the diagram. It includes a video laryngoscope 10 and a steerable guide 12. The guide is a thin, elongated, flexible instrument (which can be relatively narrower, more flexible, and longer than a laryngoscope or endotracheal tube) that can be inserted into the handle cavity for exploration, imaging, biopsy, or other clinical treatments. It includes a catheter, endoscope (with a camera), blind probe (without a camera), or other types of endoscopes or probes. The guide can be positioned to extend into the airway and steered by the user into an airway pathway (such as the pharynx, larynx, trachea, or bronchus) by advancing the distal end to the desired position and, in some embodiments, subsequently rotating or repositioning the guide. The guide can be tubular in shape.

[0032] Guide 12 includes a proximal end 14 (closest to the user) and an opposing distal end 16 (closest to the patient), and in this example, a camera 18 is located at the distal end for viewing the patient's anatomy. Guide 12 includes a distally steerable portion 20 that can be bent, twisted, rotated, or rotated. The distally steerable portion 20 can move in two-dimensional (planar) or three-dimensional space. The distally steerable portion 20 is steered by a steering system. The steering system may include one or more shape-memory metal components (e.g., memory wire, nitinol wire) that change shape based on electrical input, piezoelectric actuators (such as SQUIGGLE motors from New Scale Technologies, Victor NY), retractable sheaths (retractable to release pre-shaped bent components, such as spring steel that regains its bent shape when released from the sheath), mechanical control lines, hydraulic actuators, servo motors, or other means for bending, rotating, or rotatating the distal end or components at the distal end of the guide.

[0033] The proximal end 14 of the guide 12 is connected to a controller, which may be a reusable or single-use disposable handle 22, or a multipurpose medical device such as a video laryngoscope 10. The video laryngoscope 10 includes a handle 30 having a proximal end 32 and a distal end 34. The handle 30 includes a display screen 36 mounted proximal to the handle or handle 38.

[0034] The controller operates the steering system to steer the steerable portion 20 of the introducer, and includes a user input 24 to receive steering commands from a user. As shown, the user input 24 can include a handle 22 or a button on the video laryngoscope 10. The user presses the button to indicate a direction to turn or steer the introducer. The user input 24 can be located on the display screen 36, on the handle 38, or both. The user input 24 can be one or more physical buttons (or switches, levers, joysticks, or similar inputs), touch-sensitive graphics or icons on a touch screen (such as on the screen 36), a keyboard, or other suitable user input. Figure 1

[0035] As shown, the video laryngoscope includes a camera stem 40 extending from the distal end 34 of the handle 30. The camera stem 40 includes an elongated arm 42 carrying a camera 44 at its distal end. The camera stem 40 is mounted within a removable, disposable, transparent blade 46. More information about laryngoscope blades can be found, for example, in Applicant’s U.S. Patent Nos. 9,775,505 and 9,066,700. Images from the video laryngoscope camera 44 and / or from the introducer camera 18 (if present) are displayed on the display screen 36. Figure 1

[0036] In embodiments, as shown, the steerable introducer 12 includes an orientation sensor 56 at the distal tip of the introducer. The orientation sensor 56 can be an inertial measurement unit (IMU), an accelerometer, a gyroscope, or other suitable sensor. The orientation sensor 56 is located within the tubular housing of the introducer 12. In embodiments, the orientation sensor 56 is located at the very tip of the introducer, very close to the distal end 16, and can be collocated with the camera 18 (if present) to enable the orientation sensor 56 to capture most of the range of motion of the distal end 16 and the camera 18. In embodiments, the orientation sensor 56 is placed (e.g., positioned or located) at the distal end 16 of the steerable portion 20, away from the proximal end of the steerable portion 20, to place the orientation sensor 56 away from the fulcrum of movement of the distal end 16 and the camera 18. Figure 1

[0037] ​​​The disclosed embodiments of orientation sensor 56 contained at or near distal end 16 of introducer 12 provide more accurate orientation information relative to embodiments in which orientation information is derived from orientation sensors in a controller, such as a video laryngoscope, a wand, or a handle. In such instances, information derived from sensors located in the controller relies on the assumption that the orientation of the controller is the same as the orientation of the distal tip. To maintain the conditions of this assumption, a user can be instructed to hold the controller at a particular angle or position during operation. However, user variability in positioning of the controller during operation can result in inaccurate orientation information being reported. Thus, orientation information measured at a hand-held device located at the proximal end of the introducer can not provide accurate information. Furthermore, movements measured at the controller cannot be translated into corresponding movements of the distal tip. For example, a handle of an introducer can have some degree of compliance, so a rotation by a user at the proximal end is not fully transmitted along the length of the introducer. As another example, along a tortuous path through patient anatomy, twisting and friction can create a loss of rotation. In the embodiments disclosed herein, orientation sensor 56 positioned at or near distal end 16 of introducer 12 provides more accurate orientation information than orientation measurements based on the controller.

[0038] As provided in the disclosed embodiments, the precise orientation information captured at or near the distal end of the introducer 12 allows for dynamic image adjustment, providing a more intuitive visualization of the introducer image, and in turn, a more intuitive steering within an established frame of reference, which can be oriented to a gravity or user-defined frame of reference. Further, the introducer is steered at the distal end 16 without physical rotation of the proximal end, rather than embodiments in which distal rotation and orientation changes are driven by twisting forces transmitted from the proximal end 14 to the distal end 14. These introducers use a steering system that is effective in bending the distal tip in a desired direction at the distal tip (such as a push or pull wire) even when the length of the introducer between the proximal and distal ends is slack; the introducer does not require twisting forces to be transmitted along the introducer housing from the proximal end to the distal end. The introducer does not need to be straight or taught in order to transmit steering inputs to the distal end. Distal end bending and movement of the introducer is achieved independent of the orientation, position, or movement of the proximal end of the introducer; steering is not physically coupled between the proximal end (such as a handle) and the distal end. Further, the introducer system does not make any assumptions about how much twisting force is successfully transmitted (or lost) along the length from the proximal end to the distal end; rather, orientation sensors at the distal tip provide orientation signals indicative of the current orientation of the distal tip. In this way, the structure of the introducer 12 can have less torsional stiffness relative to embodiments in which steering relies on twisting force transmission. Thus, in embodiments, the introducer 12 is an extruded structure with low torsional stiffness (low enough that torsional rotation does not transmit from the proximal end to the distal end). In embodiments, the introducer is a non-woven structure such as an extruded polymer. In embodiments, the introducer is an extruded structure without torsional reinforcements such as woven wires or braided structures.

[0039] Figure 2 Another embodiment is shown in which the controller is a stick 50 similar to the video laryngoscope 10 but without the camera stem 40. The stick 50 contains the user input 24 that receives steering commands from the user, and contains the display screen 36. As Figure 1 and Figure 2As shown, the controller can take the form of a handle 24, a video laryngoscope 10, or a wand 50 with an integrated display screen 36. The controller can also take the form of a separate (unintegrated) touchscreen display located in the room (such as mounted on a cart or stand), spaced apart from the introducer. This touchscreen transmits user input to the introducer via a wired or wireless connection. In one embodiment, the handle 24 is integral with the tubular introducer 12, and the entire device is single-use and disposable. In another embodiment, the introducer is a two-part system, and the controller (handle, wand, laryngoscope, or other device) can be removed from the introducer 12. The introducer 12 is then discarded after use, and the controller is retained and used again with a new tubular introducer. The controller contains power, display, steering control, and other functions. In this way, the endoscope introducer can be disposable, while the relatively more expensive and complex controller can be reused.

[0040] The introducer 12 can be attached to the wand 50 from the top (proximal) end of the wand, so that the introducer extends upward beyond the top of the screen, or from the bottom (distal) end of the wand, so that the introducer extends downward away from the bottom of the screen. Figure 2 The introducer 12A in FIG. 2A is shown indicating the option to connect the introducer to the wand 50 from below the screen.

[0041] FIGS. 3A and 3B depict a method of steering an introducer, which includes translating steering commands from a user into executable actuator controls within the orientation of the introducer. For example, in FIGS. 3A-3B, the introducer is a tubular endoscope 120, with a camera 118 at its distal end 116. The endoscope 120 also has a feature located at one angular position around the tubular endoscope, such as an orientation indicator, working channel, surgical tool, light source, or other instrument. In FIGS. 3A-3B, this feature is an orientation marker 126, which is a visual sign or mark that indicates to the user the direction of the steering control upward. The marker 126 can be formed by a printed graphic, a groove or other three-dimensional feature, ink or a mark that glows in the dark, or an actively powered light such as a small LED light bar or lamp. When the endoscope is in its default resting position (not bent, twisted, or steered), the marker 126 is located on the top side of the endoscope 120. In image A, the endoscope has been rotated 180 degrees from this position, so that the marker 126 is located on the bottom of the endoscope.

[0042] Real-time images from the camera are displayed on a display screen 136, which can be a bar display, a video laryngoscope, a monitor, or any other display screen in a medical facility. Images from camera 118 can be transmitted to display screen 136 via a wired connection or wirelessly. In Figure 3A, the field of view of the endoscopic camera includes anatomical structures 152 within channel 154. In one example, channel 154 is the trachea, and structure 152 is a tumor. In other cases, channel 154 is a gastrointestinal passage, a nasal tube, or any other anatomical cavity. Structure 152 can be a polyp, tumor, blood vessel, vocal cord, suture, stent, bifurcation of the channel (such as a bronchial passage or carina), or any other visible anatomical or medical feature.

[0043] In Figure 3A, structure 152 appears facing the top of display screen 136. The user can decide to turn endoscope 120 toward structure 152 and give an "up" turning command (such as via user input 24). The user's turning command is based on the user's reference frame, such as the orientation in the image on the display screen. However, in this case, the user's intention to turn "up" differs from the endoscope's default stationary "up" orientation. The orientation of endoscope 120 has been changed relative to the user's reference frame.

[0044] Therefore, in this embodiment, the endoscope steering system translates the user's command into the current orientation of the endoscope. In Figure 3B, the user provides an "up" steering command, which means bending "up" in the reference frame of display screen 136. The steering system translates this for the endoscope as follows: the endoscope bends in the direction opposite to marker 126 (i.e., the "down" direction in the endoscope's default reference frame). As shown in Figure 3B, the endoscope bends toward structure 152, and structure 152 moves to the center of screen 136.

[0045] A schematic cross-sectional view of the distal end 16 of the guide 12 in Figure 5 The figure shows a camera 18 located at the end 17 of the distal end 16 of the guide 12 for a clear forward view. An orientation sensor 56 is located directly behind the camera 18. In one embodiment, the orientation sensor 56 is adjacent to the camera 18. In another embodiment, the orientation sensor 56 is mounted on flexible circuitry behind the camera 18. In yet another embodiment, the orientation sensor 56 is mounted on the same flexible circuitry as the camera 18, although the orientation sensor 56 and the camera 18 do not need to communicate on the shared flexible circuitry. In one embodiment, the orientation sensor has a size between 1 and 2 mm in each dimension. It should be understood that in some embodiments, the guide 12 is blind and the camera 18 is absent.

[0046] An orientation sensor is an electronic component that senses the orientation or movement of the distal end of a guide. The orientation sensor incorporates sensors, such as accelerometers, magnetometers, and gyroscopes, or combinations thereof, to achieve this. The orientation sensor detects the position and / or movement of the distal end of the guide and provides a signal indicating a change in the guide's orientation. An orientation sensor 156 is also illustrated in Figures 3A and 3B, located at the distal end 116 of the guide 120, directly behind the camera 118. In this embodiment, the signal from the orientation sensor is based solely on the accelerometer (without using other sensors such as gyroscopes or magnetometers). In this embodiment, an accelerometer is used as the orientation sensor.

[0047] A schematic diagram of the electrical components of the steerable guidance system is shown in Figure 4 As shown in the diagram. In this embodiment, the system includes a controller 210 (such as a video laryngoscope, handle, or rod) and a guide 212. The controller 210 includes a microprocessor 260, a memory 261, a power supply 262, a display screen 236, user inputs 224, and associated circuitry 263 (such as a wireless transceiver for receiving and transmitting data). When the controller is a video laryngoscope, it also includes a camera and a light source, as well as other components. The guide 212 includes a camera 218 (if present), a light source 264, an orientation sensor 256, and a steering system 265.

[0048] like Figure 4 As depicted, orientation signal 266 is transmitted from guide 212 (based on measurements from orientation sensor 256) to controller 210, and actuation control signal 268 is transmitted from controller 210 to guide 212. The orientation signal can be generated by an orientation sensor located at the distal end of the guide. The orientation signal defines the angular orientation of the distal end of the guide relative to gravity.

[0049] Orientation signal 266 and steering command from user input 224 are sent to processor 260, which converts the steering command into actuation control signal 268. Actuation control signal 268 operates the steering system by including specific executable instructions on the guide for the various actuators of steering system 265 to bend, twist, or move the steerable portion 20 of the guide.

[0050] A method 700 for controlling a steerable guide according to an embodiment is in... Figure 7 Depicted in block 701. The method includes receiving a guide orientation signal from an orientation sensor (in block 701). For example, the signal may be received from an IMU or accelerometer or other sensor according to... Figure 4The method also includes receiving a user reference frame from a user input (at block 702). The method also includes converting the steering command from the user reference frame to a guide orientation (at block 703). The method also includes steering the guide according to the converted steering command (at block 704). These steps can be accomplished by a processor located within a guide controller, such as a laryngoscope, a stick, or a handle, such as processor 260.

[0051] The user reference frame is the frame in which the user gives steering directions. This reference frame can be aligned with the direction of gravity (so that a "down" steering command means down towards the earth). As another example, the reference frame can be aligned with an image on a display screen (so that a "down" steering command means down in the image). As another example, the reference frame can be centered on the patient (so that if the patient is lying on their side, a "down" steering command means towards the patient's back, or towards some other anatomical feature of the patient). These are just a few examples.

[0052] Another example method 800 is outlined in Figure 8 The method also includes generating a variable actuation control signal according to both the steering command and the orientation signal (at block 802). The method also includes steering the guide according to the variable actuation control signal (at block 803). This can be accomplished, for example, by a processor that generates an actuator control signal with specific instructions to operate the actuators of the steering system of the guide to move the guide in a user-specified direction.

[0053] In this way, the actuation control of the steering system is not tied to the internal reference frame of the guide. Rather, the steering applied to the guide varies with the orientation of the guide. The same steering command from the user reference frame (e.g., "up" towards the top of the display screen) will be converted to different actuator controls depending on how the guide is oriented. Even with the same steering command from the user, the control signal sent to the actuators of the steering control system of the guide will vary with the orientation of the guide. For example, when the user inputs a command to bend "up" towards the top of the display screen, the steering control system can bend the guide towards an orientation marker, such as 326, or away from an orientation marker, depending on how the guide is oriented. Thus, the control signal to operate the steering control system of the guide varies with the orientation of the guide and with the steering command of the user.

[0054] In embodiments, the steering system includes two, three, four, or more actuators that control movement of the steerable tip of the guide. In embodiments, steering actuation is achieved by modeling the tip of the guide as a circle, with the modeled actuators occupying discrete positions around the circumference of the circle. At these positions, the actuators act on the tip to bend or move the guide. The circle is rotated according to orientation signals from the orientation sensor to indicate the orientation of the guide relative to a user-defined frame of reference. Thus, when a user steering command is received (e.g., bend "up" toward the top of the circle), the appropriate action for each respective actuator can be determined. Each actuator is operated or energized proportionally according to its position on the circle relative to the user command. It will be appreciated that two or more actuators can be located at any position in the guide and associated with respective modeled circumferential positions.

[0055] In embodiments, the user can define a custom frame of reference, e.g., as shown in Figures 6A to 6C which illustrates a display screen 336 of a video laryngoscope displaying two images, a first image 370 from a camera on the video laryngoscope (such as camera 44 in Figure 1 ) and a second image 372 from a camera on the endoscope 312 (such as camera 18 in Figure 1 ). As shown in Figures 6A to 6C , the endoscope 312 is within the field of view of the laryngoscope camera, so the endoscope 312 is visible in image 370. The endoscope includes an orientation marker 326 visible on the surface of the guide 312. Figure 6A The lower panel of is a schematic representation of a cross-section of the endoscope, with the orientation marker 326 shown in the upper left position of the guide 312.

[0056] Figure 6A The patient's vocal cords 374 and trachea 376 are visible in the images on screen 336. However, the endoscope image 372 is rotated counterclockwise compared to the video laryngoscope image 370. Thus, the user can decide to manually rotate the endoscope to transition from the position in Figure 6B to the position shown in Figure 6B In , the user has rotated the endoscope clockwise by an angle Figure 6BThe endoscope is then established as a desired reference orientation or frame. This can be done by pressing user input 24, a button on the touchscreen, or other input methods. The controller then stores the endoscope's current orientation at the time of user input as a reference frame for future adjustments. Subsequently, when the user gives a steering command (such as up, down, turn, etc.), these commands are interpreted within this stored reference frame, and the endoscope's orientation data is translated into endoscope movement. This allows the user to decide which reference frame to use for steering commands. For example, steering can be oriented towards the patient's handle instead of gravity. While alignment with laryngoscope image 370 is shown as an example, the user can select any other orientation to establish a reference frame.

[0057] In Figure 6B After the position in the image is established as the desired reference orientation, the system will adjust the steering and image to this reference orientation. For example, in Figure 6C In the middle, the user has further rotated away. Figure 6B The position shown indicates that the guide has been rotated clockwise by an angle α. As shown by the new position of orientation mark 326 in laryngoscope image 370, the guide itself has been rotated. However, the second image 372 (from the guide) has not been rotated. Figure 6C In the middle, the vocal cords and trachea remain upright, as if they were in... Figure 6B The orientation is the same. The system determines the amount of change (here, the amount of clockwise rotation angle α) by receiving information from an orientation sensor at the distal end of the guide, and reverses this movement to keep image 372 aligned with the orientation. Figure 6B This is achieved through the same approach. Similarly, as mentioned above, in Figure 6B or Figure 6C Steering control based on user input Figure 6B This can be explained by the user's orientation. If the user instructs... Figure 6C If the guide in the middle is turned "up" toward the top of the screen 336, the system will bend the guide in this direction, even if the orientation mark 326 is rotated by an angle α from this position.

[0058] In another embodiment, a reference frame can be established using automatic image recognition. For example, returning to... Figures 6A to 6CThe processor on the controller can automatically identify features in images, such as the vocal cords 376 in images 370 and 372, based on computer vision techniques. These techniques can include, for example, a single-shot object detector (which can identify anatomical structures), a cascaded classifier based on Haar features (to identify anatomical structures), a neural network trained to output orientations based on known anatomical structures, landmark alignment with a set of regression trees, object tracking once useful features are identified, or other computer vision techniques. The processor can then build a reference frame based on the orientation of the vocal cords, for example, identifying "upward" as facing the top of the vocal cords (such as facing the epiglottis 378). The processor can be programmed to identify other anatomical structures (e.g., the cross-sectional shape of the trachea, anterior-posterior localization) and update or store the reference frame based on these structures. Image recognition can help align the user's reference frame with the patient's anatomy, rather than relying on gravity.

[0059] In this embodiment, the user can switch between dual-screen or picture-in-picture displays (e.g., Figures 6A to 6C The image shown is displayed only for the guide (image 372 only) or only for the laryngoscope (image 370 only), and vice versa. The reference frame can be automatically adjusted based on the type of image or the image displayed. For example, the alignment of the reference frame can be based on the orientation of the laryngoscope. Typically, the laryngoscope is positioned during use such that the image captured by the laryngoscope camera is oriented relative to gravity, where the top of the image on the display is typically “up” relative to gravity. However, some procedures may involve different laryngoscope positioning relative to the patient, such as when the user is facing the patient and the laryngoscope is rotated 180 degrees. In this case, the top of the laryngoscope image displayed on the display actually corresponds to a “down” direction relative to gravity. To illustrate different positioning or alignment of the laryngoscope relative to gravity, the alignment can be based on the alignment with the laryngoscope image, which may or may not be aligned with gravity. However, when the display mode is changed to guide-only display, the reference frame can automatically switch to a gravity-based alignment, determined by an orientation signal from an orientation sensor. Furthermore, in embodiments, the technology can be used to establish a reference frame for steering commands when the guide is blind (e.g., a blind probe) and no camera image is displayed. However, steering commands can be converted to a gravity-based or user-established reference frame, and the conversion can be performed using orientation signal information from an orientation sensor.

[0060] exist Figure 6B The processor can also determine the rotation angle θ of the endoscope image 372 relative to the video laryngoscope image 370. In an embodiment, the processor corrects the endoscope image 372 by rotating it to align it with the video laryngoscope image 370, even without rotating the actual endoscope. This step keeps the two images aligned, making it easier for the user to view them simultaneously.

[0061] In embodiments, an orientation signal 266 Figure 4 is used to adjust the displayed endoscopic image (such as image 372, or on any other display screen). The processor 260 can use the signal 266 to automatically adjust the displayed image to a desired orientation, such as adjusting the image to ensure that the upward direction (forward, toward the patient's chest) remains upward (toward the top proximal surface) on the display screen, even when the endoscope is rotated or turned inside the patient. For example, as shown in Figure 6C , the user can rotate the endoscope clockwise (or any amount), for example, in order to better position the endoscope within the patient's anatomy. In Figure 6C , the image on the display screen remains stationary even when the endoscope is rotated. The orientation sensor 256 at the tip or distal end of the endoscope records the rotation, and the microprocessor 260 rotates the image on the screen in the opposite direction (counterclockwise in this example) by the same amount. If the endoscope is rotated again in either direction, the microprocessor again compensates so that the image on the screen remains oriented with the patient's front pointing upward on the display screen. In another embodiment, the microprocessor 260 receives real-time updated signals from the orientation sensor 256 indicating the relationship between the distal tip and gravity, so that the microprocessor can continuously adjust the image to keep the direction of gravity pointing down on the laryngoscope display, even when the endoscope itself is rotated.

[0062] An example method 900 is outlined in Figure 9 , which can be used in conjunction with a picture-in-picture display or dual image display of a multi-functional visualization instrument with steering control (e.g., video laryngoscope 10, see Figure 1 ). In this example, the method includes displaying an image from an endoscope camera of an endoscope (e.g., image 372, see Figures 6A to 6C ) on a display screen (block 902). Optionally, the method can also display a first video laryngoscope image from a laryngoscope camera of a video laryngoscope (e.g., image 370, see Figures 6A to 6C). The user can define a custom reference orientation or frame via user input (block 904), or alternatively, the system can automatically establish a reference frame based on gravity or image processing. The orientation of the endoscope at the time of user input is established as the reference frame (block 906). That is, when using user input to define the reference frame, the orientation of the endoscope at the time of user input is tagged or stored as the reference frame orientation. The orientation sensor subsequently provides a current orientation signal that indicates that the endoscope distal end containing the endoscope camera has a different orientation than the reference frame (block 908). For example, the current orientation of the distal end can change due to user manipulation or a steering event to move (e.g., rotate) from the orientation associated with the reference frame to the current orientation. Accordingly, subsequent or second endoscope images captured at the current orientation are transformed (e.g., modified, rotated) from the current orientation to the reference frame (block 910). In embodiments, any received steering commands received at the updated orientation (block 912) are transformed from the updated orientation to the reference frame (block 914) based on the amount and direction of rotation to facilitate steering the endoscope according to the transformed steering commands (block 916).

[0063] The user can also update the reference orientation throughout the procedure. For example, the steps outlined in Figure 9 may be repeated to enable the user to establish a new reference orientation. For example, if the patient moves, rotates, sits up or lies down, coughs, etc., the clinical user can decide to establish a new reference orientation for the scope so that the system will rotate the image information from the scope to keep the image stationary in this reference orientation and transform steering commands from the user to the scope. In embodiments, the system establishes an automatic or default orientation (such as gravity down) and the user can override or change this default orientation by establishing a new reference orientation as outlined in Figure 9 .

[0064] A guide with variable steering can be used to assist with endotracheal intubation. During endotracheal intubation, a clinician, such as an anesthesiologist or other medical professional, attempts to navigate an endotracheal tube through a limited field of view of a patient’s mouth. The clinician can rely on the relative positions of anatomical structures to navigate. During intubation, the cricoid cartilage has proven useful as an anatomical landmark; the vocal cords are anterior to the cricoid cartilage, and the esophagus is posterior. In embodiments of the present disclosure, the anterior direction is aligned with the top of the user’s display screen and is set as the reference orientation, such that the anterior direction remains “up” on the screen. During intubation, the user can input a command to steer the guide “up” to pass the tip over the cricoid cartilage and into the vocal cords. The user can then pass the endotracheal tube through the guide and ensure that the endotracheal tube enters the trachea and not the esophagus. Conversely, if the user becomes disoriented and inadvertently steers the guide into the esophagus (instead of the trachea), an esophageal intubation results, causing serious complications for the patient. Thus, a system in which the user’s orientation is maintained and steering inputs are translated accordingly can improve clinical practice.

[0065] While the present technology is discussed in the context of endotracheal intubation, it should be understood that the disclosed technology can also be used for other types of airway management or clinical procedures. For example, the disclosed technology can be used in conjunction with airway secretion removal, arthroscopic procedures, bronchoscopic visualization, tube exchange, lung biopsy, nasal or nasotracheal intubation, etc. In certain embodiments, the disclosed multi-function visualization instrument can be used for visualization of anatomical structures (stomach, esophagus, upper and lower airways, ear nose throat, vocal cords), or biopsy of tumors, masses, or tissue. The disclosed multi-function visualization instrument can also be used for or in conjunction with suction, drug delivery, ablation, or other treatment of visualized tissue. The disclosed multi-function visualization instrument can also be used in conjunction with an endoscope, stylet, guide, scope, or probe.

[0066] In operation, a laryngoscope can be used by a caregiver to assist with intubation, e.g., to visualize a patient’s airway to guide a distal tip of an endotracheal tube through the patient’s mouth, through the vocal cords, and into the tracheal passageway. Visualizing the patient’s anatomy during intubation can help the medical caregiver avoid damaging or irritating the patient’s oral and tracheal tissue, and avoid passing the endotracheal tube into the esophagus instead of the trachea. The laryngoscope can be operated with one hand, such as the user’s left hand, while the other hand, such as the right hand, grasps the endotracheal tube and guides it forward into the patient’s airway. The user can watch the advancement of the endotracheal tube on a display screen in order to guide the endotracheal tube to its proper location.

[0067] While video laryngoscopy can facilitate intubation more efficiently than direct vision intubation, certain patients can benefit from visualization and / or steering devices that provide deeper visualization of the airway than laryngoscopy. For example, patients with smoke inhalation, burns, lung cancer, and / or airway trauma can benefit from visualization past the vocal cords that is not achieved with a laryngoscope. Such visualization can be beneficial for endoscopic placement of an endotracheal tube and / or placement or positioning of a suction device in the airway. Endoscopic placement (e.g., loading an endotracheal tube into an endoscope) can be helpful for anterior airways or challenging airways. For example, patients whose anatomy cannot be properly manipulated (with head positioning or laryngoscopy) to create space for passage of an endotracheal tube can benefit from an imaging device that provides visualization beyond the range of a laryngoscope and provides a camera with a greater range of steering, or from an articulating device that can be manipulated and moved within the range of visualization of a laryngoscope.

[0068] While the disclosure can admit to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and have been described in detail herein. It should be understood, however, that the intention is not to limit the embodiments provided herein to the particular forms disclosed. On the contrary, the various embodiments can cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure as defined by the appended claims.

Claims

1. A steerable guidance system, comprising: Laryngoscope, the laryngoscope comprising: A handle, the handle including a proximal end and a distal end; The display screen on the handle; A laryngoscope camera at the distal end of the handle; The processor within the laryngoscope; and Steering input located on the handle or the display screen; and A guide, the guide being coupled to the handle and comprising: Guide camera; The distal steerable portion is capable of steering based on the steering input; An orientation sensor is located within the distal steerable portion of the guide. The processor within the laryngoscope is programmed to execute instructions for: An orientation signal is received from the orientation sensor, the orientation signal indicating the orientation of the distal end of the guide; and The steering input receives a steering command from a first reference frame, which is defined by user input or by automatic image recognition. The steering command is mapped from the first reference frame to a second reference frame based on the orientation signal, and the second reference frame is oriented to the distal end of the guide.

2. The steerable guide system according to claim 1, wherein, The processor is further programmed to execute instructions for generating control signals for steering the guide according to a mapped steering command.

3. The steerable guide system according to claim 1, wherein, The second reference frame is defined by an angular offset relative to the first reference frame.

4. The steerable guide system according to claim 3, wherein, Mapping the steering command to the second reference frame includes adjusting the steering command by means of the angle offset.

5. The steerable guide system according to claim 1, wherein, The processor is programmed to receive images from the laryngoscope camera to identify features of the images, thereby performing the automatic image recognition.

6. The steerable guide system according to claim 1, wherein, The processor is programmed to receive images from the guide camera at the distal end of the guide to identify features of the images, thereby performing the automatic image recognition.

7. The steerable guide system according to any of the preceding claims, wherein, The display screen shows an image from the laryngoscope camera in the first reference frame.

8. The steerable guide system according to claim 7, wherein, The display screen shows an image from the guide camera at the distal end of the guide in the first reference frame.

Images