Endoscopy ecosystem voice control methods and systems

Abstract

A endoscopy ecosystem control system comprises an endoscopy system and a peripheral system comprising modules for controlling capabilities of the endoscopy ecosystem provided by the endoscopy system and the peripheral system, and the control system is configured to receive a voice command from a user to enable a desired functionality of the endoscopy ecosystem, decode the voice command into a system instruction for controlling the desired functionality, parse the system instruction into a first sub-command for a first identified module of the modules for controlling a first aspect of the desired functionality and a second sub-command for a second identified module of the modules for controlling a second aspect the desired functionality, and transmit the first sub-command to the first identified module and the second sub-command to the second identified module, wherein transmission of the first sub-command and the second sub-command cause the endoscopy ecosystem to execute the desired functionality.

Description

ENDOSCOPY ECOSYSTEM VOICE CONTROL METHODS AND SYSTEMSPRIORITY CLAIM

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 709,337, filed October 18, 2024, the contents of which are incorporated herein by reference.TECHNICAL FIELD

[0002] The present disclosure is generally directed to devices, systems and methods for controlling endoscopy ecosystems. More specifically, the present disclosure is directed to providing voice control operations for endoscopy suites.BACKGROUND

[0003] Endoscopes can be used for one or more of 1) providing passage of other devices, e.g., therapeutic devices or tissue collection devices, toward various anatomical portions, and 2) imaging of such anatomical portions. Such anatomical portions can include gastrointestinal tract (e.g., esophagus, stomach, duodenum, pancreaticobiliary duct, intestines, colon, and the like), renal area (e.g., kidney(s), ureter, bladder, urethra) and other internal organs (e.g., reproductive systems, sinus cavities, submucosal regions, respiratory tract), and the like.

[0004] Conventional endoscopes can be involved in a variety of clinical procedures, including, for example, illuminating, imaging, detecting and diagnosing one or more disease states, providing fluid delivery (e.g., saline or other preparations via a fluid channel) toward an anatomical region, providing passage (e.g., via a working channel) of one or more therapeutic devices for sampling or treating an anatomical region, and providing suction passageways for collecting fluids (e.g., saline or other preparations) and the like.

[0005] In endoscopic procedures, the control of endoscopy equipment has traditionally required manual intervention, where clinicians navigate between patient care and equipment operation through tactile buttons, foot pedals, and visual interfaces. This setup demands significant attention and manual dexterity, often disrupting the procedural workflow or requiring assistance from the endoscopy staff.

[0006] Traditionally, the Video Processor (VP) in endoscopic equipment has been operated through a variety of manual inputs. 1) Endobuttons: A limited number of small buttons can be located on the endoscope itself and can be designed for quick actions. However, these are often limited in number, leading to a limited range of pre-configurable actions. 2) Front Instrument Panel: Buttons, switches and the like are typically located on the endoscopy tower or the control unit. These control features typically require the endoscopist or an assistant to reach out and manually adjust settings, potentially distracting from the procedure. 3) Keyboard Input: A conventional or a dedicated (specially designed) keyboard having buttons offers versatile means for entering commands for the endoscopy system, but can demand attention away from the procedure for input and can break sterility (in surgical procedures) if the endoscopist has to re-glove to use the keyboard. 4) Foot Pedal: A foot pedal can be located near the patient to allow the endoscopist to control various endoscopy system capabilities. Foot pedals offer hands-free operation, but are typically limited to controlling a single feature, such as a high frequency generator. Additional foot pedals can be difficult to use precisely, often requiring extensive practice, and may contribute to fatigue during long procedures. 5) Voice Instructions: Conventional voice control capabilities for endoscopy systems offer hands-free operation, but can be restricted to certain command phrases. For example, prior art voice-controlled endoscopy systems have been limited to control of video processors to perform simple, one-step, single function commands.

[0007] In view of the foregoing, there is a need in the art for improved control over features of endoscopy systems.SUMMARY

[0008] The present disclosure recognizes that problems to be solved with conventional endoscopy suites include, among other things, 1) Cumbersome for Multi-step Command: Adjusting complex settings or navigating through menus using traditional inputs can be timeconsuming and require multiple steps, interrupting the flow of the procedure, 2) Limited Configuration of Endobuttons: The fixed number of endobuttons constrains the number of pre-set actions, often leading to compromises on the functionalities readily available to the endoscopist, 3) Ergonomic Difficulties: Since the VP is typically located behind the endoscopist, it may require awkward physical movements to reach and operate, contributing to ergonomic strain and inefficiency, and 4) Connected Peripherals: There is no centralized module that deciphers commands to be executed by the VP and then by peripherals such as AI / CAD modules.

[0009] The present disclosure describes a voice-controlled interface that offers a system where endoscopists can command the video processor with their voice, maintaining technique and patient focus. This hands-free approach simplifies interactions of the clinician with technology, streamlining the procedural process and potentially enhancing patient outcomes.

[0010] The endoscopy suites of the present disclosure provide comprehensive voice-activated control systems specifically designed for use across the endoscopy suite that include among other things a natural language interface allowing for multi-step configurations and a built-in safety protocol to restrict voice commands for diagnostic functionality without affecting minimum or baseline performance of the devices during procedure.

[0011] The present disclosure can help provide solutions to the problems enumerated herein and other problems by providing devices, systems and methods relating to voice control capabilities for endoscopy suites. The voice control capabilities can perform multiple tasks including: 1) Complex Commands using Natural Language Interface: Using voice prompts to decode control sequences and execute the control sequences across peripherals in the endoscopy suite, 2) Sanity and Safety Checks: The voice control capabilities can include checks to ensure that the future state of the endoscopy suite is safe prior to command execution, and 3) Peripheral Device Control: The voice control capabilities of the present disclosure can provide a centralized command module for voice based controls of multiple connected peripherals such as AI / CAD modules, surgical instruments, energy devices, video recording devices, augmented reality solutions, etc.

[0012] The benefits of the voice control systems and methods of the present disclosure are numerous, including:

[0013] 1) Seamless Multi-step Configuration: Complex settings can be adjusted through simple voice commands, streamlining configuration and endoscopy processes and maintaining focus on the endoscopy procedure. The underlying control logic of the voice control system can translate natural language voice prompts into systemic machine commands that are automatically relayed to the endoscopy equipment for execution, 2) Elimination of Physical Barriers: With voice commands of the present disclosure, the need for the endoscopist to turn away from the patient and reach for controls is eliminated, addressing ergonomic concerns and maintaining procedural sterility, 3) Customizable Command Set: The voice systems of the present disclosure can be trained using artificial intelligence models to recognize an extensive set of commands, exceeding the limitations of physical endobutton configurations, 4) Enhanced Device Interaction: Simplified control of sophisticated endoscopy suite functionalities without the need for manual adjustments, 5)Greater Flexibility: The voice control capabilities of the present disclosure provide more freedom in configuring controls in a way that best suits the workflow of individual practitioners or procedures, and 6) Improved Procedural Focus: The voice control systems of the present disclosure can allow endoscopists to remain visually and mentally focused on the endoscopic image and the patient, thereby potentially reducing procedural times and improving outcomes.

[0014] In an example, a control system of an endoscopy ecosystem having an endoscopy system and at least one peripheral system can comprise a plurality of modules for controlling capabilities of the endoscopy ecosystem provided by one or more of the endoscopy system and the at least one peripheral system, and the control system is configured to receive a voice command from a user to enable a desired functionality of the endoscopy ecosystem, decode the voice command into a system instruction for controlling the desired functionality, parse the system instruction into a first sub-command for a first identified module of the plurality of modules for controlling a first aspect of the desired functionality and a second subcommand for a second identified module of the plurality of modules for controlling a second aspect the desired functionality, and transmit the first sub-command to the first identified module and the second sub-command to the second identified module, wherein transmission of the first sub-command and the second sub-command cause the endoscopy ecosystem to execute the desired functionality.

[0015] In another example, a method for operating an endoscopy ecosystem comprising an endoscopy system and at least one peripheral system for the endoscopy system can comprise receiving a voice command from a user at a control system, the control system configured to activate a desired functionality of the endoscopy ecosystem, decoding the voice command into a system instruction for operating modules of the endoscopy ecosystem to execute the desired functionality of the endoscopy ecosystem, parsing the system instruction into a first sub-command for a first identified module of the modules for controlling a first aspect of the desired functionality and a second sub-command for a second identified module of the modules for controlling a second aspect the desired functionality, and transmitting the first sub-command to the first identified module and the second sub-command to the second identified module, wherein transmission of the first sub-command and the second subcommand cause the endoscopy ecosystem to execute the desired functionality.BRIEF DESCRIPTION OF DRAWINGS

[0016] FIG. l is a schematic illustration of an endoscopy system suitable for use with the voice command control module of the present disclosure.

[0017] FIG. 2 is a schematic illustration of a control system for the endoscopy system of FIG.1 including the voice command control module of the present disclosure.

[0018] FIG. 3 A is an end view of an endoscope suitable for use with the endoscopy system of FIG. 1.

[0019] FIG. 3B is a cross-sectional view of the endoscope of FIG. 3 A.

[0020] FIG. 4 is a side view of a biopsy device extending from an endoscope suitable for use with the endoscope system of the present disclosure.

[0021] FIG. 5 is a schematic illustration of a surgical suite having the voice command control module of FIG. 1 and FIG. 2 in communication with peripheral systems.

[0022] FIG. 6 is a block diagram chart illustrating various operations for initializing and configuring a voice command control module of the present disclosure.

[0023] FIG. 7 is a block diagram illustrating various operations for generating user voice profiles for a voice command control module of the present disclosure.

[0024] FIG. 8 is a block diagram illustrating various operations for operating a voice command control module of the present disclosure.

[0025] FIG. 9 is a block diagram illustrating an example of performing a colonoscopy procedure using a voice command control module of the present disclosure.

[0026] FIG. 10 is a block diagram illustrating various operations in performing methods of operating an endoscopy system with a voice command control module.

[0027] FIG. 11 A is a first state diagram of the surgical suite of FIG. 5 showing the voice control module issuing instructions to multiple cloud-based modules.

[0028] FIG. 1 IB is a second state diagram of the surgical suite of FIG. 5 showing the voice control module issuing instructions to a local hardware or software module and a cloud-based module.

[0029] FIG. 11C is a third state diagram of the surgical suite of FIG. 5 showing the voice control module issuing instructions to multiple local hardware or software modules.DETAILED DESCRIPTION

[0030] FIG. 1 is a schematic diagram of endoscopy system 10 comprising imaging and control system 12, voice control module 13 and endoscope 14. Endoscopy system 10 can be connected to cloud services 15. The system of FIG. 1 is an illustrative example of anendoscopy system suitable for use with or as part of an endoscopy suite that can include the systems, devices and methods of the present disclosure relating to voice control module 13. A typical endoscopy suite can comprise a room or facility where an endoscopy procedure is performed, wherein the room or facility includes an endoscope, an endoscope control system (e.g., tower) and peripheral components. The endoscopy suite can be part of a larger endoscopy ecosystem where the systems of the endoscopy suite are connected to endoscopy peripheral modules that can comprise local hardware modules, local software modules and cloud-based modules. Endoscopy peripheral modules can comprise various hardware devices and systems and various software devices and systems that provide added functionality to the endoscope and endoscopy system, whether at the location of endoscopy system 10 or remotely. In examples, an endoscopy ecosystem can additionally comprise a digital ecosystem of interconnected information technology resources that can function together as a unit, including suppliers, customers, trading partners, applications, third-party data service providers and all respective technologies that are connected in an interoperable manner.

[0031] According to some examples, endoscope 14 can be insertable into an anatomical region for imaging and / or to provide passage of other devices, such as auxiliary scopes and biopsy devices or one or more therapeutic devices for treatment of a disease state associated with the anatomical region. Endoscope 14 can, in advantageous aspects, interface with and connect to imaging and control system 12 such as via insertion of coupler section 36 into socket 37. In the illustrated example, endoscope 14 comprises a duodenoscope, though other types of endoscopes can be used with the features and teachings of the present disclosure, such as gastroscopes, colonoscopes, and cholangioscopes. In additional examples, the voice control modules of the present disclosure can be used with bronchoscopes, cystoscopes, laparoscopes, hysteroscopes, anoscopes, arthroscopes and the like.

[0032] Imaging and control system 12 can comprise control unit 16, output unit 18, input unit 20, light source unit 22, fluid source 24 and suction pump 26. Imaging and control system 12 can be referred to as a tower. Voice control module 13 can be configured as part of imaging and control system 12, such as part of control unit 16, or can comprise a separate add-on component. In examples voice control module 13 can be included in the same housing as control unit 16 or can be remotely connected thereto by wired or wireless connections.Output unit 18, input unit 20, light source unit 22, fluid source 24 and suction pump 26, can comprise modules of endoscopy system 10. Cloud services 15 can comprise a peripheral module. Voice control module 13 can be configured to operate any or all of the modules of endoscopy system 10 or any or all peripheral modules connected thereto, such as those shownin FIG. 5. As discussed in greater detail herein, voice control module 13 can receive an audio instruction from a user of endoscopy system 10 and convert such audio input into a machine command executable by a module. Furthermore, voice control module 13 can convert audio voice command inputs for a single function of endoscopy suite 300 into parsed commands for multiple modules, performing safety checks of the commands in the process.

[0033] Imaging and control system 12 can include various ports for coupling with endoscopy system 10. For example, control unit 16 can include a data input / output port for receiving data from and communicating data to endoscope 14. Such data input / output can be provided through an interface between coupler section 36 and socket 37. Light source unit 22 can include an output port for transmitting light to endoscope 14, such as via a fiber optic link. For example, coupler section 36 can include light conductor bundle 39 (FIG. 2) that is configured to receive light from one or more lenses or bulbs within light source unit 22.Fluid source 24 can include a port for transmitting fluid to endoscope 14. Fluid source 24 can comprise a pump and a tank of fluid or can be connected to an external tank, vessel or storage unit. Suction pump 26 can comprise a port used to draw a vacuum from endoscope 14 to generate suction, such as for withdrawing fluid from the anatomical region into which endoscope 14 is inserted. In examples, fluid, such as air or another gas, can be transferred to endoscope 14 through an interface at coupler section 36 and socket 37. In examples, fluids, such as water or saline, can be directly input into coupler section 36 without emanating from socket 37. Output unit 18, e.g., a touch-screen display, and input unit 20, e.g., a keyboard, can be used by an operator of endoscopy system 10 to control functions of endoscopy system 10 and view output of endoscope 14. Control unit 16 can additionally be used to generate signals or other outputs from treating the anatomical region into which endoscope 14 is inserted. In examples, control unit 16 can generate electrical output, acoustic output, a fluid output and the like for treating the anatomical region with, for example, cauterizing, cutting, freezing and the like.

[0034] Endoscope 14 can comprise insertion section 28, functional section 30 and handle section 32, which can be coupled to cable section 34 and coupler section 36. Coupler section 36 can be connected to control unit 16 at socket 37 to connect to endoscope 14 to multiple features of control unit 16, such as input unit 20 and light source unit 22. Fluid source 24 and suction pump 26 can be connected directly to endoscope 14 without routing through control unit 16.

[0035] Insertion section 28 can extend distally from handle section 32 and cable section 34 can extend proximally from handle section 32. Insertion section 28 can be elongated andinclude a bending section, and a distal end to which functional section 30 can be attached. The bending section can be controllable (e.g., by control knob 38 on handle section 32) to maneuver the distal end through tortuous anatomical passageways (e.g., stomach, duodenum, kidney, ureter, etc.). Insertion section 28 can also include one or more working channels (e.g., an internal lumen) that can be elongate and support insertion of one or more therapeutic tools of functional section 30, such as an auxiliary scope, such as those shown in FIG. 3A, FIG. 3B and FIG. 4. The working channel can extend between handle section 32 and functional section 30. Additional functionalities, such as fluid passages, guide wires, and pull wires can also be provided by insertion section 28 (e.g., via suction or irrigation passageways, and the like).

[0036] Handle section 32 can comprise endobuttons 35, control knob 38 as well as port 40A. Control knob 38 can be coupled to a pull wire, or other actuation mechanisms, extending through insertion section 28. Coupler section 36 and port 40 A, as well as other ports, such as port 40B (FIG. 2), can be configured to couple various electrical cables, guide wires, auxiliary scopes, tissue collection devices, fluid tubes and the like to handle section 32 for coupling with insertion section 28. In examples, handle section 32 can additionally include control features for operating capabilities of functional section 30 and image processing adapters of the present disclosure.

[0037] Imaging and control system 12, according to examples, can be provided on a mobile platform (e.g., cart 41) with shelves for housing of light source unit 22, suction pump 26, image processing unit 42 (FIG. 2), etc. Alternatively, several components of imaging and control system 12 shown in FIGS. 1 and 2 can be provided directly on endoscope 14 so as to make the endoscope “self-contained.”

[0038] Functional section 30 can comprise components for treating and diagnosing anatomy of a patient. Functional section 30 can comprise an imaging device, an illumination device (e.g., the distal end of a light fiber) and an elevator. In examples, the imaging device can be used with various lenses, such as lenses that can bend light or protect the imaging device. Operation of some or all features of functional section 30 is typically performed at imaging and control system 12 or at handle section 32.

[0039] Voice control module 13 can be in communication with imaging and control system 12, such as through control unit 16. Voice control module 13 and / or control unit 16 can be used to control functionality of endoscopy system 10 and modules in communication with endoscopy system 10, such as cloud services 15. As mentioned, voice control module 13 can process a natural language voice input from a user to activate a functionality of endoscopysystem 10 and can issue instructions to implement the desired functionality, including changing the operating state of multiple hardware, software and cloud-based modules to ensure the desired functionality is implemented safely.

[0040] FIG. 2 is a schematic diagram of endoscopy system 10 of FIG. 1 comprising imaging and control system 12 and endoscope 14. FIG. 2 schematically illustrates modules or components of imaging and control system 12 coupled to endoscope 14, which in the illustrated example comprises a duodenoscope. Imaging and control system 12 can comprise control unit 16, which can include or be coupled to image processing unit 42, treatment generator 44 and drive unit 46, as well as light source unit 22, input unit 20, output unit 18 and voice control module 13. Furthermore, control unit 16 can be in communication with one or more of peripheral units 45. As discussed herein, peripheral units 45 can comprise Artificial Intelligence (Al) modules, CAD modules, such as CADx (Computer Aided diagnosis) and CADe (Computer Aided detection), high frequency energy devices, motor controllers, video recording modules, augmented reality modules, video processors and the like, as is described with reference to FIG. 5, for example. Coupler section 36 can be connected to control unit 16 to connect to endoscope 14 to multiple features of control unit 16, such as image processing unit 42 and treatment generator 44. In examples, plug portion 48 of coupler section 36 can include leads 49 for connecting to wiring within socket 37 (FIG. 1) that can connect to one or more of light source unit 22, image processing unit 42 and treatment generator 44. In examples, port 40A can be used to insert another instrument or device, such as a daughter scope or auxiliary scope, such as the device of FIG. 3 A and FIG. 3B, into endoscope 14. Such instruments and devices can be independently connected to control unit 16 via cable 47 or can extend directly from fluid source 24 and suction pump 26 without coming from control unit 16. In examples, port 40B can be used to connect coupler section 36 to various inputs and outputs, such as video, air, light and electric. Control unit 16 can be configured to activate a camera to view target tissue distal of endoscope 14. In examples, functional section 30 can include an imaging device that can transmit an imaging signal to imaging and control system 12 through leads 49 that connect to wiring within socket 37 (FIG. 1). Likewise, control unit 16 can be configured to activate light source unit 22 to direct light into endoscope 14 or other devices extending therefrom. Light source unit 22 can comprise one or more light generators, such as a xenon bulb or a light emitting diode. In example, light source unit 22 can include multiple light generators to generate light with different properties, such as different color (e.g., blue light and white light).

[0041] Image processing unit 42 and light source unit 22 can each interface with endoscope 14 (e.g., at functional section 30) by wired or wireless electrical connections. Imaging and control system 12 can accordingly illuminate an anatomical region, collect signals representing the anatomical region, process signals representing the anatomical region, and display images representing the anatomical region on output unit 18. Imaging and control system 12 can include light source unit 22 to illuminate the anatomical region using light of desired spectrum (e.g., broadband white light, narrow-band imaging using preferred electromagnetic wavelengths, and the like). Imaging and control system 12 can connect (e.g., via an endoscope connector or socket 37 (FIG. 1)) to endoscope 14 for signal transmission (e.g., light output from light source, video signals from imaging system in the distal end, diagnostic and sensor signals from a diagnostic device, and the like).

[0042] Fluid source 24 (FIG. 1) can be in communication with control unit 16 and can comprise one or more sources of air, saline or other fluids, as well as associated fluid pathways (e.g., air channels, irrigation channels, suction channels) and connectors (barb fittings, fluid seals, valves and the like). Fluid source 24 can additionally provide insufflation gas. Imaging and control system 12 can also include drive unit 46, which can be an optional component. Drive unit 46 can comprise a motorized drive for advancing a distal section of endoscope 14, as described in at least PCT Pub. No. WO 2011 / 140118 Al to Frassica et al., titled “Rotate-to-Advance Catheterization System,” the contents of which are incorporated herein by this reference.

[0043] As mentioned, coupler section 36 can be used to connected endoscope 14 with imaging and control system 12. Coupler section 36 can be used to communicate various functions between endoscope 14 and imaging and control system 12. In examples, coupler section 36 can transmit communication signals, electronic signals, electrical signals, power signals, fluids including water and air, light waves and the like. Coupler section 36 can comprise a part of endoscope 14 and can be configured for particular configurations of imaging and control system 12. For example, coupler section 36 can be configured to transmit light generated by light source unit 22 to endoscope 14 using light conductor bundle 39. Additionally, coupler section 36 can utilize leads 49 to transmit imaging signals between functional section 30 and imaging and control system 12.

[0044] With the present disclosure, voice control module 13 and / or control unit 16 can be configured to allow a user or endoscopist to control most or all of the functionality of endoscopy system 10 and other modules connected thereto, such as peripheral units 45, so that the user need not perform complicated task sequences, such as with endobuttons 35, orhave attention diverted away from the endoscopy procedure to interact with control unit 16, output unit 18, input unit 20 and other physical interaction features. Voice control module 13 can be programmed to work with different users having different speech patterns, vocabulary, accents and the like. Voice control module 13 can further be customized to the preferences of a user, the type of procedure being performed and the specific equipment, e.g., endoscope 14, being used. In particular examples, voice control module 13 can additionally interpret natural language voice commands, such as by using word disambiguation and semantic matching techniques and artificial intelligence techniques, to determine if multiple actions are needed by endoscopy system 10 to meet the requested command, including issuing instructions to components or modules of endoscopy system 10 not specifically requested by the user in order to put endoscopy system 10 in a state most readily compatible with the requested action.

[0045] FIG. 3 A illustrates an end view of end-viewing endoscope camera module 70 and FIG. 3B illustrates a cross-sectional view of end-viewing endoscope camera module 70 taken along section plane 3B - 3B of FIG. 3 A. FIG. 3 A and FIG. 3B are discussed concurrently. FIG. 3 A and FIG. 3B each illustrate end-viewing endoscope camera module 70, such as for use as a gastroscope, colonoscope, cholangioscope, and the like. In end-viewing endoscope camera module 70, illumination and imaging systems are positioned such that the viewing angle of the imaging system corresponds to a target anatomy located adjacent (e.g., distal of) an end of endoscope 14 and in line with a central longitudinal axis of endoscope 14.

[0046] End-viewing endoscope camera module 70 can be used within a cholangioscope that can be inserted into a working channel of the duodenoscope illustrated in FIG. 1. Thus, endviewing endoscope camera module 70 is configured to be aimed in the axial or distal direction relative to the working shaft of the endoscope. However, the components of endviewing endoscope camera module 70 of FIG. 3 A and FIG. 3B can be used as an alternative example of functional section 30 of endoscope 14 of FIG. 1 and FIG. 2. For example, the components of end-viewing endoscope camera module 70 can be rotated ninety-degrees to operate in a side-viewing capability.

[0047] In the example of FIG. 3 A and FIG. 3B, end-viewing endoscope camera module 70 can comprise housing 72, therapy unit 74, fluid outlets 76, illumination lens 78 and objective lens 80. Housing 72 can comprise and endcap for insertion section 28, thereby providing a seal to lumen 82.

[0048] As can be seen in FIG. 3B, insertion section 28 can comprise lumen 82 through which various components can be extended to connect end-viewing endoscope camera module 70with handle section 32 (FIG. 2), for example. For example, illumination lens 78 can be connected to light transmitter 84, which can comprise a fiber optic cable or cable bundle extending to light source unit 22 (FIG. 1) similar to light conductor bundle 39. Likewise, objective lens 80 can be coupled to imaging unit 87, which can be coupled to wiring 88. In examples, light transmitter 84 can be located in the same lumen within housing 72, but light transmitter 84 and wiring 88 can be located within separate lumens within housing 72. As can be seen, objective lens 80 and imaging unit 87 can be wider or have a larger diameter than wiring 88. Lumen 82 can be constructed to be wide to accept objective lens 80 and imaging unit 87, and can be necked down to receive wiring 88. Also, fluid outlets 76 can be coupled to fluid lines 89, which can comprise a tube extending to fluid source 24 (FIG. 1). In examples, one of fluid outlets 76 can comprise an inlet connected to a fluid line 89 configured for suction, such as being connected to a vacuum, for recovery of lavage and irrigation fluid, or can comprise an inlet for receiving a gas for insufflation. Other elongate elements, e.g., tubes, wires, cables, can extend through lumen 82 to connect functional section 30 with components of endoscopy system 10, such as suction pump 26 (FIG. 1) and treatment generator 44 (FIG. 2). For example, therapy unit 74 can comprise a wide-diameter lumen for receiving other treatment components, such as cutting devices and therapeutic devices including tissue separator devices.

[0049] End-viewing endoscope camera module 70 can also include a photosensitive element, such as a charge-coupled device (“CCD” sensor) or a complementary metal-oxide semiconductor (“CMOS”) sensor. In either example, imaging unit 87 can be coupled (e.g., via wired or wireless connections) to image processing unit 42 (FIG. 1) to transmit signals from the photosensitive element representing images (e.g., video signals) to image processing unit 42, in turn to be displayed on a display such as output unit 18. In various examples, imaging and control system 12 and imaging unit 87 can be configured to provide outputs at desired resolution (e.g., at least 480p, at least 720p, at least 1080p, at least 4K UHD, etc.) suitable for endoscopy procedures.

[0050] FIG. 4 is a side view of forceps 200 suitable for use as a biopsy device of the present disclosure. Forceps 200 can comprise base 240, jaws 242A and 242B, hinge 244, actuators 246A and 246B and control wires 248A and 248B. Forceps 200 is described with reference to engagement with endoscope 230 comprising shaft 232, working channel 234, lumen 236, imaging lens 238 and illumination lens 239. In examples, endoscope 230 can comprise a cholangioscope suitable for use within a working channel of endoscope 14 of FIG. 1. Inexamples, endoscope 230 can be configured to include the features of end-viewing endoscope camera module 70 of FIG. 3 A and FIG. 3B.

[0051] Base 240 can be configured to engage shaft 232. Base 240 can pass through the distal-most face of shaft 232 and can be configured to be shorter than height Hl of working channel 234, thereby allowing base 240 being capable of passing through working channel 234. In examples, base 240 can provide a stable connection to shaft 232, thereby inhibiting rocking or vibration, and allowing jaws 242A and 242B to firmly engage target tissue. Base 240 can additionally be configured to be positively held in place relative to shaft 232 via a mechanical coupling or the like.

[0052] Hinge 244 can comprise a connection point for jaws 242A and 242B to couple to base 240. Hinge 244 can comprise a round pin or shaft over which corresponding bores in jaws 242 A and 242B can be fit. Thus, jaws 242 A and 242B can be configured to freely rotate on hinge 244. However, rotation of jaws 242A and 242B on hinge 444 can be controlled by control wires 248A and 248B. Control wires 248A and 248B can be coupled to actuators 246A and 246B, respectively, of jaws 242A and 242B. Actuators 246A and 246B can comprise levers extending at angle from jaws 242A and 242B relative to a centerline of working channel 234. Thus, control wires 248A and 248B can be operated by a handpiece, such as handle section 32, to pull actuators 246A and 246B to rotate jaws 242A and 242B about hinge 244 to facilitate collection of tissue samples. In examples, control wires 248A and 248B can be connected to motors to enable automated, voice command operation. In examples, control wires 248A and 248B can be pre-curved to impart rotational bias to actuators 246A and 246B to an open or closed position. However, in examples, actuators 246A and 246B can be provided with other biasing elements, such as springs. As such, pulling of control wires 248 A and 248B can cause closing or opening of jaws 242 A and 242B, as desired. As illustrated, jaws 242A and 242B can include teeth to facilitate cutting and tearing of tissue away from the anatomy. Though the illustrated example is shown with reference to actuators comprising levers, other actuators, such as pull rods or screw mechanisms, can be used.

[0053] FIG. 5 is a schematic illustration of endoscopy suite 300 of the present disclosure suitable for use with the endoscopy system 10 of FIG. 1 and FIG. 2. Endoscopy suite 300 can comprise control unit 302, endoscope 304, voice control module 306 and display 308. Endoscopy suite 300 can additionally comprise hardware and / or software peripheral modules, such as robotic system 310, video recording systems 311, augmented reality systems 312, automation systems 314, fluid systems 316, light source 317 and video processing systems318. Endoscopy suite 300 can additionally comprise remote peripheral modules, such as CADe module 320, CADx module 322 and medical report generator module 324, which can be in communication with medical records database 326. Additionally, control unit 302 can be in communication with microphone 328.

[0054] As discussed herein, voice control module 306 and / or control unit 302 can be configured to control peripheral devices and modules beyond video processing systems 318, which can be conventionally voice controlled to perform simple, one-step, single function commands, in order to produce a fully voice-activated endoscopic suite. Peripheral modules can include, but are not limited to, waterjet systems, energy devices, such as acoustic, electrical and the like, AI / CAD devices, etc., such are shown and discussed with reference to FIG. 5. Voice control module 306 can either work on an on-premises computer device or work by streaming the data to a cloud data centre for computation. Thus, voice control module 306 can reside within or be connected to control unit 302 via wired or wireless connections. Voice control module 306 can comprise an instance of or be configured similarly as voice control module 13 of FIG. 1 and FIG. 2. Microphone 328 can be in communication with voice control module 306 either directly or indirectly, as illustrated, through control unit 302. Microphone 328 can be configured to receive soundwaves of the voice of a user and convert the soundwaves to digital or electronic files. As discussed below, voice control module 306, with or without the assistance of control unit 302, can perform voice-to-text and natural language processing of the soundwaves to determine a called for capability of endoscopy suite 300 and deliver appropriate instructions to one or more modules of endoscopy suite 300 to enable or activate the capability. Voice control module 306 can be operated in a closed loop with a feedback loop for continuous learning. The continuous learning of such a system may be accomplished via explicit endoscopist feedback such as user provided satisfaction with recommendations of instructions to deliver to various components or peripherals of the endoscopy suite (e.g., a user may indicate such satisfaction via a “like” button etc. through use of a touch screen interface on display 308). Voice control module 306 be used with many different types of procedures, and systems for performing such procedures, including anoscopy, arthroscopy, bronchoscopy, colonoscopy, colposcopy, cystoscopy, esophagoscopy, gastroscopy, laparoscopy, laryngoscopy, neuroendoscopy, proctoscopy, sigmoidoscopy, thoracoscopy etc.

[0055] Control unit 302 can comprise an instance of or be configured similarly as control unit 16 of FIG. 1 and FIG. 2. Control unit 302 can be in electronic communication with the modules connected thereto, either directly or indirectly. That is, control unit 302 can bedirectly connected via wired or wireless connection to each module, or control unit 302 can be directly connected via wired or wireless connection to one or less than all of the modules, and some or all of the modules can be connected to each other via wired or wireless connections. Control unit 302 and the modules can be connected to each other to operate as a mesh network. Control unit 302 can receive signals from and send signals to each of the modules. Each of the modules connected to control unit 302 can be individually connected to each other, either wholly or partially. Control unit 302 can send instructions to any one of the modules connected thereto and any of the modules can be configured to route instructions to any other module thereby acting as a bridge between modules. Control unit 302 can be in wired or wireless communication with each module connected thereto or be connected via network connections or the cloud. Control unit 302 can receive inputs from a user via direct or indirect input. For example, a user can utilize buttons, e.g., endobuttons 35 of FIG. 2, on endoscope 304, user interface features, such as a keyboard or mouse or a foot pedal, to control features of endoscopy suite 300, such as the illustrated modules and others. Thus, voice control module 306 can provide parallel functionally to user interface features, such as endobuttons 35 and a keyboard of input unit 20. With the present disclosure, control unit 302 and / or voice control module 306 via connection to control unit 302 or otherwise, provide control instructions to the modules attached to control unit 302, including endoscope 304, display 308, robotic system 310, video recording systems 311, augmented reality systems 312, automation systems 314, fluid systems 316, light source 317 and video processing systems 318, as well as CADe module 320, CADx module 322 and medical report generator module 324. In examples, voice control module 306 can be configured to receive an audio input from a user for operating a single feature or module of endoscopy suite 300 and can interpret the desired operation of the single feature or module to adjust other system settings, such as parallel, ancillary or subordinate systems or modules, accordingly. As used herein, the term “ancillary functionality” used in the context of supporting a desired functionality can refer to a functionality that may be implemented to improve the performance or outcome of such desired functionality. As a specific but non-limiting example, under circumstances in which a desired functionality is an artificial intelligence based image analysis function (e.g., a CADx module) which operates optimally under specific lighting conditions (e.g., a NBI lighting modality), then a simple natural language command of “switch to CADx” could result in a first sub-command directed toward the artificial intelligence based image analysis function and a second sub-command directed toward the ancillary functionality of adjusting the lighting conditions as needed to optimize performance. For example, the voice controlmodule 306 may receive an audio input from a user uttering the phrase “take a picture of that polyp” and may interpret the phrase as requiring a first command to be sent to the video recording systems 311 instructing that an image be captured, and a second command be sent to the medical report generator 324 instructing that the captured image be added to a medical report. In this way, the underlying intent of the uttered phrase (e.g., populating a medical report with a polyp of interest) may be facilitated despite the uttered phrase not expressly mentioning the medical report. As another example, the voice control module 306 may receive an audio input from a user uttering the phrase “switch to CADx” and may interpret the phrase as requiring a first command to be sent to CADx module 322 (e.g., to turn it on), and a second command be sent to the endoscope 304 to switch the lighting modality from standard white light to a narrow band imaging (NBI) mode (e.g., as some CADx modules are trained to analyze images captured in NBI mode). In this way, the underlying intent of the uttered phrase is facilitated without a need for the user to issue complex multi-part instructions adjusting all the system parameters needed to accomplish the intent.

[0056] Endoscope 304 can comprise an instance of or be configured similarly as endoscope 14 of FIG. 1 and FIG. 2. Endoscope 304 can include input buttons, e.g., endobuttons 35, to operate one or more capability of endoscope 14, such as to take an image or a video, turn on or off one or more light sources, to activate a fluid dispensing operation or a suction operation.

[0057] Display 308 can comprise an instance of or be configured similarly as output unit 18 of FIG. 1 and FIG. 2. Display 308 can display video output of endoscope 304. Display 308 can provide video output of a plurality of endoscopes, such as a duodenoscope and a cholangioscope. Display 308 can additionally be configured as a touch screen to provide control instructions to endoscopy suite 300. Display 308 can additionally be in communication with voice control module 306 to provide safety checks and warnings to a user, as discussed herein.

[0058] Control of Robotic Assistants: Voice control module 306 can be used to control robotic assistants during endoscopic procedures. For example, a voice-controlled robotic arm could be used to support and move endoscope 304. This could provide more precise control and reduce the physical strain on the surgeon. Robotic systems 310 can comprise one or more robotic surgical systems that can be used to automatically move medical devices and instruments within a surgical space. In examples, robotic systems 310 can comprise a robotic surgical system and described in Pub. No. US 2023 / 0320793 Al to Yanagihara et al., titled “Surgery System and Control Method for Surgery System,” the contents of which areincorporated herein by this reference. In examples, robotic systems 310 can comprise a robotic surgical system and described in Pub. No. US 2023 / 0117954 Al to Mino et al., titled “Automatic Positioning and Force Adjustment in Endoscopy,” the contents of which are incorporated herein by this reference.

[0059] Video Recording Solutions: Voice control module 306 can control operation of endoscope 14 and video recording systems 311. Voice control module 306 can control camera views across multiple surgical endoscopes that are inserted into the human body during minimally invasive surgery, such as a duodenoscope and a cholangioscope. For example, voice control module 306 can switch between output of a duodenoscope (e.g., endoscope 14 of FIG. 1) and a cholangioscope (e.g., endoscope 230 of FIG. 4), and can generate picture-in-picture formatting, additionally providing instruction to display 308 about which video stream is being provided or a hybrid video stream is being provided. Video recording systems 311 can comprise a system for recording video output of endoscope 304 or other endoscopes connected to control unit 302. Video recording system 311 can save recorded video to computer readable storage mediums in communication with control unit 302. Video recording systems 311 can additionally record output of CADe module 320 and CADx module 322 overlayed over output of cameras of endoscope 304.

[0060] Augmented Reality Solutions: Voice control module 306 can switch on or off or configure augmented reality solutions for therapy and surgical planning and execution. Augmented reality systems 312 can comprise a system for projecting indicia into a viewing area of a user of endoscopy suite 300. In examples, augmented reality systems 312 can comprise a robotic surgical system and described in Pub. No. US 2023 / 0363621 Al to Mino et al., titled “Ai-based Endoscopic Tissue Acquisition Planning,” the contents of which are incorporated herein by this reference. Augmented reality systems 312 can include a headset or projectors configured to project instructions of a surgical plan into the space of endoscopy suite 300.

[0061] Automated Endoscope Positioning: Voice control module 306 can control the positioning of the endoscope using voice commands. This can include commands for moving endoscope 304 up, down, left, right, in, or out. Voice control module 306 can also be programmed to remember certain anatomical positions and return to them with a single voice command. This can reduce the need for a second surgeon to hold and maneuver endoscope 304, potentially reducing operating time. Automation systems 314 can comprise a system for assisting human movement of a medical device or instrument. In examples, automation systems 314 can comprise an automation system described in PCT Pub. No. WO2011 / 140118 Al to Frassica et al., titled “Rotate-to-Advance Catheterization System,” the contents of which are incorporated herein by this reference.

[0062] Activation of Surgical Tools: Voice control module 306 can be used to activate and control various surgical tools during endoscopic procedures. This could include tools for cutting, coagulation, suction, and water flushing. Voice control module 306 can control a high-frequency generator, allowing the surgeon to start and stop cutting or coagulation modes, switch between modes, and adjust power levels using voice commands. As mentioned with reference to FIG. 4, voice control module 306 can control actuators for operating a biopsy device.

[0063] Fluid systems 316 can comprise a module for introducing a fluid into anatomy of a patients. In examples, fluid systems 316 can deliver a fluid through a medical device, such as endoscope 304. In examples, fluid systems 316 can deliver a liquid, such as water or saline. The liquid can be used to perform lavage. In examples, fluid systems 316 can deliver a gas, such as oxygen, air, helium and the like. The gas can be used to perform insufflation. Voice control module 306 can be used to operate, e.g., turn on and turn off, fluid systems 316.

[0064] Light source 317 can comprise a device or system configured to output light, e.g., light waves. Light source 317 can comprise or be configured similarly to light source unit 22 of FIG. 1 and FIG. 2. In examples, light source 317 can comprise a light bulb, a light emitting diode and the like. In examples, light source 317 can be configured to output different types of visible and invisible light. In examples, light source 317 can comprise a halogen bulb or a xenon lamp. Additionally, light source 317 can be configured to emit multiple types of light, such as by including multiple light sources or light emitters. Light from light source 317 can be used to enhance output of other modules of endoscopy suite 300, such as CADe module 320 and CADx module 322. For example, CADe module 320 and CADx module 322 can benefit from different types of light to facilitate identification of different tissue types.

[0065] Real-Time Imaging Adjustments: Voice control module 306 can be used to make realtime adjustments to imaging functions during endoscopic procedures, such as by controlling operation of video processing systems 318. This can include adjusting the focus or zoom, selecting image enhancement modes (e.g., smoothing, sharpening, contrast enhancements, color corrections, compression), and toggling between functions to optimize the image of a lesion before taking pictures. Video processing systems 318 can comprise a system for receiving video output from endoscope 304 and providing various operations on such video output. Video processing systems 318 can receive input from multiple endoscopes, such as aduodenoscope and a cholangioscope. Video processing can include filtering, stretching, coloring, zooming, cropping, combining and the like.

[0066] CADe module 320 can comprise a module configured to analyze video output of anatomy to identification of tissue formations therein. CADe module 320 can include capabilities for marking or annotating the video output of endoscope 304. CADx module 322 can comprise a module configured to analyze video output of anatomy to perform identification and diagnosis of tissue formations therein. CADx module 322 can include capabilities for marking or annotating the video output of endoscope 304. CADe / CADx modules can be artificial intelligence computer modules for detection and / or classification of abnormal tissue. In this context, “CAD” is an acronym for Computer Aided Detection or Computer Aided Diagnosis, with the CADe version indicating a version of computer aided detection configured to identify abnormal tissue, while the CADx version refers to a computer aided detection system configured to identify and classify detected abnormal tissue. Further description of CADe and CADx modules suitable for use with the present disclosure can be found in Pub. No. US 2023 / 0351592 Al to Filiciotto et al., titled “Clinical Decision Support System Having a Multi-ordered Hierarchy of Classification Modules,” the contents of which are incorporated herein by this reference. Further description of CADe and CADx modules suitable for use with the present disclosure can be found in Pub. No. US 2023 / 0379578 Al to Filiciotto et al., titled “Techniques for Simultaneously Running Discrete Imaging Modalities,” the contents of which are incorporated herein by this reference. As such, voice control module 306 can be used to control Al / CAD products such as Computer Aided Detection Products. The above approach is extensible to command lines that include actions performed by the VP and then related to the Al / CAD computer platform for execution. For example, user prompt like “Switch to NBI, Zoom on the Lesion and Turn on Characterization” could lead to a command sequence: NBI by CB — Zoom by the VP — CAD Characterization by the Al / CAD Compute Box — Display on Diagnostic Monitor.

[0067] Medical report generator module 324 can comprise a module configured to record patient-specific and procedure-specific information in computer readable storage mediums that include results from the medical procedure being performed. For example, video output of endoscope 304, as well as annotations made by CADe module 320 and CADx module 322, can be included in a medical record. The medical record can additionally include information from medical records database 326, which can include patient information, such as name, height, weight, sex, age, and the like. Medical report generator module 324 can downloadand upload data to medical records database 326 before and after a procedure has been performed, as well as during performance of the procedure.

[0068] As discussed herein, voice control module 306 can be configured to receive general and specific commands for operating one or more modules attached to control unit 302. For example, a user or endoscopist can provide a specific voice command to turn on a light from light source 317, and voice control module 306 can issue an appropriate specific instruction as described herein. Additionally, an user or endoscopist can provide a general voice command to begin operation of CADx module 322, and voice control module 306 can determine that multiple specific instructions are desired, such as to turn on CADx module 322, turn on an appropriate light generator or emitter at light source 317 that is compatible with the CADx module 322 operating mode, and turn on medical report generator module 324 to record identified tissue, for example.

[0069] FIG. 6 illustrates block diagram 400 illustrating various operations for initializing and configuring voice control module 306 of FIG. 5. Block diagram 400 can include command module step 402, control prompt step 404, interface step 406 and command step 408.

[0070] Command module step 402 can comprise sub-steps 410 of entering equipment details and entering user profiles. At command module step 402, endoscopy suite 300 can set-up or be provided with specific user information and specific equipment information. For example, control unit 302 can be provided with the specific types of hardware to be used to perform a procedure. In examples, the make, model and firmware version of endoscope 304 can be provided to control unit 302. In examples, control unit 302 can automatically recognize the make, model and firmware version of endoscope 304 when endoscope 304 is put in communication with, e.g., plugged into, control unit 302. Additionally, user profiles for a particular user or endoscopist can be provided, e.g., entered into, voice control module 306 or control unit 302. The user profiles can include user preferences regarding settings of endoscopy suite 300, preferences for performing types of procedures, and preferred word usage and vocabulary usage.

[0071] Control prompt step 404 can comprise sub-steps 412 of establishing rule-sets, establishing look-up tables and enabling Recommender Systems. At control prompt step 404, endoscopy suite 300 can be configured for specific voice prompt or voice command libraries. Control unit 302 can load appropriate control prompts for the particular equipment being used therewith. Control unit 302 can have specific rule-sets and / or look-up tables stored in memory for various equipment expected to be used with endoscopy suite 300. Additionally, control unit 302 can utilize Manchin learning methods, such as recommender systems. Therecommender system can comprise an artificial intelligence engine that assists the endoscopist find commands for various endoscopy system features and capabilities by suggesting voice prompts. In examples, voice control module 306 can suggest command prompts when an announced command prompt is not intelligible or does not correspond to a command prompt in a look-up table or rule-set. In examples, voice control module 306 can suggest command prompts as follow-ups to issued command prompts or command prompts for suggested next-steps. Depending on the list of peripherals connected to control unit 302, the rule-sets and / or look-up tables can vary for different systems. For example, different endoscopes can have different rule-sets and look-up tables for different capabilities of the endoscope and / or different procedures to be performed. For example, some endoscopy procedures can use insufflation gas, while others use a lavage saline for their respective fluid systems. Control unit 302 can include a graphical user interface (GUI) that is designed for clinicians to input or select voice command modifications or pre-set macros, which can then be processed by a command mapping module within control unit 302 that links these custom commands to standard system commands.

[0072] Interface step 406 can comprise sub-steps 414 of inputting and selecting commands. At interface step 406, endoscopy suite 300 can be configured to allow for procedure-specific customization. Control unit 302 can integrate endoscopy suite 300 with endoscopic health records (EHR) and leverage standardized data formats, such as Fast Healthcare Interoperability Resources (FHIR). Control unit 302 can retrieve procedure information from the EHR. Parsing referral indications (e.g., symptoms) or patient history, which can be obtained from medical records database 326, control unit 302 can generate a previously determined recommended prompt list to condition the rule-sets or look-up tables uploaded to the system. Control unit 302 can include an underlying database management system to allow for storage and retrieval of voice command sets tailored to different endoscopic procedures. Control unit 302 can use decision tree algorithms to suggest command sets based on input parameters, such as procedure type and patient history.

[0073] Command step 408 can comprise sub-steps 416 of generating and inputting custom command steps. At command step 408, endoscopy suite 300 can be configured to allow for user-specific command prompts. Control unit 302 can allow for customization of voice commands for different types of endoscopic procedures, which can be stored in memory of control unit 302 or elsewhere, such as a cloud-base or server-based, storage system, for recurring use.

[0074] FIG. 7 illustrates block diagram chart 430 showing various operations in methods for generating individual user voice profiles for voice control module 306. Block diagram 400 can comprise learning operation 432, user operation 434, speech operation 436 and finalization operation 438.

[0075] At learning operation 432, voice control module 306 can operate sample learning sessions. User profiles can be built by conducting one or more sample learning sessions where users are prompted to read out loud tuning text. The tuning text can be used by voice control module 306 to learn how a specific user annunciates particular words, how fast they speak and the like. The tuning text can additionally include asking of questions as to how a user would call for a particular endoscopy suite function, thereby learning preferred vocabulary of the user. The tuning text can additionally include asking of questions as to how a user would perform a medical procedure, such as by asking for preferred techniques and subsequent steps.

[0076] At user operation 434, voice control module 306 can parse the tuning text read by the user to determine user-specific phonetic styles and voice profiles. The tuning text can include words that voice control module 306 is programmed to associate with performing a specific endoscopy suite function and voice control module 306 can match the annunciation by the user of the tuning text to those endoscopy suite functions. As mentioned, the tuning text can additionally by used to ascertain vocabulary and preferred techniques of the user.

[0077] At speech operation 436, voice control module 306 can adapt to individual speech patterns, accent variations, and command preferences using a combination of supervised and unsupervised learning techniques. These speech processing techniques include, but are not limited to, speech recognition algorithms and deep neural networks. For example, speech processing techniques can be used to determine the intent of a user or predict the next words of a user using speech recognition algorithms and deep neural networks.

[0078] At finalization operation 438, voice control module 306 can prepare a finalized user voice profile. The finalized user voice profile can be stored in computer readable medium of voice control module 306 and in other locations locally or in the cloud.

[0079] FIG. 8 illustrates block diagram 450 showing various operations for operation of voice control module 306 and endoscopy suite 300. Block diagram 450 can comprise operation 452, operation 454, operation 456 and operation 458, as well as sub-steps illustrated in FIG. 8. 1Voice Recording and Control Prompt Disambiguation

[0080] At operation 452, voice control module 306 can receive and record voice commands of a user. Voice control module 306 can perform active listening. Voice control module 306 can include an audio processing module with an acoustic model optimized for the clinical environment and that can be used for constant voice detection, reducing false positives through a dual-pass recognition system that filters out ambient background noise such as processor or equipment noise, alarms etc. The audio processing module can be deployed using hands-free Bluetooth technology using commercial off the shelf hardware or other wireless capabilities.

[0081] Voice control module 306 can include an initiation prompt. To offer better control to the endoscopist, optionally, a voice activity detection (VAD) algorithm can be implemented to detect the presence of speech and trigger a recording function of endoscopy suite 300, such as video recording systems 311, upon the initiation prompt. Initiation prompts can include short macros like “Hey VP” which are pre-configured to a choice of the user and can be unique and unambiguous. A specific word or phrase, like “Endosuite Wake” signals control unit 302 to prepare for receiving a control prompt. Optionally, to offer better control and data privacy, a dedicated haptic interface like an endobutton or a foot-switch can be used to activate the voice capture module.

[0082] Voice control module 306 can perform voice capture. A combination of Automatic Speech Recognition (ASR) engines can be utilized, with fallbacks and cross-referencing to improve accuracy across different accents and dialects. The individualization of user profiles during set-up can be leveraged to remove user specific accentuation and convert speech into standard machine understandable template voice. This can be achieved with techniques including but not limited to digital signal processing (DSP) and machine learning-based voice recognition. In examples, voice control module 306 can use industry-standard voice control and medical command languages to facilitate interoperability between different systems and devices.

[0083] Voice control module 306 can perform speaker source separation. Often in a procedure room, there could be multiple staff engaging in parallel conversations. To avoid voice control module 306 from being confounded, voice control module can perform source separation techniques to clearly disambiguate speech of the endoscopist and mask out speech of other staff in the room. This can allow for cleaner speech input to the subsequent command generation module. Optionally, multiple acoustic devices could be deployed across the room for better source separation. Voice control module 306 can therefore beconnected to multiple microphones, such as microphone 328, located at different locations within endoscopy suite 300.

[0084] Voice control module 306 can perform speech to text conversion. The disambiguated speech endoscopist command prompt is converted into text and further tokenized for subsequent processing.

[0085] Voice control module 306 can support multiple languages. To support multi-lingual, cross-lingual transfer learning approach can be utilized to extend the capability of voice control module 306 to different languages without extensive retraining for each new language.Natural Language Processing (NLP) and Command Translation

[0086] At operation 454, voice control module 306 can perform natural language processing and command translation. Voice control module 306 can perform command interpretations. Advanced NLP techniques involving deep learning models, such as Transformer-based architectures (e.g., BERT, GPT), can be utilized for semantic analysis and intent recognition from the spoken commands.

[0087] Voice control module 306 can perform clinical word-sense disambiguation. For acronyms and abbreviations using natural language processing techniques such as word embeddings, word-level Convolutional Neural Networks, bi-directional LSTM, Recurrent Neural Networks, Dictionary Learning, or other machine learning techniques (support vector machine, Bayesian models, decision trees, k-means clustering), or any other traditional text analyses and comparison techniques (for example, edit-based similarity metrics like the Levenshtein distance, token-based similarity metrics, sequence-based metrics, or phonetic approaches). For example, in the command prompt in the case of endoscope controls, synonymous terms such as “zoom,” “magnify,” “take a closer look,” etc. can be disambiguated to a standardized command “Zoom” that voice control module 306 can interpret as increasing the magnification.

[0088] Voice control module 306 can perform Named Entity Recognition (NER) to locate and classify named entities mentioned in unstructured free text into pre-defined categories such as endoscope control command, anatomical entity, clinical finding, imaging modality, imaging observation, procedure, non-anatomical entities etc. This can be performed by using linguistic-grammar techniques using natural language processing techniques such as word embeddings, word-level Convolutional Neural Networks, bi-directional LSTM, Recurrent Neural Networks, Conditional Random Fields, Dictionary Learning or other machine learning techniques (support vector machine, Bayesian models, decision trees, k-means clustering) orany other traditional text analyses techniques. For example, given a disambiguated prompt “Switch to NBI and zoom on the Polyp” in case of colonoscopy, named entity recognition groups and categorizes related words as shown in FIG. 8. Optionally, other fields contained within command prompt can be used as inputs to learning algorithms for this step.

[0089] Once a command prompt is identified, voice control module 306 can perform semantic matching to map semantically similar terms to a standardized lexicon. Voice control module 306 can determine the most semantically closest standardized term using available ontologies or privately defined dictionaries (e.g. a private company-specific list of commands, or a standard ontology). This can be done by using word embeddings, lexical syntactic analyses, dictionary lookup, topic modelling etc. with the use of natural language processing techniques such as word embeddings, word-level Convolutional Neural Networks, bi-directional LSTM, Recurrent Neural Networks, Conditional Random Fields, Dictionary Learning or other machine learning techniques (support vector machine, Bayesian models, decision trees, k-means clustering) or any other traditional text analyses techniques.Contextual embedding layers within the NLP models can take into account the current stage of the procedure and patient data, provided by the Electronic Health Record (EHR) integration, to process commands with greater precision.

[0090] Voice control module 306 and endoscopy suite 300 can include machine readable, non-transitory storage mediums and / or memory, including lookup tables and the like. Upon standardization, voice control module 306 can leverage the use of relational databases with SQL-like querying facilitates the mapping of interpreted commands to specific control actions through lookup tables.

[0091] Voice control module can additionally, optionally, include control integration features. Voice control module 306 can combine voice command with pre-existing endoscope controls like endobuttons for times when verbal commands may not be ideal, allowing for multimodal input for command sequence generation.

[0092] Voice control module 306 can include translation to device-specific control capabilities. Once the command is interpreted and verified, the Control Mapping Module can translate the commands of the endoscopist into the specific set of device instructions for the endoscopy equipment being used.Sanity and Safety Checks

[0093] At operation 456, voice control module 306 can perform sanity and safety checks. To ensure that issued commands are sensible and sequentialized correctly, also keeping patient safety, risk mitigation and VP minimum performance in view (e.g., performance to allowbasic functionality to complete or abort a procedure safely), voice control module 306 can run sanity checks prior to command execution.

[0094] Voice control module 306 can perform logical analysis. Decision logic algorithms assess the appropriateness of the command sequence in relation to the current procedure state, confirmed by cross-referencing with a procedure state machine. A virtual state machine, such as a Petrinets or a Markov Chain Model, can be executed to ensure that future state would be stable upon command execution. For example, if an endoscopist call for a particular volume of insufflation gas that is excessively large as cross-referenced with a procedure protocol uploaded to the system, voice control module 306 can prevent the occurrence of the insufflation and issue a warning at display 308 that the user has called for an excessive amount of insufflation and an instruction or request to call for a different amount of insufflation.

[0095] Voice control module 306 can perform error detection. Anomaly detection algorithms can be included in voice control module 306 to monitor and flag any deviations from expected command sequences, using statistical outlier detection methods. In case of an error, voice control module 306 can follow predefined protocols to resolve the issue, including alerting the endoscopist and reverting to a safe state.

[0096] Voice control module 306 can perform resolution protocols. Voice control module 306 can implement a fault-tolerant architecture with fallback mechanisms, guided by predefined protocols, to maintain system stability and alert the clinician in case of errors.

[0097] Voice control module 306 can perform critical function lockout. A rule-based expert system determines critical functions that should not be controlled via voice to prevent inadvertent triggering, ensuring compliance with safety regulations. The rule-based expert system can also check if the sequence includes any critical functions that should be locked out from voice control.

[0098] Voice control module 306 can perform patient safety evaluations. Voice control module 306 can evaluate whether the control sequence could potentially compromise patient safety or the essential performance of the video processor (VP).Confirmation with Endoscopist

[0099] At operation 458, voice control module 306 can perform confirmation and execution of the voice commands. Voice control module 306 can perform visual and / or auditory confirmations. Voice control module 306 can employ multi-modal signal processing to generate auditory and visual cues for command confirmation, using a feedback generation algorithm that can takes into account an attention state of the user.

[0100] Voice control module 306 can provide haptic feedback. The integration of haptic devices, such as endobuttons or foot switches, can be leveraged to provide confirmation, such as by causing vibration of the physical user input device.

[0101] Voice control module 306 can perform sequence execution and confirmation. Voice control module can perform command relaying. Leveraging standard system interfaces and APIs, the commands can be relayed to the Processing Unit, e.g., control unit 302, of the VP for execution.

[0102] Voice control module 306 can perform command execution. Command sequences can be processed by a real-time control system that can interface with video processing systems 318 via an execution middleware that translates software commands into hardware actions. Command execution can involve communication with the API of video processing systems 318, executing the specific function such as zoom, adjust lighting, capture image, start / stop recording, and other command-specific actions.

[0103] Voice control module 306 can perform execution confirmation. Feedback loops within voice control module 306 can confirm successful execution, and display 308 can display real-time status updates to the endoscopist. The VP state machine, e.g., image processing unit 42 of FIG. 2, can be updated post command execution. Commands like “Go back to previous state” or “Default View,” etc. can be leveraged for easy navigation to pre-set states or past states.Feedback, Logging, and Reinforcement

[0104] At operation 460, voice control module 306 can perform feedback, logging and learning loop functions. Voice control module 306 can include a data warehousing module that can be utilized for logging all interactions, with time-stamp and command-action mapping for traceability and accountability. The logging data can be leveraged for improving algorithms or for device diagnostics.

[0105] Voice control module 306 can perform reinforcement learning. Leveraged methods such a chain promoting, instruction based tuning can be used for few shot fine tuning of the underlying model and model conditioning as the system is used. Voice control module 306 can incorporate a continuous learning cycle using reinforcement learning algorithms to adjust the predictive models based on clinician feedback and interaction history.

[0106] Voice control module 306 can perform performance reviews. After the procedure, voice control module 306 can analyze the entire interaction log to identify any issues and areas for improvement.

[0107] Voice control module 306 can perform system updates. The findings from the analysis can be used to update voice control module 306 for better performance in future procedures.

[0108] FIG. 9 illustrates block diagram 500 illustrating an example method of performing a colonoscopy procedure using control unit 302 and / or voice control module 306. Block diagram 500 can comprise operation 502, operation 504, operation 506, operation 508, operation 510 and operation 512.

[0109] Operation 502 through operation 512 can comprise steps in a medical procedure and can correspond to predetermined steps, such as those found in endoscopic health records (EHR) and leverage standardized data formats, such as Fast Healthcare Interoperability Resources (FHIR).

[0110] Boxes above voice command interface 514 in FIG. 9 can comprise system instructinos or machine instructions, e.g., instructions configured to operate a machine, a device, a software component or module, issued by voice control module 306. Audio voice commands uttered by a user or endoscopist are shown below voice command interface 514 in FIG. 9. Thus, voice commands below voice command interface 514 can be received by a microphone and converted to specific machine or system instructions for a module of endoscopy suite 300 as discussed herein.[oni] At operation 502, an endoscopist can issue a wake command for voice control module 306. Voice control module 306 can respond to the wake command by preparing to receive and receiving an audio input. The endoscopist can instruct voice control module 306 what type of procedure is to be performed. Voice control module 306 can respond by issue an instruction to start endoscopy system 10 as well as any ancillary or associated modules. In the illustrated example, the endoscopist can call for a colonoscopy with Al support, and voice control module 306 can activate endoscopy system 10 and CADe module 320 and / or CADx module 322. In particular, CADe module 320 can be activated to facilitate locating polyps. During operation 502, the endoscopist can insert endoscope 304 into anatomy of the patient.

[0112] At operation 504, the endoscopist can issue a command informing voice control module 306 that the cecum has been intubated and that a picture is desired. Voice control module 306 can take a freeze frame of the video output of endoscope 304 using video recording systems 311. Additionally, the endoscopist can issue a command informing voice control module 306 to record a video output of endoscope 304 capturing images of the cecum. Voice control module 306 can issue commands to video recording system 311 to start and stop video recording. Still images and video recordings can be added to medicalrecords of the patient by medical report generator module 324 automatically by voice control module 306.

[0113] In examples, voice control module 306 can be programmed or provided with user preferences that can automatically start and stop video recording or the taking of a picture. Thus, in examples, the endoscopist can simply annunciate that the secum has been reached without specifically calling for a picture or video to be recorded. Voice control module 306 can automatically perform operation of video capabilities of endoscope 14 and video recording systems 311 once the endoscopist annunciates that the landmark or trigger anatomy has been reached. Voice control module 306 can also automatically take such action without preprogramming based on machine learning-based predictive analysis of the called for voice command and the recognize point in the procedure being performed.

[0114] At operation 506, the endoscopist can issue a command informing voice control module 306 that high power water jetting is desired. Voice control module 306 can respond by initiating a high power waterjet from fluid systems 316. Similarly, the endoscopist can issue a command informing voice control module 306 that a specific amount of insufflation gas is desired. Voice control module 306 can respond by initiating a stream of insufflation gas from fluid systems 316.

[0115] Additionally, voice control module 306 can perform safety checks on the call for high power water jetting and insufflation. For example, if the high power jetting water is called for at a location in the anatomy where there is sensitive tissue, voice control module 306 can issue an warning to the endoscopist at display 308. Likewise, if the magnitude, e.g., volume or time of dispensing, the insufflation gas called for is too high, voice control module 306 can issue a warning to the endoscopist at display 308. Voice control module 306 can prevent operation of fluid systems 316 if the determined safety potential has been predetermined to be a high-level safety hazard where there is a chance for patient harm. However, voice control module 306 can allow continued operation of fluid systems 316 with confirmation from the endoscopist if the determined safety potential has been predetermined to be a low-level safety hazard where patient harm would be avoided.

[0116] At operation 508, the endoscopist can issue a command informing voice control module 306 that a polyp has been identified, that a picture of the polyp is desired and that the CADe module 320 and / or CADx module 322 is no longer needed. Voice control module 306 can respond by issuing instructions to video recording system 311 to take a picture of the polyp and to turn off the CADe module 320 and / or CADx module 322.

[0117] As discussed with reference to operation 504, voice control module 306 can be configured to automatically perform the picture taking operation and the powering down of the CAD module(s), particularly CADe module 320, without voice input from the endoscopist. As such, the voice command identifying the polyp can be considered a primary or main command, and the call for the picture and the change in power state of the CAD module can be considered secondary or ancillary commands. Voice control module 306 can determine or recognize that a voice command identifying a polyp can comprise an intention of the user to take a picture. Furthermore, the recognition by voice control module 306 that a poly has been identified an additionally inform voice control module 306 that the procedure completed a task, e.g., identifying a polyp, and steps to finish the procedure can begin, such as powering down the CAD module.

[0118] After operation 508, the endoscopist can manually remove the polyp using a biopsy device, such as the one described with reference to FIG. 4. Additionally, the endoscopist can use voice control module 306 to fully or partially automate the operation of the biopsy device, as discussed with reference to FIG. 4.

[0119] At operation 510, the endoscopist can issue a command informing voice control module 306 that the polyp has been removed, that a picture of the removed polyp is desired and that that CADx module 322 is desired to be used. Thus, CADx module 322 can be used to diagnose the removed polyp as being benign, precancerous or cancerous, for example. Voice control module 306 can respond by issuing instructions to video recording system 311 to take a picture of the removed polyp and to turn on the CADe module 320 and / or CADx module 322. As discussed herein, voice control module 306 can automatically perform one or more of the machine or system instructions issued at operation 510 based on only a single voice command input from the endoscopist.

[0120] At operation 512, the endoscopist can issue a command informing voice control module 306 that the rectum has been reached with the withdrawal of endoscope 304 and that it a picture of the rectum is desired. Voice control module 306 can respond by issuing instructions to video recording systems 311 to take a freeze frame or video of the output of endoscope 304 with the rectum in view. Additionally, the endoscopist can inform voice control module 306 that endoscopy suite 300 can be turned off, and voice control module 306 can issue multiple commands to various components of endoscopy suite 300, such as CADx module 322 and control unit 302, to power down and turn off endoscopy system 10.

[0121] FIG. 10 is a block diagram illustrating method 600 including operation 602 through operation 624 in performing methods of for operation of endoscopy suite 300 with voicecontrol module 306. Though discussed with reference to FIG. 1 through FIG. 9 and a particular endoscopy suite, endoscopy system and voice control module, method 600 can encompass the use of any endoscopy suite, endoscopy system and endoscopy voice control module consistent with the methods and systems described herein. Method 600 can additionally include fewer or greater operations other than operation 602 to operation 624. Additionally, in other examples, operation 602 through operation 624 can be performed in other sequences.

[0122] At operation 602, voice control module 306 can receive a natural language voice command, e.g., an utterance of an audible sound wave, from a user intending to operate one or more modules of endoscopy suite 300. Voice control module 306 can include means for converting a sound wave into an electronic signal, such as a microphone. In other examples, control unit 302 can include a microphone to convert sound wave into an electronic signal.

[0123] At operation 604, voice control module 306 can convert the electronic signal of the natural language voice command into a text command. In examples, voice control module 306 can perform a literal voice-to-text conversion to determine the exact words uttered by the user.

[0124] In examples, operation 602 and operation 604 can encompass operation 452 of FIG. 8.

[0125] At operation 606, voice control module 306 can disambiguate medical words in the text version of the voice command. Interchangeable medical words can be correlated to a common word. For example, voice control module 306 can convert a slang version of a specific medical term into the commonly accepted version.

[0126] At operation 608, voice control module 306 can disambiguate similar words in the text version of the voice command. Interchangeable words can be correlated to a common word. For example, voice control module 306 can determine if a word in the text command is a synonym of a word within a voice command library stored in computer readable memory accessible to voice control module.

[0127] In examples, operation 606 and operation 608 can encompass operation 454 of FIG. 8.

[0128] At operation 610, voice control module 306 can map words in the text command to a standardized lexicon. The standardized lexicon can comprise words for commands that are recognized by voice control module 306 that are correlated to specific functions of endoscopy suite 300.

[0129] At operation 612, voice control module 306 can convert the words of the standardized lexicon into machine instructions for endoscopy suite 300. The machine or systeminstructions can include information to cause an endoscopy suite module to perform its function.

[0130] In examples, operation 610 and operation 612 can encompass operation 454 of FIG. 8.

[0131] At operation 614, voice control module 306 can parse the machine instructions into specific instruction for modules of endoscopy suite 300. That is, an instruction in the voice command of operation 602 can include explicit or implicit instruction to operate one or more modules of endoscopy suite 300. Explicit instructions can be mapped one-to-one to specific endoscopy suite modules. Implicit instructions can comprise one or more unspoken instructions related to an overarching general voice command that can be mapped to multiple endoscopy suite modules.

[0132] At operation 616A, control unit 302 and / or voice control module 306 can prepare a first instruction, or first sub-command, to operate a first module of endoscopy suite 300. At operation 616B, control unit 302 and / or voice control module 306 can prepare a second instruction, or second sub-command, to operate a second module of endoscopy suite 300. In examples, the voice command of operation 602 can result in control unit 302 and / or voice control module 306 determining that multiple operations are needed or desired to comply with or execute the requested command. In examples, the voice command of operation 602 can result in control unit 302 and / or voice control module 306 determining that multiple operations would be helpful to the endoscopist to save time and alleviate the need for the endoscopist to request multiple voice commands. In such case, control unit 302 and / or voice control module 306 can automatically perform operation 618B without a specific voice command input for operation 618B and operation 618B can comprise an ancillary operation. In examples, the voice command from operation 602 can result in only one instruction for a single module being generated. In such case, method 600 can proceed to operation 618 without performing operation 618B.

[0133] At operation 618, control unit 302 and / or voice control module 306 determine the safety of executing the specific instruction for a module of endoscopy suite 300 determined at operation 614. For example, voice control module 306 can determine if a function of endoscopy suite 300 to be performed would be harmful or potentially harmful to the patient or if an existing setting of endoscopy suite 300 is incompatible with the operation of the module of endoscopy suite 300.

[0134] In examples, operation 618 can encompass operation 456 of FIG. 8.

[0135] At operation 620, control unit 302 and / or voice control module 306 can issue the specific module instruction from operation 616A and operation 616B to the specific module,if operation 618 results in a passing of the safety check. If the safety check is not passed, method 600 can proceed to operation 624. In examples, control unit 302 and / or voice control module 306 can deliver individual instructions directly to individual modules. In examples, voice control module 306 can issue multiple instructions to a first module, and the first module can relay individual instructions to a second module. As discussed, modules of system 300 can be connected in a mesh network such that instructions generated by control unit 302 and / or voice control module 306 can be delivered to any one of the modules and the instructions can be routed via direct connection or bridging so that instructions for specific modules can be routed to such module via a plurality of different routes.

[0136] At operation 622, voice control module 306 can remain in standby mode to listen for further voice commands.

[0137] At operation 624, voice control module 306 can issue instructions to endoscopy system 10 to provide warnings or instructions to a user. For example, display 308 of endoscopy suite 300 can provide a visual warning that a requested command might result in or could potentially result in an unsafe procedure being performed. Thereafter, control unit 302 can request a confirmation of the voice command or request that a different voice command be issued.

[0138] FIG. 11 A is a first state diagram of endoscopy suite 300 of FIG. 5 showing voice control module 306 issuing instructions to multiple cloud-based modules. In the illustrated example, voice control module 306 is shown communicating with CADx module 322 and medical report generator module 324. Specifically, voice control module 306 can issue communication signal 330 to CADx module 322 and can issue communication signal 332 to medical report generator module 324. Additionally, control unit 302 is illustrated issuing voice signal 334 to voice control module 306. In additional examples, voice control module 306 can issue communication signal 330 and communication signal 332 to any one of modules connected thereto, and any of those modules can distribute communication signal 330 and communication signal 332 to the appropriate modules. In examples, voice control module 306 can issue communication signal 330 and communication signal 332 to CADx module 322, and CADx module 322 can issue communication signal 332 to medical report generator module 324.

[0139] During use, an operator or user of endoscopy suite 300 can utter an audible voice command. The voice command can be in the form of a natural language word structure. The voice command can be received by microphone 328, which can convey the voice command to control unit 302. Control unit 302 can pass an electronic version of the voice command tovoice control module 306. In additional examples, voice control module 306 can be configured to directly receive the voice command through a microphone in communication with voice control module 306. As discussed below, voice control module 306 can convert the natural language audio command into instructions for operating one or more modules of endoscopy suite 300. Additionally, as mentioned, control unit 302 can without a separate voice control module convert the natural language audio command into instructions for operating the modules.

[0140] In the illustrated example of FIG. 11 A, the user can issue a voice command calling for a single operation of a cloud-based service, and voice control module 306 can parse the natural language voice command into a plurality of instructions for operating multiple modules of endoscopy suite 300. Specifically, the user can issue a voice command for operation of CADx module 322, and voice control module 306 can issue multiple instructions for operation of CADx module 322 as directly called for, and instructions for medical report generator module 324 as voice control module 306 has determined to be indirectly desirable.

[0141] The user can utter an audio command to begin operation of CADx module 322. CADx module 322 can begin to process output of endoscope 304. For example, CADx module 322 can analyze video output of endoscope 304 to identify polyps in imaging of a colon. Medical report generator module 324 can additionally begin to record output of CADx module 322 to include identified and / or diagnosed polyps in a medical report for the patient. Voice control module 306 can automatically issue communication signal 332 without a specific instruction from a user to record output of CADx module 322. For example, voice control module 306 can determine that operation of CADx module 322 to identify and / or diagnose tissue would result in the desirability of saving a picture of any identified polyps in a medical record for documentation and future reference. Thus, in examples, CADx module 322 can be operated by voice control module 306 to identify polyps and medical report generator module 324 can be activated to record images of any identified polyps in a medical report. In this scenario, operation of medical report generator module 324 and generation of communication signal 332 can be an ancillary operation. In examples, CADx module 322 may identify the polyp as being benign and the endoscopist may decide to proceed with other parts of the procedure, e.g., looking for other polyps, without changing a state of the endoscope, the automatic logging of the benign polyp by voice control module 306 allows the procedure to continue in a more expedient fashion without input from the user specifically requesting that the polyp be annoted in the medical record, e.g., “log polyp for monitoring.”

[0142] FIG. 1 IB is a second state diagram of endoscopy suite 300 of FIG. 5 showing voice control module 306 issuing instructions to a local hardware module and a cloud-based module. In the illustrated example, voice control module 306 is shown communicating with CADx module 322 and light source 317. Specifically, voice control module 306 can issue communication signal 340 to CADx module 322 and can issue communication signal 342 to light source 317. Additionally, control unit 302 is illustrated issuing voice signal 344 to voice control module 306. In additional examples, voice control module 306 can issue communication signal 340 and communication signal 342 to any one of modules connected thereto, and any of those modules can distribute communication signal 340 and communication signal 342 to the appropriate modules. In examples, voice control module 306 can issue communication signal 340 and communication signal 342 to CADx module 322, and CADx module 322 can issue communication signal 342 to light source 317.

[0143] During use, an operator or user of endoscopy suite 300 can utter an audible voice command. The voice command can be in the form of a natural language word structure. The voice command can be received by microphone 328, which can convey the voice command to control unit 302. Control unit 302 can pass an electronic version of the voice command to voice control module 306. In additional examples, voice control module 306 can be configured to directly receive the voice command through a microphone in communication with voice control module 306. As discussed below, voice control module 306 can convert the natural language audio command into instructions for operating one or more modules of endoscopy suite 300. Additionally, as mentioned, control unit 302 can without a separate voice control module convert the natural language audio command into instructions for operating the modules.

[0144] In the illustrated example of FIG. 1 IB, the user can issue a voice command calling for an operation of a cloud-based service, and voice control module 306 can parse the natural language voice command into a plurality of instructions for operating multiple modules of endoscopy suite 300. Specifically, the user can issue a voice command for operation of CADx module 322, and voice control module 306 can issue multiple instructions for operation of CADx module 322 as directly called for, and instructions for light source 317 as voice control module 306 had determined to be indirectly desirable.

[0145] The user can utter an audio command to begin operation of CADx module 322. CADx module 322 can begin to process output of endoscope 304. For example, CADx module 322 can analyze video output of endoscope 304 to identify polyps in imaging of a colon. Light source 317 can additionally begin to output light of a type compatible withCADx module 322 to allow CADx module 322 to more readily identify the polyps. Voice control module 306 can automatically issue communication signal 342 without a specific instruction from a user to generate light output compatible with CADx module 322. For example, voice control module 306 can determine that improved or optimal operation of CADx module 322 can be accomplished by use of a particular type of light to facilitate CADx module 322 being able to identify and / or diagnose tissue. Thus, in examples, CADx module 322 can be operated by voice control module 306 to identify polyps and light source 317 can be activated to illuminate potential polyps with light compatible with training of CADx module 322. In this scenario, operation of light source 317 in this manner and generation of communication signal 342 can be an ancillary operation.

[0146] FIG. 11C is a third state diagram of endoscopy suite 300 of FIG. 5 showing voice control module 306 issuing instructions to multiple local hardware modules. In the illustrated example, voice control module 306 is shown communicating with light source 317 and display 308. Specifically, voice control module 306 can issue communication signal 350 to light source 317 and can issue communication signal 352 to display 308. Additionally, control unit 302 is illustrated issuing voice signal 354 to voice control module 306. In additional examples, voice control module 306 can issue communication signal 350 and communication signal 352 to any one of modules connected thereto, and any of those modules can distribute communication signal 350 and communication signal 352 to the appropriate modules. In examples, voice control module 306 can issue communication signal 350 and communication signal 352 to light source 317, and light source 317 can issue communication signal 352 to display 308.

[0147] During use, an operator or user of endoscopy suite 300 can utter an audible voice command. The voice command can be in the form of a natural language word structure. The voice command can be received by microphone 328, which can convey the voice command to control unit 302. Control unit 302 can pass an electronic version of the voice command to voice control module 306. In additional examples, voice control module 306 can be configured to directly receive the voice command through a microphone in communication with voice control module 306. As discussed below, voice control module 306 can convert the natural language audio command into instructions for operating one or more modules of endoscopy suite 300. Additionally, as mentioned, control unit 302 can without a separate voice control module convert the natural language audio command into instructions for operating the modules.

[0148] In the illustrated example of FIG. 11C, the user can issue a voice command calling for an operation of a on-premise hardware or software module, and voice control module 306 can parse the natural language voice command into a plurality of instructions for operating multiple modules of endoscopy suite 300. Specifically, the user can issue a voice command for operation of light source 317, and voice control module 306 can issue multiple instructions for operation of light source 317 as directly called for, and instructions for display 308 as voice control module 306 has determined to be indirectly desirable.

[0149] The user can utter an audio command to begin operation of light source 317. The audio command can include instructions to operate in two different types of light modes, such as white light and blue light. Light source 317 can begin to output the desired type of lights. Display 308 can begin to provide output from two different video signals. For example, output from a duodenoscope and a cholangioscope can be provided to display 308 in alternating frames. Display 208 can thus receive a command from voice control module 306 to divide the alternating frame into two separate video feeds for display in, for example, a picture-in-picture format on display 308. Voice control module 306 can automatically issue communication signal 352 to display 308 without a specific instruction from a user to inform display 308 that formatting of the received video signal is applicable. For example, voice control module 306 can determine that operation of light source 317 for a duodenoscope would result in the desirability of operating display in a picture-in-picture format to allow output of a duodenoscope and output of a cholangioscope to be viewed simultaneously.Thus, in examples, light source 317 can be operated by voice control module 306 to output multiple light types for a duodenoscope and a cholangioscope and display 308 can be activated to display images from the duodenoscope. In this scenario, operation of display 308 in this manner and generation of communication signal 352 can be an ancillary operation.

[0150] The present disclosure describes devices, systems and methods for allowing a fully voice-controlled endoscopy system. The benefits of the presently disclosed systems and associated methods are numerous.

[0151] Comprehensive Voice- Activated Control: The system provides voice-activated control across the entire endoscopy suite, allowing clinicians to command the video processor and other connected peripherals using natural language voice prompts.

[0152] Multi-Step Configurations: The system can decode complex voice commands and execute multi-step control sequences across various peripherals in the endoscopy suite. This allows for seamless adjustment of complex settings through simple voice commands.

[0153] Safety Checks: The voice control system includes built-in safety and sanity checks to ensure that the requested commands are safe and appropriate before execution. This helps prevent potentially harmful actions and maintains patient safety.

[0154] Natural Language Interface: The system utilizes advanced natural language processing techniques to interpret voice commands, allowing clinicians to use natural speech patterns rather than rigid command structures.

[0155] Customization: The voice control system can be trained using Al models to recognize an extensive set of commands, exceeding the limitations of physical button configurations. It also allows for customization of voice commands for different procedures and user preferences. It can be programmed to work with speech patterns of different users, vocabularies, and accents, and can be customized for user preferences, procedure types, and specific equipment.

[0156] Peripheral Device Control: The system provides centralized voice-based control of multiple connected peripherals such as AI / CAD modules, surgical instruments, energy devices, video recording devices, and augmented reality solutions.

[0157] Multi -Module Control: The system can parse a single voice command into instructions for multiple modules of the endoscopy suite, including both explicitly requested actions and implicit related actions.

[0158] Automated Actions: The system can automatically perform secondary or ancillary actions based on a primary voice command, saving time and reducing the need for multiple voice inputs.

[0159] Integration with Multiple Systems: It can control various modules of the endoscopy suite, including local hardware, software, and cloud-based modules.

[0160] Hands-Free Operation: By enabling voice control, the system eliminates the need for clinicians to manually interact with controls, maintaining sterility and allowing them to focus on the procedure.

[0161] Improved Workflow: The voice control capabilities streamline the endoscopy process by allowing clinicians to adjust settings and control devices without interrupting their workflow or diverting attention from the patient.

[0162] Enhanced Focus: By enabling hands-free control, the system allows endoscopists to remain visually and mentally focused on the endoscopic image and the patient, potentially reducing procedural times and improving outcomes.

[0163] Procedure-specific Customization: The module can integrate with electronic health records to retrieve procedure information and generate recommended prompt lists.

[0164] Continuous Learning: The voice control system incorporates reinforcement learning algorithms to continuously improve its performance based on user feedback and interaction history.

[0165] Multi-modal Feedback: It can provide visual, auditory, and haptic feedback to confirm command execution.

[0166] Error Detection and Resolution: It includes anomaly detection algorithms to flag deviations from expected command sequences and implements fault-tolerant architecture with fallback mechanisms.

[0167] These features and benefits collectively aim to enhance the efficiency, safety, and effectiveness of endoscopic procedures by providing a more intuitive and hands-free control interface for clinicians.Examples

[0168] Example 1 is a control system of an endoscopy ecosystem having an endoscopy system and at least one peripheral system, the control system comprising: a plurality of modules for controlling capabilities of the endoscopy ecosystem provided by one or more of the endoscopy system and the at least one peripheral system; and the control system is configured to: receive a voice command from a user to enable a desired functionality of the endoscopy ecosystem; decode the voice command into a system instruction for controlling the desired functionality; parse the system instruction into: a first sub-command for a first identified module of the plurality of modules for controlling a first aspect of the desired functionality; and a second sub-command for a second identified module of the plurality of modules for controlling a second aspect the desired functionality; and transmit the first subcommand to the first identified module and the second sub-command to the second identified module, wherein transmission of the first sub-command and the second sub-command cause the endoscopy ecosystem to execute the desired functionality.

[0169] In Example 2, the subject matter of Example 1 optionally includes wherein the control system automatically generates the second sub-command to enable functionality of the second identified module that is compatible with the desired functionality.

[0170] In Example 3, the subject matter of any one or more of Examples 1-2 optionally include wherein the control system automatically generates the second sub-command to eliminate a need for the user to issue another voice command.

[0171] In Example 4, the subject matter of any one or more of Examples 1-3 optionally include wherein: the voice command comprises a general command for the desired functionality of the endoscopy ecosystem; the first sub-command comprises a command forthe first identified module to directly execute the desired functionality; and the second subcommand comprises a command for the second identified module to indirectly support the desired functionality with an ancillary functionality of the endoscopy ecosystem.

[0172] In Example 5, the subject matter of any one or more of Examples 1-4 optionally include wherein the plurality of modules comprises: a first module for controlling a first capability of the endoscopy system; a second module for controlling a second capability of the endoscopy system; a third module for controlling a first peripheral system; and a fourth module for controlling a second peripheral system.

[0173] In Example 6, the subject matter of Example 5 optionally includes an endoscope of the endoscopy system, the endoscope comprising: a shaft; a working channel; an imaging unit; and a lighting unit; wherein: the first module comprises an imaging module for controlling the imaging unit of the endoscope; and the second module comprises a lighting module for controlling the lighting unit of the endoscope.

[0174] In Example 7, the subject matter of Example 6 optionally includes wherein: the third module comprises a computer-aided detection or computer-aided diagnosis (CAD) module for processing images of the imaging unit of the endoscope; and the fourth module comprises a medical report generator.

[0175] In Example 8, the subject matter of any one or more of Examples 1-7 optionally include wherein: the first sub-command comprises a command for a first functionality of a first cloud-based peripheral system of the endoscopy ecosystem; and the second subcommand comprises a command for a second functionality of a second cloud-based peripheral system of the endoscopy ecosystem.

[0176] In Example 9, the subject matter of any one or more of Examples 1-8 optionally include wherein: the first sub-command comprises a command for a first functionality of a first peripheral system; and the second sub-command comprises a command for a second functionality of a second peripheral system.

[0177] In Example 10, the subject matter of Example 9 optionally includes wherein: the first functionality of the first peripheral system comprises operating a CAD module to locate an anatomic structure in one or more images of an imaging unit of an endoscope; and the second functionality of the second peripheral system comprises operating a medical report generator module to record a location of a polyp in an image from the imaging unit.

[0178] In Example 11, the subject matter of any one or more of Examples 1-10 optionally include wherein: the first sub-command comprises a command for a first functionality of a first cloud-based peripheral system of the endoscopy ecosystem; and the second sub-command comprises a command for a first functionality of a first local peripheral system of the endoscopy ecosystem.

[0179] In Example 12, the subject matter of Example 11 optionally includes wherein: the first sub-command comprises a command for a first functionality of a first peripheral system; and the second sub-command comprises a command for a first functionality of the endoscopy system.

[0180] In Example 13, the subject matter of Example 12 optionally includes wherein: the first functionality of the first peripheral system comprises operating a CAD module to locate an anatomic structure in one or more images of an imaging unit of an endoscope; and the first functionality of the endoscopy system comprises operating a light generator of the endoscopy system to generate light compatible with the CAD module to facilitate locating the anatomic structure.

[0181] In Example 14, the subject matter of any one or more of Examples 1-13 optionally include wherein: the first sub-command comprises a command for a first functionality of a local peripheral system of the endoscopy ecosystem; and the second sub-command comprises a command for a second functionality of a second local peripheral system of the endoscopy ecosystem.

[0182] In Example 15, the subject matter of Example 14 optionally includes wherein: the first sub-command comprises a command for a first functionality of the endoscopy system; and the second sub-command comprises a command for a second functionality of the endoscopy system.

[0183] In Example 16, the subject matter of Example 15 optionally includes wherein: the first functionality of the endoscopy system comprises operating a light generator of the endoscopy system; and the second functionality of the endoscopy system comprises operating a display of the endoscopy system.

[0184] In Example 17, the subject matter of any one or more of Examples 1-16 optionally include wherein the control system is further configured to locate system instructions for the first sub-command and the second sub-command within look-up tables relating capabilities of the first identified module with capabilities of the second identified module to ensure compatibility.

[0185] In Example 18, the subject matter of any one or more of Examples 5-17 optionally include wherein the control system comprises: a non-transitory computer readably storage medium having stored therein: steps for performing a medical procedure with the endoscopy system; safety parameters for various steps for performing the medical procedure; andinstructions for overriding certain voice commands that could potentially violate the safety parameters.

[0186] In Example 19, the subject matter of Example 18 optionally includes wherein the control system comprises a warning system configured to issue one or more of an audio alert and a visual alert to a user before a safety parameter is violated.

[0187] In Example 20, the subject matter of any one or more of Examples 18-19 optionally include wherein: the medical procedure comprises a colonoscopy of a colon; the first capability comprises insufflation of the colon; and the voice control module is configured to disable an insufflation device if a magnitude of insufflation that is called for could potentially perforate the colon.

[0188] Example 21 is a method for operating an endoscopy ecosystem comprising an endoscopy system and at least one peripheral system for the endoscopy system, the method comprising: receiving a voice command from a user at a control system, the voice command configured to activate a desired functionality of the endoscopy ecosystem; decoding the voice command into a system instruction for operating modules of the endoscopy ecosystem to execute the desired functionality of the endoscopy ecosystem; parsing the system instruction into: a first sub-command for a first identified module of the modules for controlling a first aspect of the desired functionality; and a second sub-command for a second identified module of the modules for controlling a second aspect the desired functionality; and transmitting the first sub-command to the first identified module and the second subcommand to the second identified module, wherein transmission of the first sub-command and the second sub-command cause the endoscopy ecosystem to execute the desired functionality.

[0189] In Example 22, the subject matter of Example 21 optionally includes automatically generating the second sub-command to enable functionality of the second identified module that is compatible with the desired functionality.

[0190] In Example 23, the subject matter of any one or more of Examples 21-22 optionally include automatically generating the second sub-command to eliminate a need for the user to issue another voice command.

[0191] In Example 24, the subject matter of any one or more of Examples 21-23 optionally include wherein: receiving the voice command comprises receiving a general command for the desired functionality of the endoscopy ecosystem; parsing the first sub-command comprises generating a command for the first identified module to directly execute the desired functionality; and parsing the second sub-command comprises generating a commandfor the second identified module to indirectly support the desired functionality with an ancillary functionality of the endoscopy ecosystem.

[0192] In Example 25, the subject matter of any one or more of Examples 21-24 optionally include controlling a first capability of the endoscopy system with a first module; controlling a second capability of the endoscopy system with a second module; controlling a first peripheral system with a third module; and controlling a second peripheral system with a fourth module.

[0193] In Example 26, the subject matter of Example 25 optionally includes providing an endoscope of the endoscopy system, the endoscope comprising: a shaft; a working channel; an imaging unit; and a lighting unit; wherein: controlling the first capability comprises controlling the imaging unit of the endoscope with an imaging module; and controlling the second capability comprises controlling the lighting unit of the endoscope with a lighting module.

[0194] In Example 27, the subject matter of Example 26 optionally includes wherein: controlling the first peripheral system comprises processing images of the imaging unit of the endoscope with a computer-aided detection or computer-aided diagnosis (CAD) module; and controlling the second peripheral system comprises generating medical reports with a medical report generator.

[0195] In Example 28, the subject matter of any one or more of Examples 21-27 optionally include wherein: transmitting the first sub-command comprises commanding a first functionality of a first cloud-based peripheral system of the endoscopy ecosystem; and transmitting the second sub-command comprises commanding a second functionality of a second cloud-based peripheral system of the endoscopy ecosystem.

[0196] In Example 29, the subject matter of any one or more of Examples 21-28 optionally include wherein: transmitting the first sub-command comprises commanding a first functionality of a first peripheral system; and transmitting the second sub-command comprises commanding a second functionality of a second peripheral system.

[0197] In Example 30, the subject matter of Example 29 optionally includes wherein: commanding the first functionality of the first peripheral system comprises operating a CAD module to locate an anatomic structure in one or more images of an imaging unit of an endoscope; and commanding the second functionality of the second peripheral system comprises operating a medical report generator module to record a location of a polyp in an image from the imaging unit.

[0198] In Example 31, the subject matter of any one or more of Examples 21-30 optionally include wherein: transmitting the first sub-command comprises commanding a first functionality of a first cloud-based peripheral system of the endoscopy ecosystem; and transmitting the second sub-command comprises commanding a first functionality of a first local peripheral system of the endoscopy ecosystem.

[0199] In Example 32, the subject matter of Example 31 optionally includes wherein: transmitting the first sub-command comprises commanding a first functionality of a first peripheral system; and transmitting the second sub-command comprises commanding a first functionality of the endoscopy system.

[0200] In Example 33, the subject matter of Example 32 optionally includes wherein: commanding the first functionality of the first peripheral system comprises operating a CAD module to locate an anatomic structure in one or more images of an imaging unit of an endoscope; and commanding the first functionality of the endoscopy system comprises operating a light generator of the endoscopy system to generate light compatible with the CAD module to facilitate locating the anatomic structure.

[0201] In Example 34, the subject matter of any one or more of Examples 21-33 optionally include wherein: transmitting the first sub-command comprises commanding a first functionality of a local peripheral system of the endoscopy ecosystem; and transmitting the second sub-command comprises commanding a second functionality of a second local peripheral system of the endoscopy ecosystem.

[0202] In Example 35, the subject matter of Example 34 optionally includes wherein: transmitting the first sub-command comprises commanding a first functionality of the endoscopy system; and transmitting the second sub-command comprises commanding a second functionality of the endoscopy system.

[0203] In Example 36, the subject matter of Example 35 optionally includes wherein: commanding the first functionality of the endoscopy system comprises operating a light generator of the endoscopy system; and commanding the second functionality of the endoscopy system comprises operating a display of the endoscopy system.

[0204] In Example 37, the subject matter of any one or more of Examples 21-36 optionally include locating system instructions for the first sub-command and the second sub-command within look-up tables relating capabilities of the first identified module with capabilities of the second identified module to ensure compatibility.

[0205] In Example 38, the subject matter of any one or more of Examples 25-37 optionally include storing in a non-transitory computer readable storage medium of the control system:steps for performing a medical procedure with the endoscopy system; safety parameters for various steps for performing the medical procedure; and instructions for overriding certain voice commands that could potentially violate the safety parameters.

[0206] In Example 39, the subject matter of Example 38 optionally includes issuing one or more of an audio alert and a visual alert to a user before a safety parameter is violated.

[0207] In Example 40, the subject matter of any one or more of Examples 38-39 optionally include wherein: performing the medical procedure comprises performing a colonoscopy of a colon; controlling the first capability comprises insufflating the colon; and disabling an insufflation device if a magnitude of insufflation that is called for could potentially perforate the colon.

[0208] Example 41 is a control system for and endoscopy ecosystem that can comprise a control unit for operating an endoscopy system comprising an endoscope and a plurality of modules for controlling capabilities of the endoscopy ecosystem, the control system configured to , receive a voice command from a user to enable a desired functionality of the endoscopy ecosystem, decode the voice command into a system instruction for controlling the desired functionality, parse the system instruction into: a first sub-command for a first module of the plurality of modules for controlling a first aspect of the desired functionality and a second sub-command for a second module of the plurality of modules for controlling a second aspect the desired functionality, and transmit the first sub-command and the second sub-command to the plurality of modules, wherein transmission of the first sub-command and the second sub-command cause the endoscopy ecosystem to execute the desired functionality.

[0209] Each of these non-limiting examples can stand on its own, or can be combined in various permutations or combinations with one or more of the other examples.Notes

[0210] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples can include elements in addition to those shown or described. However, the present inventor also contemplates examples in which only those elements shown or described are provided. Moreover, the present inventor also contemplates examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0211] In the event of inconsistent usages between this document and any documents so incorporated by reference, the usage in this document controls.

[0212] In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the terms “including” and “in which” are used as the plain- English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, composition, formulation, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0213] Method examples described herein can be machine or computer-implemented at least in part. Some examples can include a computer-readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform methods as described in the above examples. An implementation of such methods can include code, such as microcode, assembly language code, a higher-level language code, or the like. Such code can include computer readable instructions for performing various methods. The code may form portions of computer program products. Further, in an example, the code can be tangibly stored on one or more volatile, non-transitory, or nonvolatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media can include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memories (RAMs), read only memories (ROMs), and the like.

[0214] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments can be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimeddisclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that such embodiments can be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

CLAIMS:

1. A control system of an endoscopy ecosystem having an endoscopy system and at least one peripheral system, the control system comprising: a plurality of modules for controlling capabilities of the endoscopy ecosystem provided by one or more of the endoscopy system and the at least one peripheral system; and the control system configured to: receive a voice command from a user to enable a desired functionality of the endoscopy ecosystem; decode the voice command into a system instruction for controlling the desired functionality; parse the system instruction into: a first sub-command for a first identified module of the plurality of modules for controlling a first aspect of the desired functionality; and a second sub-command for a second identified module of the plurality of modules for controlling a second aspect the desired functionality; and transmit the first sub-command to the first identified module and the second sub-command to the second identified module, wherein transmission of the first sub-command and the second sub-command cause the endoscopy ecosystem to execute the desired functionality.

2. The control system of claim 1, wherein the control system automatically generates the second sub-command to enable functionality of the second identified module that is compatible with the desired functionality.

3. The control system of claim 1, wherein the control system automatically generates the second sub-command to eliminate a need for the user to issue another voice command.

4. The control system of claim 1, wherein:the voice command comprises a general command for the desired functionality of the endoscopy ecosystem; the first sub-command comprises a command for the first identified module to directly execute the desired functionality; and the second sub-command comprises a command for the second identified module to indirectly support the desired functionality with an ancillary functionality of the endoscopy ecosystem.

5. The control system of claim 1, wherein the plurality of modules comprises: a first module for controlling a first capability of the endoscopy system; a second module for controlling a second capability of the endoscopy system; a third module for controlling a first peripheral system; and a fourth module for controlling a second peripheral system.

6. The control system of claim 5, further comprising an endoscope of the endoscopy system, the endoscope comprising: a shaft; a working channel; an imaging unit; and a lighting unit; wherein: the first module comprises an imaging module for controlling the imaging unit of the endoscope; and the second module comprises a lighting module for controlling the lighting unit of the endoscope.

7. The control system of claim 6, wherein: the third module comprises a computer-aided detection or computer-aided diagnosis (CAD) module for processing images of the imaging unit of the endoscope; and the fourth module comprises a medical report generator.

8. The control system of claim 1, wherein:the first sub-command comprises a command for a first functionality of a first cloudbased peripheral system of the endoscopy ecosystem; and the second sub-command comprises a command for a second functionality of a second cloud-based peripheral system of the endoscopy ecosystem.

9. The control system of claim 1, wherein: the first sub-command comprises a command for a first functionality of a first peripheral system; and the second sub-command comprises a command for a second functionality of a second peripheral system.

10. The control system of claim 9, wherein: the first functionality of the first peripheral system comprises operating a CAD module to locate an anatomic structure in one or more images of an imaging unit of an endoscope; and the second functionality of the second peripheral system comprises operating a medical report generator module to record a location of a polyp in an image from the imaging unit.

11. The control system of claim 1, wherein: the first sub-command comprises a command for a first functionality of a first cloudbased peripheral system of the endoscopy ecosystem; and the second sub-command comprises a command for a first functionality of a first local peripheral system of the endoscopy ecosystem.

12. The control system of claim 11, wherein: the first sub-command comprises a command for a first functionality of a first peripheral system; and the second sub-command comprises a command for a first functionality of the endoscopy system.

13. The control system of claim 12, wherein:the first functionality of the first peripheral system comprises operating a CAD module to locate an anatomic structure in one or more images of an imaging unit of an endoscope; and the first functionality of the endoscopy system comprises operating a light generator of the endoscopy system to generate light compatible with the CAD module to facilitate locating the anatomic structure.

14. The control system of claim 1, wherein: the first sub-command comprises a command for a first functionality of a local peripheral system of the endoscopy ecosystem; and the second sub-command comprises a command for a second functionality of a second local peripheral system of the endoscopy ecosystem.

15. The control system of claim 14, wherein: the first sub-command comprises a command for a first functionality of the endoscopy system; and the second sub-command comprises a command for a second functionality of the endoscopy system.

16. The control system of claim 15, wherein: the first functionality of the endoscopy system comprises operating a light generator of the endoscopy system; and the second functionality of the endoscopy system comprises operating a display of the endoscopy system.

17. The control system of claim 1, wherein the control system is further configured to locate system instructions for the first sub-command and the second sub-command within look-up tables relating capabilities of the first identified module with capabilities of the second identified module to ensure compatibility.

18. The control system of claim 5, wherein the control system comprises: a non-transitory computer readably storage medium having stored therein: steps for performing a medical procedure with the endoscopy system; safety parameters for various steps for performing the medical procedure; andinstructions for overriding certain voice commands that could potentially violate the safety parameters.

19. The control system of claim 18, wherein the control system comprises a warning system configured to issue one or more of an audio alert and a visual alert to a user before a safety parameter is violated.

20. The control system of claim 18, wherein: the medical procedure comprises a colonoscopy of a colon; the first capability comprises insufflation of the colon; and the voice control module is configured to disable an insufflation device if a magnitude of insufflation that is called for could potentially perforate the colon.

21. A method for operating an endoscopy ecosystem comprising an endoscopy system and at least one peripheral system for the endoscopy system, the method comprising: receiving a voice command from a user at a control system, the voice command configured to activate a desired functionality of the endoscopy ecosystem; decoding the voice command into a system instruction for operating modules of the endoscopy ecosystem to execute the desired functionality of the endoscopy ecosystem; parsing the system instruction into: a first sub-command for a first identified module of the modules for controlling a first aspect of the desired functionality; and a second sub-command for a second identified module of the modules for controlling a second aspect the desired functionality; and transmitting the first sub-command to the first identified module and the second subcommand to the second identified module, wherein transmission of the first sub-command and the second sub-command cause the endoscopy ecosystem to execute the desired functionality.

22. The method of claim 21, further comprising automatically generating the second subcommand to enable functionality of the second identified module that is compatible with the desired functionality.

23. The method of claim 21, further comprising automatically generating the second subcommand to eliminate a need for the user to issue another voice command.

24. The method of claim 21, wherein: receiving the voice command comprises receiving a general command for the desired functionality of the endoscopy ecosystem; parsing the first sub-command comprises generating a command for the first identified module to directly execute the desired functionality; and parsing the second sub-command comprises generating a command for the second identified module to indirectly support the desired functionality with an ancillary functionality of the endoscopy ecosystem.

25. The method of claim 21, further comprising: controlling a first capability of the endoscopy system with a first module; controlling a second capability of the endoscopy system with a second module; controlling a first peripheral system with a third module; and controlling a second peripheral system with a fourth module.

26. The method of claim 25, further comprising: providing an endoscope of the endoscopy system, the endoscope comprising: a shaft; a working channel; an imaging unit; and a lighting unit; wherein: controlling the first capability comprises controlling the imaging unit of the endoscope with an imaging module; and controlling the second capability comprises controlling the lighting unit of the endoscope with a lighting module.

27. The method of claim 26, wherein: controlling the first peripheral system comprises processing images of the imaging unit of the endoscope with a computer-aided detection or computer-aided diagnosis (CAD) module; andcontrolling the second peripheral system comprises generating medical reports with a medical report generator.

28. The method of claim 21, wherein: transmitting the first sub-command comprises commanding a first functionality of a first cloud-based peripheral system of the endoscopy ecosystem; and transmitting the second sub-command comprises commanding a second functionality of a second cloud-based peripheral system of the endoscopy ecosystem.

29. The method of claim 21, wherein: transmitting the first sub-command comprises commanding a first functionality of a first peripheral system; and transmitting the second sub-command comprises commanding a second functionality of a second peripheral system.

30. The method of claim 29, wherein: commanding the first functionality of the first peripheral system comprises operating a CAD module to locate an anatomic structure in one or more images of an imaging unit of an endoscope; and commanding the second functionality of the second peripheral system comprises operating a medical report generator module to record a location of a polyp in an image from the imaging unit.

31. The method of claim 21, wherein: transmitting the first sub-command comprises commanding a first functionality of a first cloud-based peripheral system of the endoscopy ecosystem; and transmitting the second sub-command comprises commanding a first functionality of a first local peripheral system of the endoscopy ecosystem.

32. The method of claim 31, wherein: transmitting the first sub-command comprises commanding a first functionality of a first peripheral system; and transmitting the second sub-command comprises commanding a first functionality of the endoscopy system.

33. The method of claim 32, wherein: commanding the first functionality of the first peripheral system comprises operating a CAD module to locate an anatomic structure in one or more images of an imaging unit of an endoscope; and commanding the first functionality of the endoscopy system comprises operating a light generator of the endoscopy system to generate light compatible with the CAD module to facilitate locating the anatomic structure.

34. The method of claim 21, wherein: transmitting the first sub-command comprises commanding a first functionality of a local peripheral system of the endoscopy ecosystem; and transmitting the second sub-command comprises commanding a second functionality of a second local peripheral system of the endoscopy ecosystem.

35. The method of claim 34, wherein: transmitting the first sub-command comprises commanding a first functionality of the endoscopy system; and transmitting the second sub-command comprises commanding a second functionality of the endoscopy system.

36. The method of claim 35, wherein: commanding the first functionality of the endoscopy system comprises operating a light generator of the endoscopy system; and commanding the second functionality of the endoscopy system comprises operating a display of the endoscopy system.

37. The method of claim 21, further comprising locating system instructions for the first sub-command and the second sub-command within look-up tables relating capabilities of the first identified module with capabilities of the second identified module to ensure compatibility.

38. The method of claim 25, further comprising: storing in a non-transitory computer readable storage medium of the control system:steps for performing a medical procedure with the endoscopy system; safety parameters for various steps for performing the medical procedure; and instructions for overriding certain voice commands that could potentially violate the safety parameters.

39. The method of claim 38, further comprising issuing one or more of an audio alert and a visual alert to a user before a safety parameter is violated.

40. The method of claim 38, wherein: performing the medical procedure comprises performing a colonoscopy of a colon; controlling the first capability comprises insufflating the colon; and disabling an insufflation device if a magnitude of insufflation that is called for could potentially perforate the colon.

41. A control system for and endoscopy ecosystem, the control system comprising: a control unit for operating an endoscopy system comprising an endoscope and a plurality of modules for controlling capabilities of the endoscopy ecosystem; the control system configured to: receive a voice command from a user to enable a desired functionality of the endoscopy ecosystem; decode the voice command into a system instruction for controlling the desired functionality; parse the system instruction into: a first sub-command for a first module of the plurality of modules for controlling a first aspect of the desired functionality; and a second sub-command for a second module of the plurality of modules for controlling a second aspect the desired functionality; and transmit the first sub-command and the second sub-command to the plurality of modules, wherein transmission of the first sub-command and the second sub-command cause the endoscopy ecosystem to execute the desired functionality.

Images