Systems, methods and apparatus for remote and autonomous control of flexible endoscopes

Abstract

Systems, methods, and devices are provided for robotic endoscopy using a robotically controlled dual-arm flexible endoscope having a first robotic arm that holds and controls a robotically controlled endoscope handle manipulator and a second robotic arm that holds and controls an endoscope actuator for advancing and retracting the distal tip of the endoscope. Each robotic arm is capable of multiple movements that essentially mimic the functions of a human arm and hand, allowing the flexible endoscope to be repositioned, twisted, and torqued as a human operator would to change the orientation of the flexible endoscope as it navigates through non-linear organ structures within an animal or human body cavity. The robotic arms can be mounted on a single base and controlled via a single controller mechanism, or can be mounted on separate bases with separate controller mechanisms, potentially allowing for increased angles of movement and positioning for each robotic arm.

Description

[Technical Field]

[0001] The systems, methods, and devices provided herein relate to remotely controlled endoscopes, and more particularly to dual-arm robotic endoscopes and integrated sensors for performing endoscopic procedures both automatically and autonomously. [Background technology]

[0002] Endoscopic surgery is now the leading option for performing minimally invasive surgery because it allows for the use of small incisions to insert a scope and a flexible tube carrying multiple tools that enable surgeons to perform multiple procedures. The tools typically include a camera and a light, allowing surgeons to visualize sites inside the body without making large incisions. The functionality of each endoscope is often highly complex, requiring multiple operators or dedicated controls, such as foot pedals, to allow surgeons to carefully control the movement and manipulation of the tools simultaneously.

[0003] While the early field of endoscopy traditionally involved only inserting scopes and tubes into existing openings, it has now expanded to include multiple disciplines, including surgery and interventional radiology, and the types of procedures that can be completed via endoscopy are expanding rapidly as well. As the types of procedures performed become more complex, interventional physicians now must learn multiple disciplines in order to select the best way to solve a problem.

[0004] Recent improvements have attempted to automate some of the control and movement of flexible endoscopes to reduce the complexity of the surgeon's procedure or to enable remote controlled operation. This includes enabling remote controlled operation of the endoscope using actuators and manipulators to control all of the flexible endoscope's functions, from depth within the body to direction of movement, operation of tools, cameras, lights, etc. Some of these improvements are described in U.S. Patent Nos. 5,629,999 and 5,729,999, which are incorporated herein by reference. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Patent No. 8,409,080 [Patent Document 2] U.S. Patent No. 9,706,907 Summary of the Invention [Problem to be solved by the invention]

[0006] However, as the types of procedures become more complex and more advantageously performed via remote control, there remains a need for improved capabilities to navigate and control flexible endoscopes. As remote control options improve, additional options for autonomous control of endoscopes may be considered. [Means for solving the problem]

[0007] Embodiments described herein provide systems, methods, and devices for robotic endoscopy using a robotically controlled, dual-arm flexible endoscope having a first robotic arm that holds and controls a robotically controlled endoscope handle manipulator and a second robotic arm that holds and controls an endoscope actuator for advancing and retracting the distal tip of the endoscope. Each robotic arm is capable of multiple movements that essentially mimic the functions of a human arm and hand, allowing the flexible endoscope to be repositioned, twisted, and torqued as a human operator would to change the orientation of the flexible endoscope as it navigates through non-linear organ structures within an animal or human body cavity. The robotic arms can be mounted on a single base and controlled via a single controller mechanism, or they can be mounted on separate bases with separate controller mechanisms, potentially allowing for increased angles of movement and positioning for each robotic arm.

[0008] Manual control of the endoscope handle knobs can be achieved via foot pedals, and when using a robotically controlled guidewire, a seat that allows each foot to control a separate function via a separate foot pedal can also be beneficial. Alternatively, the console can be constructed to provide greater control options.

[0009] The controller mechanism can be remotely operated for remote control of complex endoscopic procedures and can also perform automated operations by automatically performing designated functions using a combination of sensors and tools on the flexible endoscope through computer-assisted or image-guided surgery and intervention. Additionally, the device and system also enable artificial intelligence surgery, where one or more controllers enable autonomous navigation and operation through reinforcement learning algorithms and machine learning.

[0010] Robotically controlled flexible endoscopes may also be adaptable for use with non-traditional endoscopes that have wider channels that allow for more complex functionality of single-use endoscopes and grips, as well as larger guidewires and endoscope tips. These non-traditional endoscopes may be modified to enhance their autonomous functionality, such as by including shape-sensing fibers to measure the shape and length of the endoscope.

[0011] In one embodiment, a robotically controlled dual-arm flexible endoscope includes a flexible endoscope having a shaft, a proximal handle, and a distal tip; a first robotic arm having a proximal handle fixed thereto, the first robotic arm configured to move the proximal handle of the flexible endoscope; an endoscope handle manipulator connected to the first robotic arm and the proximal handle and configured to control the flexible endoscope; a second robotic arm having an endoscope actuator fixing the distal tip and configured to advance or withdraw the flexible endoscope into or from a body cavity; and a controller in communication with the first robotic arm and the second robotic arm to control the movement and position of each of the robotic arms, the handle manipulator, and the endoscope actuator.

[0012] In another embodiment, a method for performing an endoscopic procedure using a robotically controlled flexible endoscope includes the steps of controlling, via a controller, the movement and function of an endoscope handle manipulator connected to a proximal handle of the flexible endoscope via a first robotic arm; controlling, via the controller, the movement and function of an endoscope actuator to which a distal tip of the flexible endoscope is fixed via a second robotic arm; and performing the endoscopic procedure via the movement and control of the flexible endoscope by the controller.

[0013] In a further embodiment, a method of screening for disease using an autonomous flexible endoscope includes inserting the flexible endoscope into a body cavity using a first robotic arm and a second robotic arm, where the first robotic arm and the second robotic arm are controlled by a controller to coordinate movement and function of the flexible endoscope; identifying pathologies indicative of the presence of disease utilizing a plurality of sensors disposed along and at the tip of the endoscope; performing at least one diagnostic test using the plurality of sensors and at least one tool disposed at the tip of the endoscope to determine the presence of disease; and performing at least one endoscopic surgical procedure to treat the identified disease.

[0014] Other features and advantages of the present invention will become more readily apparent to those of ordinary skill in the art after reviewing the following detailed description and accompanying drawings. The structure and operation of the present invention will be understood from consideration of the following detailed description and the accompanying drawings in which like reference numerals refer to like parts. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram of a robotically controlled dual-arm flexible endoscope according to one embodiment of the present invention. [Figure 2] FIG. 1 is a diagram of a robotically controlled dual-arm flexible endoscope with a separate base for each arm, according to one embodiment of the present invention. [Figure 3] FIG. 2 is a diagram of an endoscope handle manipulator connected to a first robotic arm, according to one embodiment of the present invention. [Figure 4] FIG. 10 is a diagram of an endoscope actuator connected to a second robotic arm, according to one embodiment of the present invention. [Figure 5] FIG. 1 is a diagram of a sheath or condom having multiple location sensing devices configured thereon, according to one embodiment of the present invention. [Figure 6A] FIG. 1 is a side view of an endoscope having a monorail formed along the inner circumference of the scope shaft for deploying tools, according to one embodiment of the present invention. [Figure 6B] FIG. 1 is a top view of an endoscope having a monorail formed along the circumference of the scope shaft for deploying tools, according to one embodiment of the present invention. [Figure 6C] FIG. 1 is a cross-sectional view of a scope shaft showing a monorail positioned along the inner circumference of the shaft, according to one embodiment of the present invention. [Figure 6D] FIG. 1 is a close-up view of the distal portion of a scope shaft with a monorail showing the tool in a deployed position, according to one embodiment of the present invention. [Figure 7A] FIG. 1 is a diagram of a scope shaft with a light source and sensor located within an intestinal cavity, according to one embodiment of the present invention. [Figure 7B] FIG. 10 is a close-up view of a light source and sensor positioned on the scope shaft to detect reflected light from a tissue wall, according to one embodiment of the present invention. [Figure 7C] FIG. 1 is a cross-sectional view of multiple optical sensors arranged around the circumference of a scope shaft, according to one embodiment of the present invention. [Figure 7D] FIG. 1 is a cross-sectional view of multiple optical sensors arranged around the circumference of a scope shaft, according to one embodiment of the present invention. [Figure 8] FIG. 1 is a diagram of a system for remotely or autonomously performing endoscopic procedures using a robotically controlled dual-arm flexible endoscope, according to one embodiment of the present invention. [Figure 9] 1A-1C are diagrams of a method of performing an endoscopic procedure using a robotically controlled dual-arm flexible endoscope, according to one embodiment of the present invention. [Figure 10] FIG. 1 is a block diagram illustrating an example wired or wireless processor-enabled device that can be used in connection with various embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0016] Some embodiments disclosed herein provide systems, methods, and devices for robotic endoscopy using a robotically controlled, dual-arm flexible endoscope having a first robotic arm that holds and controls a robotically controlled endoscope handle manipulator and a second robotic arm that holds and controls an endoscope actuator for advancing and retracting the distal tip of the endoscope. Each robotic arm is capable of multiple movements that essentially mimic the functions of a human arm and hand, allowing the flexible endoscope to be repositioned, twisted, and torqued as a human operator would to change the orientation of the flexible endoscope as it navigates through non-linear organ structures within an animal or human body cavity. The robotic arms can be mounted on a single base and controlled via a single controller mechanism, or they can be mounted on separate bases with separate controller mechanisms, potentially allowing for increased angles of movement and positioning for each robotic arm.

[0017] Manual control of the endoscope handle knobs can be achieved via foot pedals, and when using a robotically controlled guidewire, a seat that allows each foot to control a separate function via a separate foot pedal can also be beneficial. Alternatively, the console can be constructed to provide greater control options.

[0018] The controller mechanism can be remotely operated for remote control of complex endoscopic procedures and can also perform automated operations by automatically performing designated functions using a combination of sensors and tools on the flexible endoscope through computer-assisted or image-guided surgery and intervention. Additionally, the device and system also enable artificial intelligence surgery, where one or more controllers enable autonomous navigation and operation through reinforcement learning algorithms and machine learning.

[0019] The possibility of robotic control, more specifically automatic and autonomous control, of flexible endoscopy significantly improves the safety and accuracy of both endoluminal and transluminal therapeutic procedures. Robotizing conventional flexible endoscopes means eliminating the drawbacks of cumbersome paths to reach specific locations within the digestive tract or other body cavities with complex anatomical structures.

[0020] After reading this specification, it will become apparent to those skilled in the art how to implement the invention in various alternative embodiments and alternative applications. However, while various embodiments of the invention are described herein, it is understood that these embodiments are presented for purposes of illustration only, and not limitation. As such, this detailed description of various alternative embodiments should not be construed as limiting the scope or breadth of the invention, which is set forth in the appended claims.

[0021] Robot-controlled double-arm endoscope 1 shows one embodiment of a robotically controlled dual-arm flexible endoscope, showing the overall endoscope 100 with a first robotic arm 102 having an endoscope handle manipulator 104 attached thereto and a second robotic arm 106 having an endoscope actuator 108 attached thereto. The flexible endoscope has a shaft 110 extending from the handle manipulator to a distal tip 112 that contains tools, cameras, lights, etc. used to complete the endoscopic procedure. In this embodiment, a single base 114 is used for both arms, which may also contain a controller and other computer-related equipment to perform the predetermined functions of the desired procedure.

[0022] A first robotic arm 102 controls the location, movement, and torque of an endoscope handle manipulator 104 at the proximal end of the flexible endoscope, while a second robotic arm 106 controls the location, movement, and torque of an endoscope actuator 108 used to translate a scope shaft 110 by advancing or withdrawing a distal tip 112 into or from a body cavity. Together, the first robotic arm 102 and the second robotic arm 106 allow the flexible endoscope to move with six degrees of freedom: rotational movement along the x-, y-, and z-axes, and translational movement along these three axes. In one embodiment, actuation of the endoscope handle manipulator 104 can be controlled via foot pedals, although if the guidewire within the scope shaft 110 is robotically controlled, a seat can be incorporated for the operator to operate the manipulator 104 with a right foot pedal and the endoscope actuator 108 with a left foot pedal. Alternatively, a console can be provided to allow for a centralized control module.

[0023] In one embodiment, shown in FIG. 2, each robotic arm has its own base 114, potentially allowing for greater movement and positioning of the flexible endoscope and providing alternative configurations for setting up a procedure room. FIG. 2 also provides a more detailed view of the individual components of each robotic arm (102, 106), with each robotic arm providing essentially any-direction movement to mimic the movement of a human arm performing a procedure. The first robotic arm 102 and the second robotic arm 106 may be essentially identical in design and overall movement, differing only in the attachments at their distal ends, where the first robotic arm 102 has an endoscope handle manipulator 104 attached and the second robotic arm 106 has an endoscope actuator 108 attached. The robotic arms (102, 106) may include a rotating base portion 116, an upper arm extension 118, a middle arm extension 120, a lower arm extension 122, and a mounting extension 124. Each extension is rotatably attached to the next extension to mimic the movement of a human arm with options for torque and twisting motion, from the rotator cuff shoulder joint of the rotating base portion 116 and upper arm extension 118 to the wrist joint represented by the lower arm extension 122 and mounting extension 124.

[0024] FIG. 3 is a detailed view of the endoscope handle manipulator 104 attached to the first robotic arm 102, according to one embodiment of the present invention. The endoscope handle manipulator 104 is attached to a mounting extension 124, which rotates relative to the adjacent lower arm extension 122 but can also rotate axially about the length of the mounting extension 124. The manipulator 104 is configured to control the primary operation of the flexible endoscope with respect to the camera, lighting, tools, and any other functions located at the distal tip 112. These elements are connected via a scope 126, which becomes a scope shaft 110 that extends from the manipulator and terminates at the distal tip 112 (not shown), where the tools and other elements of the endoscope are located. In one embodiment, a separate connector 128 may be utilized to provide power and output images and other data from the endoscope to a terminal, monitor, or other console used by an operator to perform an endoscopic procedure. However, this may be communicated via the robotic arm 102 to a controller located at the base 114.

[0025] 4 is a detailed view of the endoscope actuator 108 showing a pair of axial rollers 130 used to translate the movement of the shaft 110 to advance or withdraw the endoscope's distal tip 112 into, out of, or around a body cavity. The mounting extension 124 may be connected to a support arm 132 that maintains the endoscope actuator at a constant angle relative to the second robotic arm 106. The support arm 132 may also include any power and data cables necessary to provide power, control, and feedback to and from the endoscope actuator to the operator.

[0026] Remote and autonomous control In one embodiment, the use of one or more controllers and two highly flexible robotic arms provides additional options for improved remote control of endoscopic procedures, including the ability to perform automatic or autonomous functions. The controllers operating each robotic arm can provide detailed feedback and complex data regarding the movement and control of the flexible endoscope, used in conjunction with images and other sensor data obtained from the endoscope tip, which can be processed automatically by the operator or by a computing device managing both the robotic arms and the flexible endoscope into recommendations and actions regarding the movement and operation of the endoscope. The operator may be located remotely from the device so that they can remotely control the endoscope and robotic arms.

[0027] Additionally, robotically controlled dual-arm flexible endoscopes can incorporate artificial intelligence (AI) machine learning algorithms within the controllers and computing devices that operate the flexible endoscope and robotic arms to develop autonomous endoscopic procedure options, such as colonoscopy, upper gastrointestinal endoscopy, bronchoscopy, thoracoscopy, and cystoscopy. Unlike traditional AI, which is primarily used to analyze complex data sets, these algorithms are used to translate computer vision data, robotic arm controller data, and other sensing tools into the physical motion and movement of the endoscope and robotic arm, and even the execution of interventional actions and movements. The learning algorithms aid in both the navigation and execution of various procedures, with the assistance of a surgeon or operator, or fully autonomously without any human intervention.

[0028] Extensions The design of an improved robotically controlled dual-arm flexible endoscope also allows for the incorporation of several further improvements in the equipment and functionality of the flexible endoscope. For example, the robotically controlled dual-arm flexible endoscope allows for the use of an inexpensive disposable endoscope, which can be manufactured with wider channels to allow for thicker guidewires that provide more complex functionality at the distal tip 112.

[0029] Additionally, advanced technologies such as fibers with shape-sensing technology for accurately locating the location and shape of the endoscope within a body cavity can be incorporated into the scope, but additional disposable elements, such as a sheath or condom, may be required to hold the fiber in place until the shape-sensing fiber can be incorporated into the scope shaft 110 and the optical fiber end at the distal tip 112. Figure 5 illustrates one embodiment of a sheath (condom) 500 with wires 502 in four quadrants, configured to be placed over an existing endoscope tip and attached to a system 504 that interprets acoustic signals. Using a reinforcement learning algorithm to triangulate signals from the four quadrants of the endoscope tip enables autonomous and safe endoscopy. Unlike traditional guidance methods using image interpretation and computer vision, algorithms that analyze acoustic signals have the advantage of utilizing less data, which should speed safe autonomous operation and make real-world applications possible.References include "Surgical Audio Guidance: Feasibility Check for Robotic Surgery Procedures" by Anna Schaufler, Alfredo Illanes, Ivan Maldonado, Axel Boese, Roland Croner, and Michael Friebe; "Current Directions in Biomedical Engineering" by Michael Friebe; and "Surgical Audio Guidance: Feasibility Check for Robotic Surgery Procedures" by Anna Schaufler, Alfredo Illanes, Ivan Maldonado, Axel Boese, Roland Croner, and Michael Friebe. Engineering, Vol. 6, No. 3, 2020, pp. 571-574, https: / / doi.org / 10.1515 / cdbme-2020-3146 and Mahmoodian, Naghmeh, Schaufler, Anna, Pashazadeh, Ali, Boese, Axel, Friebe, Michael et al., "Proximal detection of guide wire perforation using feature extraction from bispectral audio signal analysis combined with machine learning," Computers in Biology and Medicine. Medicine, Oxford, Vol. 107, (April 2019), pp. 10-17, DOI: 10.1016 / j.compbiomed.2019.02.001.

[0030] Additional portions of the device may also be disposable to avoid wear and tear and any potential contamination from reusable components that come into contact with the portion of the endoscope that enters the body cavity, such as the grips on the axial rollers located on the endoscope actuator that translate the shaft into or out of the body cavity.

[0031] Rail Delivery System 6A-6D show a rail delivery system in which a monorail is positioned within the endoscope shaft to more easily deliver one or more tools or tools used in an endoscopic procedure. FIG. 6A is a side view of an endoscope 600, with the manipulator 602 and scope shaft 604 shown in transparent view to illustrate the monorail 608 positioned along the length of the scope shaft 604, along the inner circumferential wall 610 of the scope shaft 604, as more clearly shown in FIG. 6C. In this embodiment, the monorail 608 is used to deliver a lateral stapler 612, which is shown in a deployed configuration still attached to the monorail 608 via a connector 606 that allows the stapler 612 to be rotated into position for use but also to be withdrawn along the monorail 608 as needed. The manipulator 602 can manipulate and control the tool 612 as a whole, or it can provide a separate controller or handle for partial or full manual manipulation.

[0032] In another embodiment shown in Figure 6B, a monorail 608 can be used to deliver a tool along the periphery 614 of the shaft 604 using a separate insertion point 616 for attaching a tool 612 via a connector 606. The monorail 608 protrudes from the periphery of the shaft and can be shaped similar to that shown in Figure 6C so that the tool can slide along a hollowed-out channel formed by the monorail 608.

[0033] 6D is a close-up view of the distal portion of scope shaft 604 with monorail 608, showing tool 612 in a deployed position connected to monorail 608 via connector 606. Monorail 608 allows tool 612 to be more easily deployed through a flexible endoscope such as that illustrated herein, and preserves space within shaft 604 for other tools and instruments. Monorail 608 can also allow for different types of tools that may traditionally be difficult to insert, such as circular staplers or vascular sealing devices used to seal the walls of the digestive tract or remove polyps and other growths.

[0034] Sensor embedded sheath In one embodiment, an endoscope shaft can incorporate various types of sensors used to detect location and other navigation data that can be used to position and move scopes and tools to specific locations or to avoid problem areas. Figure 7A shows one embodiment of a scope shaft 702 positioned between intestinal walls 704 of an intestinal cavity 706, such as during an anoscopy. While a variety of sensors can be utilized, one embodiment shown in Figure 7B is the use of a combination of a light source 708 and a light sensor 710 positioned along the length of the shaft 702, which cooperate to transmit light 712B outward from the light source 708 along the length of the shaft, which then reflects the light 712B off the surrounding tissue walls 704, where it is then detected by the adjacent sensor 710.

[0035] The sensor 710 can then use the reflected light 712B to detect the distance, location, and position of the shaft relative to the tissue wall, as well as use multispectral, narrowband, hyperspectral, and multispectral imaging to enhance navigation or detect different wavelengths reflected from adjacent tissue to determine whether there are areas of tissue that may require further investigation or treatment as part of the autonomous diagnosis of pathology during an endoscopic procedure. In one embodiment, an ultrasound endoscope can be attached to a robotically controlled dual-arm flexible endoscope to enable autonomous endoscopic echocardiography. For example, during an upper gastrointestinal endoscopy, the endoscope can perform autonomous monitoring of organs such as the pancreas and may be further configured to perform autonomous fine-needle aspiration of suspicious lesions.

[0036] In one embodiment, the light source is a light-emitting diode (LED) and the light sensor 710 is a camera that can be used as part of an autonomous echoendoscopy. The light can be emitted circumferentially at a specific point to substantially cover the circumference of the shaft, and the paired sensor detects light reflected from nearby tissue, allowing calculation of the distance of the shaft from the surrounding tissue. In one embodiment, the light source and light sensor are positioned every few centimeters and pulsed at different times, allowing the sensor and connected computing device to calculate which part of the shaft is in which part of the body cavity and whether the shaft may be touching the cavity wall.

[0037] As mentioned above, the use of sensors enables autonomous echoendoscopy by allowing the scope and connected tools to autonomously navigate through and screen for pathologies such as pancreatic cancer. Various hardware and software products can analyze video data, such as video surgomics, a field that uses computer vision, machine learning, and multimodal data during endoscopic interventional procedures to provide predictions based on the data collected therein.

[0038] Figure 7C shows a cross section of the scope shaft 702 at the location of a bank of light sensors 710 positioned at different locations around the circumference of the shaft 702. Similarly, Figure 7D shows a cross section of the scope shaft 702 at the location of a bank of light sources 708 positioned at different locations around the circumference of the shaft 702. As previously shown in Figures 7A and 7B, the light sources 708 and light sensors 710 may be positioned adjacent to one another so that the light sources transmit light proximate to the light sensors, which then reflect the light back to the light sensors for processing relevant information such as distance from tissue, tissue type, abnormal color, texture, depth, or pattern, which may be indicative of a pathology.

[0039] In another embodiment, different sensors are placed on different sections of the shaft depending on the data that needs to be collected from each location along the shaft. For example, light source 708 and light sensor 710 or other location type sensors may be positioned along the length of shaft 702 to identify the position and location of the shaft, while other types of sensors may be placed towards the distal end of shaft 702 to more accurately identify the type of tissue, organ, or abnormality therein to aid in performing any type of related endoscopic procedure.

[0040] In one embodiment, the sensor can be a vibroacoustic sensor placed within the wall of the scope shaft for use as part of an audio-based guidance system known as Surgical Audio Guidance (SURAG). The vibroacoustic sensor is a passive, plug-and-play sensing solution that can be attached to the proximal end of the vibroacoustic sensor outside the patient's body, where it is integrated into the endoscope handle. The vibroacoustic sensor can be a string surrounded by a fluid within a thin, flexible tube, using the fluid (such as oil or water) to transmit vibrations detected by the string. Thus, the vibroacoustic sensor can be embedded within the shaft to provide insulation to attenuate spurious vibrations.

[0041] Anoscopy A robotically controlled, dual-arm flexible endoscope can be integrated with various types of scopes to perform a variety of different procedures. In one embodiment, the robotically controlled, dual-arm flexible endoscope can be integrated with an anoscope to perform anoscopy and provide treatment of the anal canal or rectum. In one example, a trifurcated endoscopic device, such as that described in U.S. Pat. No. 10,729,318, can be incorporated into a robotically controlled, dual-arm flexible endoscope, which involves using a flexible ablation wire to provide endoluminal treatment within the digestive tract or bronchial tree. An ablation wire with integrated temperature control allows for safe ablation of hollow visceral structures without perforation, and also dramatically reduces post-ablation bleeding after endobronchial ablation, due to the use of specially selected low temperatures sufficient to perform the desired treatment but low enough to prevent perforation and / or post-procedure bleeding.

[0042] In another embodiment, a hollow ablation needle can be used to inject immunotherapy into the post-ablation area. In some cases, it has been shown that post-ablation immunological factors released after ablation have the potential to generate tumor vaccines. By injecting immunotherapy directly into tumors immediately after ablation, vaccines against tumors can be developed. This can reduce local and distant recurrence of these conditions.

[0043] Computer Vision To provide fully autonomous endoscopic procedures, the use of automated devices, such as robotically controlled dual-arm flexible endoscopes, must be combined with computer vision and other machine learning tools to assist in navigation, diagnosis, and treatment during any type of endoscopic procedure. As discussed above, the use of sensors along and at the tip of the scope shaft provides additional imaging data (including narrowband, hyperspectral, and multispectral imaging) that can be used for navigation, diagnosis, and treatment. The incorporation of biophotonics into computer vision can further improve the algorithm's ability to diagnose pathologies.

[0044] In one embodiment, computer vision and diagnostic enhancements can be combined with a robotically controlled, dual-arm flexible endoscope and the aforementioned anoscope to develop a guaiac-based fecal occult blood test. This embodiment utilizes a clear plastic anoscope, allowing computer vision to determine whether a patient is guaiac-positive by analyzing the color of the stool sample (indicating the presence of iron, and therefore blood, in the stool). A positive test result during anoscopy triggers an automated colonoscopy, which can be performed by the robotically controlled, dual-arm flexible endoscope already in place for the anoscopy. An ablation or other therapeutic device, such as a trident, can then be introduced into the anoscope to perform the colonoscopy and automatically administer any necessary treatment.

[0045] System Overview 8 is a diagram of a system 850 for performing an endoscopic procedure using a robotically controlled dual-arm flexible endoscope and a remotely located device from either a remote user or an autonomous system. In this embodiment, the robotic endoscope 852 may be operated by a controller 854, which may be controlled autonomously by a surgeon 858 operating a remote device 856 or via a computing device such as a server 860 that runs software and performs endoscopic operations via data obtained from a procedure database 862 and an image database 864. For example, the flexible endoscope may transmit images from a camera at its distal tip 112 to the server 860, which then analyzes the images compared to images in the image database 864 and determines possible movements and manipulations of the endoscope and associated tools.

[0046] Methods for performing endoscopic procedures FIG. 9 illustrates an exemplary method for performing an endoscopic procedure using a robotically controlled, dual-arm flexible endoscope. In step 902, a particular type of endoscopic procedure is selected, and in step 904, a first robotic arm moves the endoscope handle manipulator to the location, position, and angle required for optimal execution and function of the desired movement and manipulation. In step 906, a second robotic arm moves the endoscope actuator to the location, position, and angle required for optimal execution and function of the desired movement and manipulation. With both arms in place, in step 908, the distal tip of the endoscope is translated to the required location within the body cavity, after which, in step 910, the endoscope handle manipulator can cause the performance of the specified type of endoscopic procedure selected above. In step 912, either the surgeon or the autonomous system can evaluate the completed steps and determine whether further work needs to be performed or whether additional work still needs to be completed, and then can instruct the controller to perform the new procedure.

[0047] In a further embodiment, the method of performing an endoscopic procedure may include the additional step of using sensors to acquire location and position data in order to navigate the endoscope to the required location. In yet another step, imaging data from the sensors may be used to analyze possible pathologies and determine whether additional diagnostic or therapeutic procedures should be performed. The endoscope may then be automatically reconfigured with the appropriate tools to perform the required diagnostic or therapeutic procedures.

[0048] These methods described above can be incorporated into methods for automated screening, diagnosis and treatment of multiple disease states, such as colon or pancreatic cancer. Computer-compatible embodiment 10 is a block diagram illustrating an example wired or wireless system 550 that can be used in connection with various embodiments described herein. For example, system 550 can be used as or in combination with the apparatus and methods described above with respect to FIGS. 1-9. System 550 can be a conventional personal computer, a computer server, a personal digital assistant, a smartphone, a tablet computer, or any other processor-enabled device capable of wired or wireless data communication. Other computer systems and / or architectures can also be used, as will be apparent to those skilled in the art.

[0049] System 550 preferably includes one or more processors, such as processor 560. Additional processors may be provided, such as auxiliary processors to manage input / output, auxiliary processors to perform floating-point mathematical operations, dedicated microprocessors (e.g., digital signal processors) having architectures suitable for high-speed execution of signal processing algorithms, slave processors (e.g., back-end processors) subordinate to the main processing system, additional microprocessors or controllers or coprocessors for dual or multiprocessor systems, etc. Such auxiliary processors may be separate processors or may be integrated with processor 560.

[0050] Processor 560 is preferably connected to a communications bus 555. Communications bus 555 may include a data channel for facilitating information transfer between storage and other peripheral components of system 550. Communications bus 555 may further provide a set of signals used for communication with processor 560, including a data bus, an address bus, and a control bus (not shown). Communications bus 555 may include any standard or non-standard bus architecture, such as, for example, an industry standard architecture ("ISA"), an extended industry standard architecture ("EISA"), a Micro Channel Architecture ("MCA"), a peripheral component interconnect ("PCI") local bus, or a bus architecture conforming to standards including the IEEE 488 general-purpose interface bus ("GPIB") promulgated by the Institute of Electrical and Electronics Engineers ("IEEE"), IEEE 696 / S-100, etc.

[0051] System 550 preferably includes a main memory 565 and may also include a secondary memory 570. Main memory 565 provides storage of instructions and data for programs executing on processor 560. Main memory 565 is typically semiconductor-based memory such as dynamic random access memory (“DRAM”) and / or static random access memory (“SRAM”). Other semiconductor-based memory types include, for example, synchronous dynamic random access memory (“SDRAM”), Rambus dynamic random access memory (“RDRAM”), ferroelectric random access memory (“FRAM”), etc., including read-only memory (“ROM”).

[0052] Secondary memory 570 may optionally include internal memory 575 and / or removable media 580, such as a floppy disk drive, a magnetic tape drive, a compact disc ("CD") drive, a digital versatile disc ("DVD") drive, etc. Removable media 580 may be read from and / or written to in known manner. Removable storage media 580 may be, for example, a floppy disk, a magnetic tape, a CD, a DVD, an SD card, etc.

[0053] The removable storage medium 580 is a non-transitory computer-readable medium that stores computer-executable code (i.e., software) and / or data that may be loaded into the system 550 for execution by the processor 560.

[0054] In alternative embodiments, secondary memory 570 may include other similar means for allowing computer programs or other data or instructions to be loaded into system 550. Such means include, for example, external storage medium 595 and interface 570. Examples of external storage medium 595 include an external hard disk drive or an external optical or magneto-optical drive.

[0055] Other examples of secondary memory 570 may include semiconductor-based memory such as programmable read-only memory ("PROM"), erasable programmable read-only memory ("EPROM"), electrically erasable read-only memory ("EEPROM"), or flash memory (a block-oriented memory similar to EEPROM). Any other removable storage media 580 and communication interface 590 that allow software and data to be transferred to system 550 from external media 595 are also included.

[0056] System 550 may also include input / output ("I / O") interface 585. I / O interface 585 facilitates input from and output to external devices. For example, I / O interface 585 may receive input from a keyboard or mouse and provide output to a display. I / O interface 585 may similarly facilitate input from and output to various alternative types of human interface and machine interface devices.

[0057] System 550 may also include a communications interface 590. Communications interface 590 allows software and data to be transferred between system 550 and external devices (such as a printer), networks, or information sources. For example, computer software or executable code may be transferred to system 550 from a network server via communications interface 590. Examples of communications interface 590 include a modem, a network interface card ("NIC"), a wireless data card, a communications port, a PCMCIA slot and card, an infrared interface, and an IEEE 1394 Firewire, to name a few.

[0058] Communications interface 590 preferably implements industry-promulgated protocol standards such as the Ethernet IEEE 802 standard, Fibre Channel, digital subscriber line ("DSL"), asynchronous digital subscriber line ("ADSL"), frame relay, asynchronous transfer mode ("ATM"), integrated digital services network ("ISDN"), personal communications service ("PCS"), transmission control protocol / Internet protocol ("TCP / IP"), serial line Internet protocol / point to point protocol ("SLIP / PPP"), etc., but may implement customized or non-standard interface protocols as well.

[0059] The software and data transferred via communications interface 590 are typically in the form of electrical communications signals 605. These signals 605 are preferably provided to communications interface 590 via communications channel 600. In one embodiment, communications channel 600 may be a wired or wireless network or any of a variety of other communications links. Communications channel 600 carries signals 605 and may be implemented using a variety of wired or wireless communications means, including wire or cable, optical fiber, conventional telephone lines, cellular phone links, wireless data communications links, radio frequency ("RF") links, or infrared links, to name a few.

[0060] Computer executable code (i.e., computer programs or software) is stored in main memory 565 and / or secondary memory 570. Computer programs may also be received via communications interface 590 and stored in main memory 565 and / or secondary memory 570. When executed, these computer programs enable system 550 to perform the various functions of the present invention as previously described.

[0061] As used herein, the term "computer-readable medium" refers to any non-transitory computer-readable storage medium used to provide computer-executable code (e.g., software and computer programs) to system 550. Examples of these media include main memory 565, secondary memory 570 (including internal memory 575, removable media 580, and external storage media 595), and any peripheral devices (including network information servers or other network devices) communicatively coupled to communication interface 590. These non-transitory computer-readable media are means for providing executable code, programming instructions, and software to system 550.

[0062] In embodiments implemented using software, the software may be stored on a computer-readable medium and loaded into system 550 via removable medium 580, I / O interface 585, or communication interface 590. In such embodiments, the software is loaded into system 550 in the form of electrical communication signals 605. When executed by processor 560, the software preferably causes processor 560 to perform the features and functions of the present invention as previously described herein.

[0063] System 550 also includes optional wireless communication components that facilitate wireless communication over voice and data networks. The wireless communication components include an antenna system 610, a radio system 615, and a baseband system 620. In system 550, radio frequency ("RF") signals are transmitted and received over the air by antenna system 610 under the control of radio system 615.

[0064] In one embodiment, the antenna system 610 may include one or more antennas and one or more multiplexers (not shown) that perform a switching function to provide transmit and receive signal paths for the antenna system 610. In the receive path, the received RF signal may be coupled from the multiplexer to a low noise amplifier (not shown), which amplifies the received RF signal and transmits the amplified signal to the radio system 615.

[0065] In alternative embodiments, the radio system 615 may include one or more radios configured to communicate at various frequencies. In one embodiment, the radio system 615 may combine a demodulator (not shown) and a modulator (not shown) in a single integrated circuit (“IC”). The demodulator and modulator may also be separate components. In the receive path, the demodulator removes the RF carrier signal, leaving a baseband receive audio signal, which is transmitted from the radio system 615 to the baseband system 620.

[0066] If the received signal contains audio information, the baseband system 620 decodes and converts the signal to an analog signal. The signal is then amplified and transmitted to a speaker. The baseband system 620 also receives analog audio signals from a microphone. These analog audio signals are converted to digital signals and encoded by the baseband system 620. The baseband system 620 also encodes the digital signals for transmission and generates baseband transmit audio signals that are routed to a modulator portion of the radio system 615. The modulator mixes the baseband transmit audio signal with an RF carrier signal to generate an RF transmit signal that can be routed to an antenna system and passed through a power amplifier (not shown). The power amplifier amplifies the RF transmit signal and routes it to the antenna system 610, where the signal is switched to an antenna port for transmission.

[0067] Baseband system 620 is also communicatively coupled to processor 560. Central processing unit 560 has access to data storage areas 565 and 570. Central processing unit 560 is preferably configured to execute instructions (i.e., computer programs or software), which may be stored in memory 565 or secondary memory 570. Computer programs may also be received from baseband processor 610 and stored in data storage area 565 or secondary memory 570, or executed upon receipt. Such computer programs, when executed, enable system 550 to perform the various functions of the present invention as previously described. For example, data storage area 565 may include various software modules (not shown) executable by processor 560.

[0068] Various embodiments may also be implemented primarily in hardware using, for example, components such as application specific integrated circuits ("ASICs") or field programmable gate arrays ("FPGAs"). Implementation of a hardware state machine capable of performing the functions described herein will also be apparent to those skilled in the relevant art. Various embodiments may also be implemented using a combination of both hardware and software.

[0069] Furthermore, those skilled in the art will appreciate that the various illustrative logic blocks, modules, circuits, and method steps described in connection with the above-described figures and embodiments disclosed herein may often be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the invention. Additionally, the grouping of functions within a module, block, circuit, or step is for ease of description. Particular functions or steps may be moved from one module, block, or circuit to another without departing from the invention.

[0070] Furthermore, the various illustrative logic blocks, modules, and methods described in connection with the embodiments disclosed herein may be implemented or performed with a general-purpose processor, a digital signal processor ("DSP"), an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration.

[0071] Furthermore, the steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium, including a network storage medium. An exemplary storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and the storage medium may also reside in an ASIC.

[0072] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles described herein may be applied to other embodiments without departing from the spirit or scope of the present invention. As such, it should be understood that the description and drawings presented herein represent presently preferred embodiments of the invention and, therefore, represent the subject matter broadly contemplated by the present invention. It is further understood that the scope of the present invention fully encompasses other embodiments that may become apparent to those skilled in the art, and that the scope of the present invention is not limited accordingly.

Claims

1. 1. A robotically controlled dual-arm flexible endoscope, comprising: a flexible endoscope having a shaft, a proximal handle, and a distal tip; a first robotic arm having the proximal handle fixed thereto, the first robotic arm being configured to move the proximal handle of the flexible endoscope; an endoscope handle manipulator connected to the first robotic arm and the proximal handle and configured to control the flexible endoscope; a second robotic arm having an endoscope actuator configured to secure the distal tip and advance or withdraw the flexible endoscope into or from a body cavity; a controller in communication with the first robotic arm and the second robotic arm to control the movement and position of each robotic arm, the handle manipulator, and the endoscope actuator; A robotically controlled dual-arm flexible endoscope including:

2. The endoscope of claim 1 , wherein the first robotic arm controls the location, movement, and torque of the endoscope handle manipulator at the proximal end of the flexible endoscope.

3. The endoscope of claim 2 , wherein the second robotic arm controls the location, movement, and torque of the endoscope actuator to advance or withdraw the flexible endoscope into or from the body cavity.

4. The endoscope of claim 3 , wherein the first and second robotic arms provide six degrees of freedom to the flexible endoscope.

5. The endoscope of claim 1 , wherein the controller is autonomously controlled by a computing device to control the movement and position of each robotic arm, the handle manipulator, and the endoscope actuator.

6. The endoscope of claim 5 , wherein a computing device utilizes computer vision and machine learning to control the movement and position of each robotic arm, the handle manipulator, and the endoscope actuator.

7. The endoscope of claim 1 , wherein the flexible endoscope further comprises a scope shaft having at least one sensor integrated therein.

8. The endoscope of claim 7 , wherein the scope shaft includes a plurality of sensors positioned along the length of the scope shaft.

9. The endoscope of claim 8 , wherein the plurality of sensors determine at least one of position, location, and movement of the scope shaft within a body cavity.

10. The endoscope of claim 1 , further comprising a rail disposed along an inner circumference of the scope shaft configured to deliver at least one endoscopic tool to the distal end of the endoscope.

11. 1. A method for performing an endoscopic procedure using a robotically controlled flexible endoscope, comprising: controlling, via the controller, the movement and function of an endoscope handle manipulator connected to a proximal handle of the flexible endoscope via a first robotic arm; Controlling, via a controller, the movement and function of an endoscope actuator to which the distal tip of the flexible endoscope is fixed via a second robot arm; performing the endoscopic treatment through movement and control of the flexible endoscope by the controller; A method comprising:

12. 12. The method of claim 11, further comprising the controller using the first robotic arm to control the location, movement, and torque of the endoscope handle manipulator at the proximal end of the flexible endoscope.

13. 13. The method of claim 12, further comprising the controller using the second robotic arm to control the location, movement, and torque of the endoscope actuator to advance or withdraw the flexible endoscope into or from a body cavity.

14. 14. The method of claim 13, further comprising utilizing the first and second robotic arms to provide six degrees of freedom of movement of the flexible endoscope.

15. The method of claim 11 , further comprising autonomously controlling the controller using a computing device.

16. 16. The method of claim 15, further comprising utilizing computer vision and machine learning to control the movement and position of each robotic arm, the handle manipulator, and the endoscope actuator.

17. 12. The method of claim 11, further comprising determining at least one of position, location and movement of the scope shaft within the body cavity using a plurality of sensors positioned along the length of the scope shaft.

18. 12. The method of claim 11, further comprising delivering at least one endoscopic tool to the distal end of the endoscope using a rail disposed along the inner circumference of the scope shaft.

19. 1. A method of screening for disease using an autonomous flexible endoscope, comprising: inserting a flexible endoscope into a body cavity using a first robotic arm and a second robotic arm, the first robotic arm and the second robotic arm being controlled by a controller to coordinate movement and function of the flexible endoscope; utilizing a plurality of sensors disposed along and at the distal end of the endoscope to identify pathologies indicative of the presence of disease; performing at least one diagnostic test using the plurality of sensors and at least one tool disposed at the distal end of the endoscope to determine the presence of the disease; performing at least one endoscopic surgical procedure to treat the identified condition; A method comprising:

20. 20. The method of claim 19, wherein the cancer is one of colon cancer or pancreatic cancer.

Images