Automated Treatment Evaluation

Abstract

A robotic system configured to evaluate an identified stage of a medical procedure includes a video capture device, a robotic manipulator, one or more sensors, an input device, a data store, and control circuitry. The control circuitry is configured to determine a first state of the robotic manipulator based on sensor data from the one or more sensors, identify a first input from the input device to initiate a first action of the robotic manipulator, perform a first analysis of a video of a patient site captured by the video capture device, identify a first stage of the medical procedure based at least in part on the first state of the robotic manipulator, the first input, and the first analysis of the video, and generate an evaluation of the first stage of the medical procedure based on one or more metrics associated with the first stage.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 116,798, filed November 20, 2020, and U.S. Patent Application No. 63 / 132,875, filed December 31, 2020, each of which is incorporated by reference in its entirety.

[0002] FIELD OF THE INVENTION The present disclosure relates to the fields of medical devices and procedures, and artificial intelligence assisted data processing. [Background technology]

[0003] Various medical procedures involve the use of robotic systems to assist in the use of one or more medical instruments configured to penetrate the human anatomy to reach a treatment site. One particular surgical process may involve inserting one or more medical instruments through a patient's skin and an orifice to reach a treatment site and extract an object, such as a urinary stone, from the patient. Summary of the Invention [Means for solving the problem]

[0004] Described herein is one or more systems, devices, and / or methods for assisting a physician or medical professional in controlling a medical instrument for accessing an object, such as a urinary stone, located within the anatomy of the human body.

[0005] For purposes of summarizing the present disclosure, certain aspects, advantages, and novel features have been described. It is to be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment. Thus, the disclosed embodiments may be practiced in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other advantages that may be taught or suggested herein.

[0006] One or more computer systems can be configured to perform specific operations or actions by having software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform the actions during operation. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions. One general aspect includes a robotic system for evaluating identified stages of a medical procedure performed by a robotic system. The robotic system also includes a video capture device, a robotic manipulator, one or more sensors configured to detect a configuration of the robotic manipulator, an input device configured to receive one or more user interactions and initiate one or more actions by the robotic manipulator, a data store configured to store metrics associated with the stages of the medical procedure, and control circuitry communicatively coupled to the input device and the robotic manipulator. The control circuitry is configured to: determine a first state of the robotic manipulator based on sensor data from the one or more sensors; identify a first input from the input device to initiate a first action of the robotic manipulator; perform a first analysis of video of the patient site captured by the video capture device; identify a first stage of the medical procedure based at least in part on the first state of the robotic manipulator, the first input, and the first analysis of the video; and generate an assessment of the first stage of the medical procedure based on one or more metrics associated with the first stage. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0007] Implementations may include one or more of the following features: The first stage of the medical procedure may include one of ureteroscope actuation, ureteroscope laser procedure, ureteroscope basket actuation, and percutaneous needle insertion. The first stage may include ureteroscope basket actuation, and generating the assessment may include counting the number of basket movements, counting the number of ureteroscope retractions, determining a ratio of the number of basket movements to the number of ureteroscope retractions, and comparing the determined ratio with other ratios from previous ureteroscope basket procedures. The first stage may include ureteroscope actuation, and generating the assessment may include counting the number of times a user manually actuates the scope, counting the number of times a user robotically actuates the scope, determining a ratio of the number of times a user manually actuates the scope to the number of times a user robotically actuates the scope, and comparing the determined ratio with other ratios from previous ureteroscope basket procedures. The first stage may include percutaneous needle insertion, and generating the assessment may include counting the number of attempts a user makes to insert the needle until the user successfully inserts the needle and comparing the counted number with recorded needle insertion attempts from a previous percutaneous needle insertion operation. The first stage may include percutaneous needle insertion, and generating the assessment may include counting the time it takes to explore the kidney before selecting a target calyx for percutaneous access and comparing the counted number with recorded times from the previous percutaneous needle insertion operation. The first stage may include percutaneous needle insertion, and generating the assessment may include counting the number of times a navigation field generator for tracking the needle is repositioned and comparing the counted number with recorded repositionings from the previous percutaneous needle insertion operation. The first stage may include percutaneous needle insertion, and generating the assessment may include counting the number of times automatic alignment between an end effector of the robotic manipulator and the catheter is initiated and comparing the counted number with recorded automatic alignments from the previous operation.The first stage may include a ureteroscopy laser procedure, and generating an assessment may include counting the laser treatment time of the concretion, determining the size of the concretion, and comparing the ratio of the laser treatment time to the size of the concretion with previous ratios from other operations. The first stage may include a ureteroscopy laser procedure, and generating an assessment may include determining the type of the concretion and aggregating statistics across the surgical procedure based on the type of concretion. The first stage may include a ureteroscopy laser procedure, and generating an assessment may include counting the number of times the view of the video capture device is obstructed by dust from the fragmentation of the concretion and comparing the counted number of times with the recorded number of dust obstructions from the operation. Implementations of the described technology may include hardware, methods, processes, or computer software on a computer-accessible medium.

[0008] One general aspect includes a method for evaluating an identified stage of a medical procedure performed by a robotic system, which may include a video capture device. The method also includes determining a first state of a robotic manipulator based on sensor data from one or more sensors, identifying a first input from an input device for initiating a first action of the robotic manipulator, performing a first analysis of a video of a patient site captured by the video capture device, identifying a first stage of the medical procedure based at least in part on the first state of the robotic manipulator, the first input, and the first analysis of the video, and generating an evaluation of the first stage of the medical procedure based on one or more metrics associated with the first stage. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0009] Implementations may include one or more of the following features: A method in which the first stage may include ureteroscopic basket processing, and generating an assessment may include counting the number of basket movements, counting the number of ureteroscopic retractions, determining a ratio between the number of basket movements and the number of ureteroscopic retractions, and comparing the determined ratio with other ratios from previous ureteroscopic basket movements; The first stage may include ureteroscopic driving, and generating an assessment may include counting the number of times a user manually drives the scope, counting the number of times a user robotically drives the scope, determining a ratio between the number of times a user manually drives the scope and the number of times a user robotically drives the scope, and comparing the determined ratio with other ratios from previous ureteroscopic basket movements; The first stage may include percutaneous needle insertion, and generating an assessment may include counting the number of times a user attempts to insert a needle until the user successfully inserts the needle, and comparing the counted number with recorded needle insertion attempts from previous percutaneous needle insertion operations. The first stage may include percutaneous needle insertion, and generating the assessment may include counting the time it takes to explore the kidney before selecting a target calyx for percutaneous access and comparing the counted time with a recorded time from a previous percutaneous needle insertion operation. The first stage may include percutaneous needle insertion, and generating the assessment may include counting the number of times a navigation field generator for tracking the needle is repositioned and comparing the counted number with a recorded number of repositionings from a previous percutaneous needle insertion operation. The first stage may include percutaneous needle insertion, and generating the assessment may include counting the number of times automatic alignment between an end effector of the robotic manipulator and the catheter is initiated and comparing the counted number with a recorded number of automatic alignments from a previous operation. The first stage may include percutaneous antegrade ureteroscopy laser treatment, and generating the assessment may include counting the time for laser treatment of the stone, determining the size of the stone, and comparing the ratio of the laser treatment time to the stone size with a previous ratio from another operation.The first stage may include a ureteroscopy laser procedure, and generating an assessment may include determining the type of stone and aggregating statistics across the surgical procedure based on the type of stone. The first stage may include a ureteroscopy laser procedure, and generating an assessment may include counting the duration that the view of the video capture device is obstructed by debris from the stone fragmentation and comparing the counted duration with recorded durations from previous operations. Implementations of the described technologies may include hardware, methods, or processes, or computer software on a computer-accessible medium.

[0010] One general aspect includes a control system for a robotic device for assessing an identified stage of a medical procedure. The control system may also include a communications interface configured to receive sensor data, user input data, and video data from the robotic device; a memory configured to store the sensor data, user input data, and video data; and one or more processors configured to: determine a first state of a manipulator of the robotic device based on sensor data from the one or more sensors; identify a first input from the user input data to initiate a first action of the manipulator; perform a first analysis of a video of the patient site captured by a video capture device; and identify a first stage of the medical procedure based at least in part on the first state of the manipulator, the first input, and the first analysis of the video. The system may also generate an assessment of the first stage of the medical procedure based on one or more metrics associated with the first stage. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. [Brief explanation of the drawings]

[0011] Various embodiments are shown in the accompanying drawings for purposes of illustration and should not be construed as limiting the scope of the present disclosure in any way. In addition, various features of different disclosed embodiments may be combined to form further embodiments that are part of the present disclosure. Throughout the drawings, reference numerals may be reused to indicate correspondence between referenced elements. [Figure 1] FIG. 1 illustrates an example medical system for performing or assisting in performing a medical procedure, according to certain embodiments. [Figure 2A] 1 is a perspective view of a medical system during a urinary stone trapping procedure, according to certain embodiments. FIG. [Figure 2B] 1 is a perspective view of a medical system during a urinary stone trapping procedure, according to certain embodiments. FIG. [Figure 3] FIG. 1 is a block diagram of a control system of a medical system with associated inputs and outputs, according to certain embodiments. [Figure 4A] FIG. 1 is a block diagram of a control system configured to utilize machine learning to generate output from video data, according to certain embodiments. [Figure 4B] FIG. 1 is a block diagram of a control system configured to utilize machine learning to generate outputs from several types of data, according to certain embodiments. [Figure 5] FIG. 1 is a flow diagram of a stage identification process, according to certain embodiments. [Figure 6] FIG. 1 illustrates a flow diagram of a trigger process for automated robotic actions, according to certain embodiments. [Figure 7] FIG. 1 illustrates different types of triggered actions of a robotic system, in accordance with certain embodiments. [Figure 8] FIG. 1 is a flow diagram of a process for evaluating tasks performed during identified stages, according to certain embodiments. [Figure 9] FIG. 1 is a flow diagram of a scoring process for a medical task, according to certain embodiments. [Figure 10]FIG. 10 is a flow diagram of another scoring process for a medical task, according to certain embodiments. [Figure 11] FIG. 1 illustrates exemplary details of a robotic system, in accordance with certain embodiments. [Figure 12] FIG. 2 illustrates exemplary details of a control system, according to certain embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0012] The directions provided herein are for convenience only and do not necessarily affect the scope or meaning of the disclosure. While certain preferred embodiments and examples are disclosed below, the subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as modifications and equivalents thereof. Accordingly, the scope of claims that may arise from this specification is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described in sequence as multiple separate operations, in a manner that may be helpful in understanding a particular embodiment. However, the order of description should not be construed to imply that these operations are order-dependent. Furthermore, structures, systems, and / or devices described herein may be embodied as integrated or separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are discussed. Not necessarily all such aspects or advantages are achieved by any particular embodiment. Thus, for example, various embodiments may be implemented in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other aspects or advantages that may also be taught or suggested herein.

[0013] Certain standard anatomical terms of location may be used herein to refer to animal, i.e., human, anatomical structures with respect to preferred embodiments. While certain spatially relative terms, such as "outer," "inner," "superior," "lower," "below," "upper," "vertical," "horizontal," "top," "bottom," and similar terms, are used herein to describe the spatial relationship of one device / element or anatomical structure to another, it should be understood that these terms are used herein for ease of description to describe positional relationships between elements / structures, as illustrated in the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of elements / structures during use or operation in addition to the orientation shown in the drawings. For example, when an element / structure is described as being "above" another element / structure, it may refer to a position below or beside such other element / structure relative to the intended patient or alternative orientations of the element / structure, and vice versa.

[0014] Overview The present disclosure relates to techniques and systems for collecting and analyzing data from robotic-assisted medical procedures, such as those performed by a robotic system for stone management (e.g., urinary stone retrieval, stone fragment aspiration, etc.) or the performance of other medical procedures. A medical procedure may progress through several different phases. For example, in a ureteroscopy, the phases may include percutaneous insertion of a medical instrument into the body, navigation to the urinary stone location, laser treatment of the urinary stone, and / or basket treatment of the fragmented stone. Robotic systems typically have several sensors and input devices, enabling the generation of large amounts of data during a medical procedure. This procedural data can be used to automatically determine the different stages of the surgery. By identifying these stages, the robotic system can anticipate and prepare for the actions of the medical professional operating the robotic system during the medical procedure.

[0015] Medical systems, including robotic systems, may also enable annotating video footage of procedures with metadata identifying different steps. This allows the video footage to be more easily reviewed by users and allows for more advanced analysis of the video footage using artificial intelligence (AI). This may make it easier to evaluate and score actions performed by a user or operator by comparing them with similar actions from corresponding steps performed during other procedures. For example, video footage and associated data can be analyzed by an AI system to generate statistics about operations, such as attempts before success per step or entire procedure, the time for each step, the number of articulation commands provided by the operator, and the accuracy of needle insertion. Furthermore, data can be aggregated across several operations and used to generate statistics about types of operations in general, such as success rates, average operation time per step or entire procedure, etc. Such medical systems may also provide additional benefits, such as by generating case summaries.

[0016] In one exemplary scenario, there are distinct stages during a percutaneous renal access or other procedure. In an exemplary workflow, a user drives the scope to the desired calyx, marks the papilla, and retracts the scope to view the target papilla. The user then holds the needle, selects an insertion site, and uses a graphical user interface ("GUI") to align the needle trajectory with the target papilla. Finally, the user follows the graphical user interface to insert the needle and gain access to the kidney through the target papilla. To improve procedural efficiency and assess user skill, the medical system can label the start and end of these events and obtain ground truth data regarding whether a percutaneous access ("perc") attempt was successful.

[0017] After dividing the case data into distinct phases and generating a phase transition chart showing these phases, the transition chart can be used to evaluate the procedure. For example, one exemplary transition chart may show that the physician selected a target and insertion site but did not move forward with the needle alignment step, instead moving to a different calyx to select a new target. The chart may show that the physician did not obtain visual confirmation of access in a first percutaneous access attempt and instead moved the scope to position the needle. The chart may show that the physician performed another percutaneous access attempt using the same target, this time obtaining visual confirmation. Such charts may be displayed on a medical system GUI, as a digital or printed report, on a mobile application, and / or as a similar type of output.

[0018] Another potential benefit is providing ground truth (success / failure) annotation. Phase segmentation allows for prediction of whether a particular percutaneous access attempt will be successful, thereby serving as ground truth for the case. The medical system can track a set of feature descriptors during the needle insertion phase to determine whether percutaneous access was successful. The feature descriptors can include various quantities or metrics measured by the medical system, such as needle and scope speed and the needle's relative pose to the scope. They can also include scope articulation commands and features detected by a computer vision algorithm that detects whether the needle is visible in the camera view and quantifies how much anatomical motion is present. For example, there can be a direct correlation between visual confirmation and success. In one scenario, if the computer vision algorithm detects the needle in the endoscopic view, the percutaneous access attempt can be annotated or otherwise indicated as successful. In another scenario, the distance between the needle and the scope may be very small, but there is no visual confirmation of the needle on the scope. If the scope begins to move, it implies that the percutaneous access attempt has failed and the user is searching for the needle or driving to another calyx to select a new target. Thus, detection of scope movement in that situation can be used to annotate or otherwise indicate that the percutaneous access attempt has failed.

[0019] Another potential benefit is providing skill assessment. Phase segmentation can enable performance-phase-specific data analysis to assess physician skill and calculate case statistics intraoperatively or postoperatively. The table below shows postoperative metrics for some of the percutaneous access phases. For example, by knowing when needle insertion begins (e.g., identified via video capture, sensor data, etc.), the medical system can determine the entry point on the skin (e.g., using kinematic data, video analysis, etc.) and calculate site selection metrics such as tract length (e.g., distance from skin to nipple).

[0020] [Table 1]

[0021] For example, during the scope drive phase, the user's skill can be evaluated based on the number of articulation commands received by the system. If fewer commands are received, it means the movement was performed smoothly and indicates higher skill. If more commands are received, it means multiple attempts must be performed and indicates room for improvement. These metrics can also provide information about the parts of the anatomy the user is having difficulty navigating. The number of articulation commands may be recorded and / or displayed for the procedure or multiple procedures (all cases, all cases over a period of time, all cases performed by the user, etc.). For example, the medical system can generate metrics that compare over time across multiple procedures, for a given case, across physicians, and / or by location for the same physician.

[0022] In another example, during the needle insertion phase, the user's skill can be evaluated based on success rate and / or needle insertion accuracy. Success rate can be calculated more specifically based on kidney location, such as the lower pole, middle pole, or upper pole. Needle insertion accuracy can be compared to an average value for experts. Needle insertion accuracy may be recorded and / or displayed for the procedure or multiple procedures (e.g., all cases, all cases over a period of time, all cases performed by the user, etc.).

[0023] In a further example, during the site selection phase, the user's skill may be evaluated based on site selection time or the time it took the user to select a site and average tract length. The site selection time may be compared to an expert average. The site selection time may be recorded and / or displayed for this procedure or multiple procedures (all cases, all cases in a period of time, all cases performed by the user, etc.). The average tract length may be more specifically calculated based on kidney location, such as the inferior pole, middle pole, or superior pole. The patient's tract length may be used as an indicator of the patient's body mass index (BMI). This may allow case outcomes to be aggregated based on patient population characteristics, such as BMI values ​​or ranges.

[0024] The above table shows only some examples of possible metrics that can be evaluated. Furthermore, the above table shows only some of the specificities that may be applied to those metrics. For example, some of the specificities that apply to one metric may also apply to other metrics. In some embodiments, needle insertion accuracy can be further categorized based on kidney location. Success rates can be shown with higher specificity by comparing to an expert average or across multiple procedures (e.g., all cases, all cases over a period of time, all cases performed by a user, etc.).

[0025] Another potential benefit of such a medical system is that it provides skill assessment workflow optimization. Workflow analysis can show correlations between the sequence of workflow steps and the success and efficiency of percutaneous access. For example, an algorithm can compare cases where site selection is performed before target selection with cases where target selection is performed before site selection and assess the impact on percutaneous access time and accuracy.

[0026] Such medical systems can be used in several types of procedures, including ureteroscopy. Kidney stone disease, also known as urolithiasis, is a relatively common medical condition involving the formation of solid pieces of material in the urinary tract, referred to as "kidney stones," "urinary calculi," "renal stones," "kidney stones," or "nephrolithiasis." Urinary stones can form and / or be found in the kidneys, ureters, and bladder (referred to as "bladder stones"). Such urinary stones result from mineral concentrations and can cause significant abdominal pain if they reach a size sufficient to obstruct urine flow through the ureter or urethra. Urinary stones can be formed from calcium, magnesium, ammonia, uric acid, cysteine, or other compounds.

[0027] To remove urinary stones from the bladder and ureters, a surgeon can insert a ureteroscope through the urethra and into the urinary tract. The ureteroscope typically includes an endoscope at its distal end configured to allow visualization of the urinary tract. The ureteroscope may also include a stone extraction mechanism, such as a basket retrieval device, for capturing or fragmenting urinary stones. During a ureteroscopy procedure, one physician / technician can control the position of the ureteroscope, and another physician / technician can control the stone extraction mechanism.

[0028] In many embodiments, the techniques and systems are discussed in the context of minimally invasive procedures. However, it should be understood that the techniques and systems can be implemented in the context of any medical procedure, including, for example, percutaneous surgery, non-invasive procedures, therapeutic procedures, diagnostic procedures, non-percutaneous procedures, or other types of procedures in which access to a target location is gained by making a puncture and / or small incision in the body to insert a medical instrument. For example, such techniques can be used in tumor biopsy or ablation for urology and bronchoscopy, where an automated biopsy operation can be triggered when the system detects proximity to a suspicious site. Endoscopic procedures can include bronchoscopy, ureteroscopy, gastroscopy, nephrology, kidney stone removal, and the like. Furthermore, in many embodiments, the techniques and systems are described as being implemented as robotic-assisted procedures. However, it should be understood that the techniques and systems can also be implemented in other procedures, such as fully robotic medical procedures.

[0029] For ease of illustration and discussion, the techniques and systems will be discussed in the context of removing urinary stones, such as kidney stones from the kidney, although, as noted above, the techniques and systems can be used to perform other procedures.

[0030] medical system 1 illustrates an example of a medical system 100 for performing or assisting in performing a medical procedure in accordance with one or more embodiments. Embodiments of the medical system 100 may be used for surgical and / or diagnostic procedures. The medical system 100 includes a robotic system 110 configured to engage and / or control a medical instrument 120 and perform a procedure on a patient 130. The medical system 100 also includes a control system 140 configured to interface with the robotic system 110, provide information about the procedure, and / or perform various other operations. For example, the control system 140 may include a display 142 that presents a user interface 144 to assist a physician 160 in using the medical instrument 120. Additionally, the medical system 100 may include a table 150 configured to hold the patient 130 and / or an imaging sensor 180, such as a camera, X-ray, computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET) device, or the like.

[0031] In some embodiments, a physician performs a minimally invasive medical procedure, such as a ureteroscopy. The physician 160 can interact with the control system 140 to control the robotic system 110 and navigate the medical instrument 120 (e.g., a basket retrieval device and / or a scope) from the urethra and into the kidney 170 where the stone 165 is located. The control system 140 can provide information about the medical instrument 120 via the display 142, such as real-time images from the medical instrument 120 or the imaging sensor 180, to assist the physician 160 in navigation. Once the site of the kidney stone is reached, the medical instrument 120 can be used to break up and / or capture the urinary stone 165.

[0032] In some implementations using the medical system 100, the physician 160 can perform a percutaneous procedure. Illustratively, if the patient 130 has a kidney stone 165 in the kidney 170 that is too large to remove through the urinary tract, the physician 160 can perform a procedure to remove the kidney stone through a percutaneous access point on the patient 130. For example, the physician 160 can interact with the control system 140 to control the robotic system 110 to navigate the medical instrument 120 (e.g., a scope) from the urethra into the kidney 170 where the stone 165 is located. The control system 140 can provide information about the medical instrument 120, such as real-time images from the medical instrument 120 or the imaging sensor 180, via the display 142 to assist the physician 160 in navigating the medical instrument 120. Upon reaching the site of the kidney stone, the medical instrument 120 can be used to specify a target location (e.g., a desired point for accessing the kidney) for a second medical instrument (not shown) to percutaneously access the kidney. To minimize damage to the kidney, the physician 160 may designate a particular papilla as a target location for entry into the kidney with the second medical instrument. However, other target locations may be designated or determined. Once the second medical instrument reaches the target location, the physician 160 may use the second medical instrument and / or another medical instrument to remove the kidney stone from the patient 130, such as through a percutaneous access point. While the percutaneous procedures described above are discussed in the context of using the medical instrument 120, in some implementations, the percutaneous procedures may be performed without the assistance of the medical instrument 120. Additionally, the medical system 100 may be used to perform a variety of other procedures.

[0033] Minimally invasive surgery offers the possibility of video recording of the procedure, as cameras (e.g., the scope of the medical instrument 120) can be inserted into the body during the procedure. Additional cameras and sensors placed outside the body can be used to capture video and / or data of the patient and the medical system 100. For example, operating room (OR) camera(s) can capture video of activity within the OR, such as the operator's or physician's hand movements, needle location, fluid bag changes, and patient bleeding. Details such as the number of contrast injections during fluoroscopy may also be captured by OR cameras and used to estimate radiation exposure to the patient. Audio recorded on the video can also be used to aid in stage identification. For example, some robotic systems beep or otherwise generate audible noise when laser processing is occurring. Videos can be archived and later used for reasons such as cognitive training, skills assessment, and workflow analysis.

[0034] Computer vision, a form of artificial intelligence (AI), enables quantitative analysis of video by computers for object and pattern identification. For example, in endoscopic surgery, AI video systems can be used for gesture / task classification, skill assessment, tool type recognition, and shot / event detection and retrieval. AI systems can watch videos of surgical procedures and track the movements and timing of instruments used during the procedure. Using metrics, AI systems can track tool timing, such as which instruments were used when and for how long. Additionally, AI systems can track instrument paths, which can be useful for evaluating procedures or identifying stages in a procedure. AI systems can determine how far apart tools are within the surgical field, which can be correlated to surgical quality, as better surgeons tend to manipulate instruments in focused areas. AI systems can also determine metrics to evaluate multiple aspects of a medical professional's performance, including the medical professional's economy of motion, how often they switched back and forth between instruments, and their efficiency at each step of the procedure.

[0035] In the example of FIG. 1 , medical instrument 120 is implemented as a basket retrieval device. Therefore, for ease of description, medical instrument 120 will also be referred to as “basket retrieval device 120.” However, medical instrument 120 may be implemented as various types of medical instruments, including, for example, a scope (sometimes referred to as an “endoscope”), a needle, a catheter, a guidewire, a lithotriptor, forceps, a vacuum, a scalpel, combinations of the above, etc. In some embodiments, the medical instrument is a steerable device, while in other embodiments, the medical instrument is a non-steerable device. In some embodiments, a surgical tool refers to a device configured to puncture or be inserted through a body structure, such as a needle, scalpel, or guidewire. However, a surgical tool can also refer to other types of medical equipment. In other embodiments, multiple medical instruments may be used. For example, an endoscope may be used in conjunction with basket retrieval device 120. In some embodiments, medical instrument 120 may be a composite device incorporating several instruments, such as a vacuum, a basket retrieval device, a scope, or various combinations of instruments.

[0036] In some embodiments, the medical device 120 may include a radio frequency identification (RFID) chip for identifying the medical device 120. The medical system 100 may include an RFID reader for reading the RFID chip in the medical device to aid in device identification. Such information may be used to facilitate procedure and stage identification. For example, if the RFID data identifies the device as a needle, the stage may relate to needle insertion, but determining the exact stage may require combining the RFID data with additional data, such as video, device status, telemetry (e.g., magnetic tracking, robotic data, fluidics data, etc.), etc.

[0037] The robotic system 110 can be configured to facilitate a medical procedure. The robotic system 110 can be configured in a variety of ways depending on the particular procedure. The robotic system 110 can include one or more robotic arms 112 (robotic arms 112(a), 112(b), 112(c)) configured to engage and / or control a medical instrument 120 to perform the procedure. As shown, each robotic arm 112 can include multiple arm segments coupled to joints, thereby providing multiple degrees of mobility. In the example of FIG. 1 , the robotic system 110 is positioned adjacent the lower torso of the patient 130, and the robotic arms 112 are actuated to engage and position the medical instrument 120 for access to an access point, such as the urethra of the patient 130. Once the robotic system 110 is properly positioned, the medical instrument 120 can be inserted into the patient 130 robotically using the robotic arms 112, manually by a physician 160, or a combination thereof.

[0038] The robotic system 110 may also include a base 114 coupled to the one or more robotic arms 112. The base 114 may include various subsystems, such as control electronics, power supplies, pneumatics, light sources, actuators (e.g., motors for moving the robotic arms), control circuits, memory, and / or communications interfaces. In some embodiments, the base 114 includes input / output (I / O) devices 116 configured to receive inputs, such as user inputs, for controlling the robotic system 110 and provide outputs, such as patient status, medical instrument location, etc. The I / O devices 116 may include a controller, a mouse, a keyboard, a microphone, a touchpad, other input devices, or a combination of the above. The I / O devices may include output components, such as a speaker, a display, a haptic feedback device, other output devices, or a combination of the above. In some embodiments, the robotic system 110 is mobile (e.g., the base 114 includes wheels) so that the robotic system 110 can be positioned at an appropriate or desired location for a procedure. In other embodiments, the robotic system 110 is a stationary system. Additionally, in some embodiments, the robotic system 110 is integrated into the table 150 .

[0039] The robotic system 110 can be coupled to any component of the medical system 100, such as the control system 140, the table 150, the imaging sensor 180, and / or the medical instrument 120. In some embodiments, the robotic system is communicatively coupled to the control system 140. In one example, the robotic system 110 can receive control signals from the control system 140 to perform actions such as positioning the robotic arm 112 in a particular manner, manipulating the scope, etc. In response, the robotic system 110 can control components of the robotic system 110 to perform actions. In another example, the robotic system 110 can receive images from the scope depicting the internal anatomical structures of the patient 130 and / or transmit the images to the control system 140 (which can then be displayed on the control system 140). Additionally, in some embodiments, the robotic system 110 is coupled to components of the medical system 100, such as the control system 140, to receive data signals, power, etc. Other devices, such as other medical instruments, IV bags, blood packs, etc., may also be coupled to the robotic system 110 or other components of the medical system 100 depending on the medical procedure being performed.

[0040] Control system 140 can be configured to provide various functions to assist in the performance of a medical procedure. In some embodiments, control system 140 is coupled to robotic system 110 and can operate in cooperation with robotic system 110 to perform a medical procedure on patient 130. For example, control system 140 can communicate with robotic system 110 via a wireless or wired connection (e.g., to control robotic system 110, basket retrieval device 120, receive images captured by a scope, etc.), control fluid flow through robotic system 110 via one or more fluid channels, provide power to robotic system 110 via one or more electrical connections, provide optical signals to robotic system 110 via one or more optical fibers or other components, etc. Additionally, in some embodiments, control system 140 can communicate with a scope to receive sensor data. Additionally, in some embodiments, control system 140 can communicate with table 150 to position table 150 in a particular orientation or otherwise control table 150.

[0041] 1 , control system 140 includes various I / O devices configured to assist physician 160 or others in performing a medical procedure. In some embodiments, control system 140 includes input devices 146 employed by physician 160 or another user to control basket retrieval device 120. For example, input devices 146 can be used to navigate basket retrieval device 120 within patient 130. Physician 160 can provide input via input devices 146, and in response, control system 140 can send control signals to robotic system 110 to operate medical instrument 120.

[0042] In some embodiments, the input device 146 is a controller similar to a game controller. The controller can have multiple axes and buttons that can be used to control the robotic system 110. While the input device 146 is shown as a controller in the example of FIG. 1 , the input device 146 can be implemented as various types of I / O devices or combinations of various types of I / O devices, such as a touchscreen / pad, a mouse, a keyboard, a microphone, a smart speaker, etc. As also shown in FIG. 1 , the control system 140 can include a display 142 that provides various information related to the procedure. For example, the control system 140 can receive real-time images captured by a scope and display the real-time images via the display 142. Additionally or alternatively, the control system 140 can receive signals (e.g., analog, digital, electrical, acoustic / sonic, pneumatic, tactile, hydraulic, etc.) from sensors associated with the medical monitor and / or the patient 130, and the display 142 can present information related to the health of the patient 130 and / or the patient 130's environment. Such information may include, for example, information displayed via a medical monitor, such as heart rate (e.g., electrocardiogram (ECG), heart rate variability (HRV), etc.), blood pressure / blood flow velocity, muscle biosignals (e.g., electromyography (EMG)), body temperature, oxygen saturation (e.g., SpO2), carbon dioxide (CO2), brain waves (e.g., electroencephalogram (EEG)), environmental temperature, etc.

[0043] 1 also illustrates various anatomical structures of a patient 130 that are relevant to certain aspects of the present disclosure. Specifically, the patient 130 includes a kidney 170 that is fluidly connected to a bladder 171 via a ureter 172, and a urethra 173 that is fluidly connected to the bladder 171. As shown in the enlarged depiction of the kidney 170, the kidney includes a calyx 174 (e.g., a major calyx and a minor calyx), a renal papilla (including a renal papilla 176, also referred to as "papilla 176"), and a renal pyramid (including a renal pyramid 178). In these examples, the kidney stone 165 is located proximate to the papilla 176. However, kidney stones may be located elsewhere within the kidney 170.

[0044] As shown in FIG. 1 , to remove a kidney stone 165 in an exemplary minimally invasive procedure, a physician 160 may position the robotic system 110 at the foot of the table 150 to begin delivery of the medical instrument 120 to the patient 130. Specifically, the robotic system 110 may be positioned adjacent the patient's 130 lower abdomen and aligned for direct, linear access to the patient's 130 urethra 173. From the foot of the table 150, the robotic arm 112(B) may be controlled to provide access to the urethra 173. In this example, the physician 160 inserts the medical instrument 120 at least partially into the urethra along this direct, linear access path (sometimes referred to as a “virtual rail”). The medical instrument 120 may include a lumen configured to receive a scope and / or a basket retrieval device, thereby assisting in the insertion of these devices into the patient's 130 anatomy.

[0045] Once the robotic system 110 is properly positioned and / or the medical instrument 120 is at least partially inserted into the urethra 173, a scope may be inserted into the patient 130 robotically, manually, or a combination thereof. For example, the physician 160 may connect the medical instrument 120 to the robotic arm 112(C). The physician 160 may then interact with the control system 140, such as the input device 146, to navigate the medical instrument 120 within the patient 130. For example, the physician 160 may provide input via the input device 146 to control the robotic arm 112(C) to navigate the basket retrieval device 120 through the urethra 173, the bladder 171, the ureter 172, and to the kidney 170.

[0046] Control system 140 may include various components (sometimes referred to as "subsystems") to facilitate its function. For example, control system 140 may include various subsystems such as control electronics, power supplies, pneumatics, light sources, actuators, control circuits, memory, and / or communication interfaces. In some embodiments, control system 140 includes a computer-based control system that stores executable instructions that, when executed, cause various operations to be performed. In some embodiments, control system 140 is mobile, as shown in FIG. 1 , while in other embodiments, control system 140 is a fixed system. Although various functions and components are discussed as being implemented by control system 140, any of the functions and / or components may be integrated into and / or performed by other systems and / or devices, such as robotic system 110 and / or table 150.

[0047] The medical system 100 can provide various benefits, such as providing guidance (e.g., instrument tracking, patient status, etc.) to assist the physician in performing the procedure, allowing the physician to perform the procedure from an ergonomic position without requiring awkward arm movements and / or positions, allowing a single physician to perform the procedure using one or more medical instruments, avoiding radiation exposure (e.g., associated with fluoroscopy techniques), allowing the procedure to be performed in a single surgical setting, and providing continuous suction for more efficient object removal (e.g., removing kidney stones). Additionally, the medical system 100 can provide non-radiation-based navigation and / or localization techniques to reduce the physician's exposure to radiation and / or reduce the amount of equipment in the operating room. Furthermore, the medical system 100 can divide its functions into a control system 140 and a robotic system 110, each of which may be independently mobile. Such division of functionality and / or mobility can allow control system 140 and / or robotic system 110 to be located where it is most appropriate for a particular medical procedure, thereby maximizing the working area around the patient and / or providing an optimized location for the physician to perform the procedure. For example, many aspects of the procedure may be performed by robotic system 110 (located relatively close to the patient), while the physician manages the procedure remotely from control system 140 (which may be located remotely).

[0048] In some embodiments, the control system 140 can function even when located in a different geographic location than the robotic system 110. For example, in a telemedicine implementation, the control system 140 is configured to communicate with the robotic system 110 over a wide area network. In one scenario, the physician 160 may be located in one hospital along with the control system 140, while the robotic system 110 is located in a different hospital. The physician can then perform medical procedures remotely. This may be beneficial when remote hospitals, such as rural hospitals, have limited expertise in a particular procedure. These hospitals can rely on more experienced physicians in other locations. In some embodiments, the control system 140 can pair with various robotic systems 110, for example, by selecting a particular robotic system and forming a secure network connection (e.g., using a password, encryption, authentication token, etc.). Thus, a physician at one location may be able to perform medical procedures at various different locations by establishing connections with robotic systems 110 located at each of the various different locations.

[0049] In some embodiments, the robotic system 110, the table 150, the medical instruments 120, the needle, and / or the imaging sensor 180 are communicatively coupled to one another via a network, which may include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), one or more local area networks (LANs), one or more wide area networks (WANs), one or more Internet area networks (IANs), one or more cellular networks, the Internet, etc. Additionally, in some embodiments, the control system 140, the robotic system 110, the table 150, the medical instruments 120, and / or the imaging sensor 180 are connected via one or more support cables for communication, fluid / gas exchange, power exchange, etc.

[0050] 1 , in some embodiments, the medical system 100 includes and / or is associated with a medical monitor configured to monitor the health of the patient 130 and / or the environment in which the patient 130 is located. For example, the medical monitor can be located in the same environment in which the medical system 100 is located, such as in an operating room. The medical monitor can be physically and / or electrically coupled to one or more sensors configured to detect or determine one or more physical, physiological, chemical, and / or biological signals, parameters, characteristics, states, and / or conditions associated with the patient 130 and / or the environment. For example, the one or more sensors can be configured to determine / detect any type of physical property, including temperature, pressure, vibration, kinematic / tactile characteristics, sound, optical levels or properties, load or weight, flow rate (e.g., of a target gas and / or liquid), amplitude, phase, and / or orientation of magnetic and electric fields, concentration of elements associated with a substance in gas, liquid, or solid form, etc. The one or more sensors can provide sensor data to a medical monitor, which can present information regarding the health of the patient 130 and / or the environment of the patient 130. Such information can include, for example, heart rate (e.g., ECG, HRV, etc.), blood pressure / blood velocity, muscle biosignals (e.g., EMG), body temperature, oxygen saturation (e.g., SpO2), CO2, brain waves (e.g., EEG), environmental temperature, etc., displayed via the medical monitor. In some embodiments, the medical monitor and / or the one or more sensors are coupled to a control system 140, which is configured to provide information regarding the health of the patient 130 and / or the environment of the patient 130.

[0051] Urinary stone capture 2A-2B show perspective views of the medical system 100 during a urinary stone trapping procedure. In these examples, the medical system 100 is positioned in an operating room to remove a kidney stone from a patient 130. In many instances of such procedures, the patient 130 is positioned in a modified supine position, with the patient 130 tilted slightly to the side, to access the patient's posterior or lateral sides. The urinary stone trapping procedure may also be performed with the patient in a normal supine position, as shown in FIG. 1. While FIGS. 2A and 2B illustrate the use of the medical system 100 to perform a minimally invasive procedure to remove a kidney stone from a patient 130, the medical system 100 may be used to remove kidney stones in other ways and / or to perform other procedures. Furthermore, the patient 130 may be positioned in other positions as desired for the procedure. Various actions are described in FIGS. 2A and 2B and throughout this disclosure as being performed by a physician 160. It should be understood that these actions may be performed directly by the physician 160, indirectly by the physician with the assistance of the medical system 100, by a user under the direction of the physician, by another user (e.g., a technician), and / or by any other user.

[0052] 2A and 2B, any of the robotic arms 112 may be used to perform those functions. Additionally, any additional robotic arms and / or systems may be used to perform the procedure. Additionally, the robotic system 110 may be used to perform other portions of the procedure.

[0053] 2A , basket retrieval device 120 is maneuvered into kidney 170 to approach urinary stone 165. In some scenarios, physician 160 or other user uses input device 146 to directly control the movement of basket retrieval device 120. Such directly controlled movement can include insertion / retraction, bending basket retrieval device 120 left or right, rotation, and / or systematic opening / closing of the basket. Using various motions, basket retrieval device 120 is positioned near the stone.

[0054] In some embodiments, a laser, shock wave device, or other device is used to fragment the concretion. The laser or other device may be incorporated into basket retrieval device 120 or may be a separate medical instrument. In some situations, concretion 165 is small enough that breaking the concretion into smaller pieces is not necessary.

[0055] 2B, the open basket is manipulated to encircle the urinary stone 165 or urinary stone fragment. The basket retrieval device 120 is then withdrawn from the kidney 170 and then removed from the patient's body.

[0056] If additional stones (or larger fragments of the fragmented stone 165) are present, the basket retrieval device 120 may be reinserted into the patient to capture the remaining larger fragments. In some embodiments, a vacuum instrument may be used to facilitate removal of the small pieces. In some circumstances, the stone fragments may be small enough that they can be passed naturally by the patient.

[0057] Stage Segmentation and Stage Recognition Automated surgical workflow analysis can be used to detect different stages in a procedure and evaluate surgical skills and procedural efficiency. Data collected during a procedure (e.g., video data) can be segmented into sections using machine learning methods, including, but not limited to, hidden Markov models (HMMs) and long-term-short-memory (LTSM) networks.

[0058] In surgical stage segmentation, captured medical procedure data is automatically segmented into stages using input data from the operating room to identify the stages. Segmentation can be performed in real time during the procedure or post-operatively on recorded data. In one embodiment, the surgical data can be pre-processed using dynamic time warping to divide the stages into equivalent segments. Input data can consist of instrument signals, annotations, instrument (e.g., EM) tracking, or information obtained from video.

[0059] Surgical workflow recognition can be performed at different levels of granularity depending on the procedure. This can be done for stages and steps (higher level) or gestures and activities (lower level). Surgical stage recognition can be performed on time series, kinematic data, and video data using machine learning techniques such as HMM, Gaussian Mixture Model (GMM), and Support Vector Machine (SVM), as well as deep learning-based methods for stage recognition from video data using Convolutional Neural Networks (CNN). For surgical gesture and activity recognition, similar methods (SVM, Markov model) can be used primarily for video data or a combination of video and kinematic data, as well as more recent deep learning-based methods such as CNN, which can be used to recognize the presence of tools, tasks, and activities in video data. Stage segmentation can use multiple data sources to segment case data into different subtasks, as shown in Figure 3, or a single data source, such as a video, can be used to classify the current stage, as shown in Figure 4. In FIG. 4, additional data (eg, sensor data or UI data) can then be incorporated to further refine the output generated by control system 140.

[0060] 3, the control system 140 receives various input data from the medical system 100. Such inputs can include video data 305 captured by the imaging sensor 180, robotic sensor data 310 from one or more sensors of the robotic system 110, and user interface (UI) data received from the input device 146.

[0061] The video data 305 can include video captured from a scope deployed within the patient, video captured from a camera in the operating room, and / or video captured by a camera on the robotic system 110. The robotic sensor data 310 can include kinematic data from the robotic system 110 (e.g., using vibration, accelerometer, positioning, and / or gyroscope sensors), device status, temperature, pressure, vibration, haptic / tactile characteristics, sound, optical levels or properties, load or weight, flow rates (e.g., of target gases and / or liquids), magnetic and electric field amplitudes, phases, and / or orientations, constituent concentrations for substances in gas, liquid, or solid form, etc. The UI data 315 can include button presses, menu selections, page selections, gestures, voice commands, and / or the like made by a user and captured by input devices on the medical system 100. Patient sensor data, such as that described above in FIG. 1, can also be used as input to the control system 140.

[0062] Control system 140 can analyze video data 305 (e.g., using machine learning algorithms) and use robotic sensor data 310 and UI data 315 to identify stages of a medical procedure. In one example, a medical procedure, such as a ureteroscopy, includes several tasks (e.g., Task 1 through Task 5). Each task may be performed during one or more stages of the medical procedure. In the example shown in FIG. 3 , Task 1 is performed during Stage 1. Task 2 is performed during Stages 2 and 4. Task 3 is performed during Stages 3 and 5. Task 4 is performed during Stages 6 and 8. Task 5 is performed during Stage 7. Time 1 (T1) indicates the time it takes to complete Stage 1, Time 2 (T2) indicates the time it takes to complete Stage 2, and Time 3 (T3) indicates the time it takes to complete Stage 3. Other procedures may have a different number of tasks and / or a different number of stages.

[0063] For robotic procedures where there are manual and automated tasks, surgical stage detection can be used to automatically and seamlessly transition between the manual and automated tasks. For example, T1 may correspond to a manual task, T2 may be an automated task, and T3 may again be a manual task. In one embodiment, when the target selection stage is active, the target selection step can be performed autonomously by the robot driving the scope. Alternatively, the user can perform site selection by picking a point on the skin using an EM marker, and the robot can autonomously align the needle to the target insertion trajectory.

[0064] 4A shows a block diagram of a control system 140 configured to generate output from video data from a medical procedure using machine learning, according to certain embodiments. In some embodiments, the control system 140 is configured to first process the video data 305 using a machine learning algorithm. In one embodiment, the video data 305 is processed by a CNN 405 to generate output 412, identifying features captured in the video, such as surgical tools, stones, and human anatomical structures (e.g., nipples). Such identified features 415, along with the original video, may be provided as input to a recurrent neural network (RNN) 410. The RNN 410 may then process the video data 305 and the identified features 415 to generate output 412 for identifying a stage 420 in the medical procedure.

[0065] Supplemental data, such as robot sensor data 310 or UI data 315, can then be used to further refine (e.g., increase accuracy or increase the number of identifications) the identified features 415 and identified steps 420. In other embodiments, the robot sensor data 310 and / or UI data 315 can be used prior to the processing of the video data 305 by the control system 140 to narrow the possible options considered by the control system 140. For example, the supplemental data can be used to identify a specific procedure, which narrows the universe of possible tasks and steps to those that correspond to the specific procedure. The control system 140 can then limit the identified features 415 and identified steps 420 to those that correspond to the specific procedure. For example, if a task is initially identified in the video data 305 by the control system 140, but the task is not associated with a specific procedure, the control system 140 can reprocess the video until the task is re-identified as a task that corresponds to the specific procedure.

[0066] After completing processing of the video data 305, the control system 140 can generate an annotated video that includes the identified features 415 and / or the identified stages 420. Such annotations may be stored as part of the video (e.g., in the same video file), in metadata stored in a database along with the video, and / or in other data formats.

[0067] Creating metadata-enhanced videos makes it easier to use videos to review medical procedures. For example, viewers can jump forward or backward to a particular step of interest rather than manually searching for when a particular step occurred. Additionally, multiple videos can be more easily processed to aggregate data and generate metrics. For example, multiple videos can be searched for instances of a particular step (e.g., needle insertion or stone capture) and analyzed to generate metrics related to that step (e.g., success rate, average attempts, number of attempts, etc.).

[0068] While FIG. 4A shows video data 305 being processed by control system 140, other types of data may be processed by control system 140, either serially or in parallel with one another. For example, such data may include robotic system 110 data, such as instrument positioning as measured by electromagnetic tracking sensors, how far the scope is inserted, how far the scope is articulated, whether the basket is open or closed, how far the basket is inserted, and / or the connectivity status of the robotic system. Data may be provided as input to a single neural network or multiple neural networks. For example, each different type of sensor (e.g., video, device status, telemetry, e.g., magnetic tracking, robotic data, and / or fluid data) may have its own network, and the outputs of the networks may be concatenated before the final stage classification layer to obtain a single stage prediction.

[0069] 4B shows one such embodiment in which different types of data from different devices and / or sensors are processed by different neural networks. The video data 305 can be processed by a first neural network 425 (e.g., a CNN and / or RNN as illustrated in FIG. 4A), the robot sensor data 310 can be processed by a second neural network 430, and the UI data can be processed by a third neural network 435. The outputs from the different neural networks can then be combined to generate an output 412 (e.g., a stage prediction) for the medical system 100.

[0070] Stage Identification Process Figure 5 is a flow diagram of a stage identification process 500, according to certain embodiments. The stage identification process 500 may be performed by the control system 140 or by another component of the medical system 100 of Figure 1. The following describes one possible sequence for the process, although other embodiments may perform the process in a different order or may include additional steps or may exclude one or more of the steps described below.

[0071] In block 505, the control system 140 identifies input from the UI of the robotic system. For example, the input may be received from an input device 146, such as a controller or a touchscreen. Possible inputs may include a selection of a procedure stage or a selection of a UI screen associated with a particular procedure stage. For example, a first screen may list options for a first procedure, while a second screen may list options for a second procedure. When a user makes selections on the first screen, these selections indicate that the user is performing a first procedure. When a user makes selections on the second screen, these selections indicate that the user is performing a second procedure. Thus, by organizing the screens of the UI to correspond to particular stages, the control system 140 can obtain stage information based on the user's selection. In another example, one embodiment of the medical system 100 may include a UI having a first screen showing selectable stone management procedures, such as ureteroscopy, percutaneous access, or mini-percutaneous nephrolithotomy (PCNL). If the user selects ureteroscopy, the control system 140 may determine that the step is related to ureteroscopy (e.g., intrarenal basket procedure, laser procedure, and / or survey). Similarly, selecting other stone management procedures indicates that the step is related to the corresponding procedure.

[0072] In block 510, the control system 140 determines a procedure from a set of procedures based on at least one of the UI input and the sensor data. As described above, input from the UI interface can be used to identify the current possible procedure stage. In addition, robotic sensor data can also be used to identify the procedure. For example, if it is determined that the arm of the robotic system 110 is approaching a patient while holding a surgical instrument, the control system 140 may determine that the current procedure involves inserting a medical instrument.

[0073] In block 515, the control system 140 can narrow the set of identifiable procedure steps to a subset of procedure steps based on the determined procedure. For example, laser treatment can be associated with a task or step such as activating the laser or deactivating the laser. Basket treatment can be associated with a task or step such as capturing a stone or retracting a basket. Insertion of the medical instrument 120 can be associated with aligning the instrument with a target and inserting the instrument into the target site. In one example, if the control system 140 determines that the current procedure is a basket procedure during ureteroscopy, the control system 140 can narrow the possible steps to capturing a stone or retracting a basket.

[0074] In block 520, the control system 140 may determine the position of the robotic manipulator (e.g., the robotic arm 112) from sensor data of the robotic system 110. As described in Figure 3, various types of sensors may be used to generate sensor data, which may then be used to determine the position.

[0075] At block 525, the control system 140 can perform an analysis of the captured video. In some embodiments, such as that described in FIG. 4, a machine learning algorithm is used to perform the analysis and generate an output, such as identified features and a tentative identification of the stage. The output can include identification of a physical object, such as a surgical tool or part of an anatomical structure. For example, if the control system 140 identifies a ureter in the captured video, it indicates that the stage is not related to percutaneous access. Similarly, identifying a papilla indicates that the stage is not related to a basket procedure. Identification of other types of anatomical structures can similarly be used to eliminate the possibility of a particular stage.

[0076] In block 530, the control system 140 can identify a stage from the subset of procedure stages based on at least the position of the robotic manipulator and the performed analysis. For example, if the control system 140 receives a basket input via the controller, the control system 140 can determine that the stage is one of the basket processing stages. Additionally, if the performed analysis identifies that the captured video shows the basket approaching a fragmented kidney stone, the control system 140 can determine that the current stage is capturing the stone. In another example, if the performed analysis identifies that the captured video shows the basket being withdrawn from a fragmented kidney stone, the control system 140 can determine that the current stage is retracting the basket into a sheath. In a further example, kinematic data from the robotic system 110 can indicate that a medical instrument is being withdrawn from within a patient, and the control system 140 can determine that the current stage is retracting the basket into a sheath.

[0077] At block 535, control system 140 may generate video markers for the identified stages of the captured video. The video markers may be embedded as metadata in the same file as the video, as separate files associated with the video file, as metadata stored in a database for video annotation, etc.

[0078] In some embodiments, a video file is annotated to allow a viewer of the video file to jump to specific stages within the video. For example, the video may be divided into chapters or segments corresponding to different stages. In one embodiment, the seek bar of the video may be marked with colored segments corresponding to different stages, each stage being indicated by a different color.

[0079] In block 550, control system 140 may determine whether the end of the video has been reached. If yes, process 500 may end. If no, process 500 may loop back to block 520 to continue identifying additional stages. For example, process 500 may loop once, twice, three times, or more times to identify a first stage, a second stage, a third stage, or more stages. The captured video may then end with one or more video markers, depending on the number of stages identified.

[0080] Triggering automatic actions 6 is a flow diagram of a trigger process 600 for automated robotic actions, according to certain embodiments. The trigger process 600 can be performed by the control system 140 or by another component of the medical system 100 of FIG. 1. The following describes one possible sequence for the process, although other embodiments may perform the process in a different order or may include additional steps or may exclude one or more of the steps described below.

[0081] In block 605, the control system 140 may determine the state of the robotic manipulator (e.g., the robotic arm 112) from sensor data (e.g., kinematic data) of the robotic system 110. As described in Figure 3, various types of sensors may be used to generate sensor data, which may then be used to determine the position or other state of the robotic manipulator.

[0082] In block 610, the control system 140 can determine an input to initiate an action of the robotic manipulator. For example, the input can be from a user operating a controller to control the basket device. In another example, the input can be a screen or menu selection on the UI of the medical system 100.

[0083] In block 615, the control system 140 may perform an analysis of the captured video. In some embodiments, such as that described in Figure 4, machine learning algorithms are used to perform the analysis and generate outputs such as identified features and tentative identification of stages.

[0084] In block 620, the control system 140 can identify a stage of the medical procedure based at least on the state of the manipulator, the identified input, and the performed analysis. For example, if the control system 140 receives a basket input via the controller, the control system 140 can determine that the stage is one of the basket processing stages. Additionally, if the performed analysis identifies that the captured video shows the basket approaching a fragmented kidney stone, the control system 140 can determine that the current stage is capturing the stone. In another example, if the performed analysis identifies that the captured video shows the basket being withdrawn from a fragmented kidney stone, the control system 140 can determine that the current stage is retracting the basket into a sheath. In a further example, kinematic data from the robotic system 110 can indicate that the medical instrument is being withdrawn from within the patient, and the control system 140 can determine that the current stage is retracting the basket into a sheath.

[0085] In block 625, the control system 140 can trigger an automatic action in the robotic system 110 based on the identified stage. The triggered action can vary based on the type of procedure being performed. Some possible actions are shown in blocks 630, 635, and 640. In block 630, the robotic system 110 performs an action during a ureteroscopy laser procedure. In block 635, the robotic system 110 performs an action during insertion of a medical instrument, such as a needle. In block 635, the robotic system 110 performs an action during a ureteroscopy basket procedure. After invoking an action by the robotic system 110, the trigger process 600 can end. FIG. 7 provides additional details regarding specific actions that can be triggered.

[0086] FIG. 7 illustrates different types of triggered actions of the robotic system 110, according to an embodiment. Actions may be triggered in response to identifying a current stage of operation or identifying a user action. In some embodiments, actions may be fully automatic and executed without requiring additional input from the user. In other embodiments, actions may be partially automated and require confirmation from the user before being executed by the robotic system 110. Different combinations of stages may be executed based on the procedure being performed by the robotic system 110. Some exemplary procedures include (retrograde) ureteroscopy, percutaneous nephrolithotomy (PCNL), mini-PCNL, etc. For example, ureteroscopy may include a survey stage (not shown), a laser treatment stage, and a basket treatment stage. PCNL may include a percutaneous access stage, a survey stage, a laser treatment stage, and a basket treatment stage. Mini-PCNL may include additional alignment and / or aspiration stages.

[0087] For example, during laser treatment 705, actions that may be triggered include applying a laser to stone 710 and stopping the laser when not aimed at stone 715. In one scenario, the robotic system 110 can use various sensors (e.g., cameras) to detect when the laser is aimed at the stone. It may then determine the size of the stone, for example, by using a machine learning algorithm trained using recordings of previous ureteroscopy procedures or by using a traditional computer vision algorithm (e.g., by comparing the known size of the basket to the size of the stone). Based on the determined size, the robotic system 110 can then determine an initial laser treatment time based on recorded laser treatment times for stones of similar size and / or type. The robotic system 110 can then stop the laser after the determined laser treatment time or when it detects that the stone has fragmented. In other scenarios, the user can provide additional input, such as setting the laser treatment time or providing permission for the laser to be activated by the robotic system.

[0088] In another scenario, application of the laser may be triggered by a user, and deactivation of the laser is automatically triggered by the robotic system 110. For example, the robotic system 110 can use its sensors to detect when the laser target drifts from the stone or is otherwise not centered on the stone, and deactivate the laser in response.

[0089] In another example, actions that may be triggered during basket processing 725 include capturing a concretion inside basket 730 and retracting the basket into sheath 735. In one scenario, the robotic system 110 can trigger actuation of basket 730 when it detects that basket 730 is aligned with and within a specified distance from the concretion. The basket 730 can then be actuated to capture the concretion. The robotic system 110 can then use its sensors (e.g., a camera or pressure sensor) to determine if a concretion is captured inside basket 730 and trigger retraction of the basket into sheath 735. The user may then retract the sheath from the patient's body, thereby removing the concretion. In another example, actions that may be triggered during percutaneous access 740 include target (calyx) selection 745, insertion site selection 750, and needle insertion 755 into the target site. In one scenario, the robotic system 110 may determine a target and an insertion site at the target (e.g., marked by a user or identified by the system). The robotic system 110 may then wait for confirmation from the user to proceed. After receiving confirmation, the robotic system 110 may then insert the needle (or other instrument) into the target site.

[0090] In another example, during a mini-PCNL procedure, additional stages can include robotic alignment with the PCNL sheath 765 and laser treatment of the stone with active irrigation and suction 770. Actions triggered during these stages can include aligning the instrument with the PCNL sheath and increasing suction. For example, if the robotic system 110 detects an increase in stone fragments or larger debris that otherwise limits visibility during laser treatment, the robotic system 110 can increase suction or suction force to remove more stone fragments. As visibility or field of view increases, the robotic system 110 can reduce suction.

[0091] Although the above describes some examples and scenarios of automated actions of the robotic system 110 that can be triggered based on the identified stages, the triggerable actions are not limited to those described above. The robotic system 110 may be programmed to perform other triggerable actions based on the needs of the user and patient.

[0092] Evaluation of tasks performed during a phase 8 is a flow diagram of an evaluation process 800 of tasks performed during the identified stages, according to certain embodiments. The evaluation process 800 may be performed by the control system 140 or by another component of the medical system 100 of FIG. 1. The following describes one possible sequence for the process, although other embodiments may perform the process in a different order or may include additional steps or may exclude one or more of the steps described below.

[0093] In block 805, the control system 140 may determine the state of the robotic manipulator (e.g., the robotic arm 112) from sensor data of the robotic system 110. As described in Figure 3, various types of sensors may be used to generate sensor data, which may then be used to determine the position or other state of the robotic manipulator.

[0094] In block 810, the control system 140 can determine an input to initiate an action of the robotic manipulator. For example, the input can be from a user operating a controller to control the basket device. In another example, the input can be a screen or menu selection on the UI of the medical system 100.

[0095] In block 815, the control system 140 may perform an analysis of the captured video. In some embodiments, such as that described in Figure 4, machine learning algorithms are used to perform the analysis and generate outputs such as identified features and tentative identification of stages.

[0096] In block 820, the control system 140 can identify a stage of the medical procedure based at least on the state of the manipulator, the identified input, and the performed analysis. For example, if the control system 140 receives a basket input via the controller, the control system 140 can determine that the stage is one of the basket processing stages. Additionally, if the performed analysis identifies that the captured video shows the basket approaching a fragmented kidney stone, the control system 140 can determine that the current stage is capturing the stone. In another example, if the performed analysis identifies that the captured video shows the basket being withdrawn from a fragmented kidney stone, the control system 140 can determine that the current stage is retracting the basket into a sheath. In a further example, kinematic data from the robotic system 110 can indicate that the medical instrument is being withdrawn from within the patient, and the control system 140 can determine that the current stage is retracting the basket into a sheath.

[0097] In block 825, the control system 140 can generate a rating of the identified stage based on one or more metrics. The stage being evaluated can vary based on the type of procedure being performed. Several possible stages are shown in blocks 830, 835, and 840. In block 830, the control system 140 evaluates the ureteroscopy laser process stage. In block 835, the control system 140 evaluates the medical instrument insertion stage. In block 840, the control system 140 evaluates the ureteroscopy basket process stage. Below are some examples of various evaluations:

[0098] 9 is a flow diagram of a scoring process 900 for a medical task, according to certain embodiments. The scoring process 900 can be performed by the control system 140 or by another component of the medical system 100 of FIG. 1. The following describes one possible sequence for the process, although other embodiments may perform the process in a different order or may include additional steps or may exclude one or more of the steps described below.

[0099] In block 905, the control system 140 counts the number of times a first treatment task is performed. In block 910, the control system 140 counts the number of times a second treatment task is performed. In block 915, the control system 140 determines a ratio of the counts for the first treatment task to the counts for the second treatment task. In block 920, the control system 140 can compare the determined ratio to a historical ratio. For example, the historical ratio may be generated by analyzing historical records of the same treatment to determine an average or median ratio.

[0100] In one example, during a ureteroscopic basket procedure, the control system 140 can count the number of basket movements and count the number of ureteroscopic retractions. The control system 140 can then determine a ratio of the number of basket movements to the number of ureteroscopic retractions and compare the determined ratio to other ratios from previous ureteroscopic basket procedures.

[0101] In one example of ureteroscopy drive, the control system 140 can count the number of times a user manually drives the scope and the number of times a user robotically drives the scope. Manual drive is commonly used to explore the kidney, while the scope is typically docked to a robotic system to perform basket manipulations. The control system 140 can then determine the ratio between the number of times a user manually drives the scope and the number of times a user robotically drives the scope and compare the determined ratio to other recorded ratios from previous ureteroscopy procedures. This ratio can measure the user's level of adaptation to robotic ureteroscopy.

[0102] In another example, during ureteroscopy laser treatment, the control system 140 can count the time the stone is lasered and determine the size and / or type of the stone. The control system 140 can then determine the ratio between the time the stone is lasered and the stone's size and compare the determined ratio to previous ratios from other operations. By determining the stone type (e.g., uric acid, calcium oxalate monohydrate, struvite, cysteine, brushite, etc.), the control system 140 can aggregate the statics of the entire surgical procedure based on the stone type. For example, the laser treatment duration and procedure duration can be divided by stone type.

[0103] At block 925, the control system 140 may generate an output of the comparison. Such output may be a report, a visual indicator, a guide, a score, a graph, etc. For example, the control system 140 may indicate that the user is performing at, below, or above the median or average ratio compared to recorded ratios from previous performances. In some embodiments, the output may track the user's personal performance by comparing the current user to records of previous performances by that user. In some embodiments, the output may compare the user to other medical professionals.

[0104] In one embodiment, the output may include a real-time indicator showing how the user's current performance compares to previous actions. Such an output may assist the user during surgery, for example, by providing user input regarding how long to run the laser treatment based on the size of the stone. Other outputs may provide other relevant information to the user.

[0105] Various types of procedural tasks can be evaluated using the scoring process 900. For example, some ratios can include the number of basket movements to the number of ureteroscope retractions, the number of times the user robotically drives the scope to the number of times the user manually drives the scope, and the time to laser the stone to the size of the stone.

[0106] Figure 10 is a flow diagram of another scoring process for a medical task, according to certain embodiments. The scoring process 1000 can be performed by the control system 140 or by another component of the medical system 100 of Figure 1. The following describes one possible sequence for the process, although other embodiments may perform the process in a different order or may include additional steps or may exclude one or more of the steps described below.

[0107] In block 1005, the control system 140 can count a first procedure task. In block 1010, the control system 140 can compare the first procedure task to a historical count of first procedures. For example, during a ureteroscopy operation, the control system 140 can count the number of times the user attempts to insert a needle until the user is successful and compare that count to recorded needle insertion attempts from previous percutaneous needle insertion operations.

[0108] In another example, during percutaneous needle insertion, control system 140 may count the time it takes to explore the kidney before selecting a target calyx for percutaneous access and compare the counted time with the recorded time from the previous percutaneous needle insertion operation. Control system 140 may also count the number of times automatic alignment of the robotic manipulator with the catheter is initiated during percutaneous needle insertion and compare the counted number with the recorded number of automatic alignments from the previous operation.

[0109] During mini-PCNL alignment, the control system 140 may count the number of times that automatic alignment of the end effector of the robotic manipulator with the catheter or sheath is initiated and compare the counted number to the recorded number of automatic alignments from previous operations. In another example, the control system 140 may count the number of times that the view of the video capture device is obstructed by dust from stone fragmentation during ureteroscopy laser processing and compare the counted number to the recorded number of dust obstructions from previous operations.

[0110] At block 1015, control system 140 may generate an output of the comparison. Such output may be a report, a visual indicator, a guide, a score, a graph, etc. For example, control system 140 may indicate that the user is performing at, below, or above the median or average compared to recorded metrics from previous performance. In some embodiments, the output may track the user's personal performance by comparing the current user to records of previous performance by that user. In some embodiments, the output may compare the user to other users.

[0111] In one embodiment, the output may include a real-time indicator showing how the user's current performance compares to previous actions. Such an output may assist the user during surgery, for example, by indicating whether the amount of dust from the fragment is out of the ordinary. Other outputs may provide other relevant information to the user.

[0112] Various types of procedural tasks may be evaluated using the scoring process 1000. For example, some tasks may include counting the number of times a user attempts to insert a needle until the user successfully inserts the needle, counting the time it takes to explore the kidney before selecting a target calyx for percutaneous access, counting the number of times a navigation field generator for tracking the needle is repositioned, counting the number of times automated alignment of the robotic manipulator with the catheter is initiated, and counting the number of times the view of the video capture device is obstructed by dust from stone fragmentation.

[0113] Exemplary Robotic System 11 illustrates exemplary details of a robotic system 110 according to one or more embodiments. In this example, the robotic system 110 is illustrated as a mobile, cart-based, robotically controllable system. However, the robotic system 110 can be implemented as a fixed system, integrated into a table, etc.

[0114] The robotic system 110 may include a support structure 114 including an elongated section 114(A) (sometimes referred to as a “column 114(A)”) and a base 114(B). The column 114(A) may include one or more carriages, such as carriage 1102 (alternatively referred to as an “arm support 1102”), for supporting the deployment of one or more robotic arms 112 (three shown in the figures). The carriage 1102 may include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robotic arm 112 for positioning relative to the patient. The carriage 1102 may also include a carriage interface 1104 that allows the carriage 1102 to translate vertically along the column 114(A). The carriage interface 1104 is connected to the column 114(A) through slots, such as slots 1106, positioned on either side of the column 114(A) to guide the vertical translation of the carriage 1102. The slot 1106 includes a vertical translation interface for positioning and holding the carriage 1102 at various vertical heights relative to the base 114(B). The vertical translation of the carriage 1102 allows the robotic system 110 to adjust the reach of the robotic arm 112 to accommodate various table heights, patient sizes, physician preferences, etc. Similarly, individually configurable arm mounts on the carriage 1102 allow the robotic arm base 1108 of the robotic arm 112 to be angled in various configurations. The column 114(A) can include mechanisms therein, such as gears and / or motors, designed to use a vertically aligned lead screw to translate the carriage 1102 in a mechanized manner in response to control signals generated in response to user input, such as input from the I / O device 116.

[0115] In some embodiments, the slot 1106 may be complemented with a slot cover that is flush with and / or parallel to the slot surface to prevent dirt and / or fluid from entering the interior chamber of the column 114(A) and / or the vertical translation interface as the carriage 1102 translates vertically. The slot cover may be deployed through a pair of spring spools positioned near the vertical top and bottom of the slot 1106. The cover may be coiled within the spools until it is deployed to extend and retract from its coiled state as the carriage 1102 translates vertically up and down. The spring load of the spools provides a force to retract the cover into the spools as the carriage 1102 translates toward the spools, while also maintaining a tight seal as the carriage 1102 translates away from the spools. The cover may be connected to the carriage 1102 using, for example, a bracket within the carriage interface 1104 to ensure proper extension and retraction of the cover as the carriage 1102 translates.

[0116] The base 114(B) can balance the weight of the column 114(A), carriage 1102, and / or arm 112 on a surface, such as a floor. Thus, the base 114(B) can house heavy components, such as one or more electronics, motors, and power supplies, as well as components that allow the robotic system 110 to be moved and / or stabilized. For example, the base 114(B) can include rollable wheels 1116 (also referred to as “casters 1116”) that allow the robotic system 110 to be moved around a room for a procedure. Once in the proper position, the casters 1116 can be locked with wheel locks to hold the robotic system 110 in place during a procedure. As shown, the robotic system 110 also includes a handle 1118 to assist in steering and / or stabilizing the robotic system 110.

[0117] The robotic arm 112 can generally include a robotic arm base 1108 and an end effector 1110 separated by a series of linkages 1112 connected by a series of joints 1114. Each joint 1114 can include an independent actuator, and each actuator can include an independently controllable motor. Each independently controllable joint 1114 represents an independent degree of freedom available to the robotic arm 112. For example, each arm 112 can have seven joints, thus providing seven degrees of freedom. However, any number of joints can be implemented with any degrees of freedom. In examples, multiple joints can provide multiple degrees of freedom, allowing for "redundant" degrees of freedom. Redundant degrees of freedom enable the robotic arm 112 to position its individual end effectors 1110 at specific positions, orientations, and / or trajectories in space using different linkage positions and / or joint angles. In some embodiments, the end effectors 1110 can be configured to engage and / or control medical instruments, devices, objects, etc. The freedom of movement of the arm 112 allows the robotic system 110 to position and / or orient medical equipment from a desired point in space and / or allows a physician to move the arm 112 to a clinically convenient position away from the patient for access while avoiding arm collisions.

[0118] 11 , the robotic system 110 can also include an I / O device 116. The I / O device 116 can include a display, a touchscreen, a touchpad, a projector, a mouse, a keyboard, a microphone, a speaker, a controller, a camera (e.g., for receiving gesture input), or another I / O device for receiving input and / or providing output. The I / O device 116 can be configured to receive touch, speech, gesture, or any other type of input. The I / O device 116 can be positioned at a vertical end of the column 114(A) (e.g., at the top of the column 114(A)) and / or can provide a user interface for receiving user input and / or providing output. For example, the I / O device 116 can include a touchscreen (e.g., a dual-purpose device) for receiving input and providing pre-operative and / or intra-operative data to a physician. Exemplary pre-operative data may include pre-operative planning, navigation, and / or mapping data derived from a pre-operative computerized tomography (CT) scan, and / or notes from a pre-operative patient interview. Exemplary intra-operative data may include tools / instruments, optical information provided by sensors, and / or coordinate information from sensors, as well as vital patient statistics such as respiration, heart rate, and / or pulse. The I / O device 116 may be positioned and / or tilted to allow a physician to access the I / O device 116 from various positions, such as on the opposite side of the column 114(A) from the carriage 1102. From this position, the physician may operate the I / O device 116 from behind the robotic system 110 while viewing the I / O device 116, the robotic arm 112, and / or the patient.

[0119] The robotic system 110 can include various other components. For example, the robotic system 110 can include one or more control electronics / circuitry, power sources, pneumatics, light sources, actuators (e.g., motors for moving the robotic arm 112), memory, and / or a communication interface (e.g., for communicating with another device). In some embodiments, the memory can store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to perform any of the operations discussed herein. For example, the memory can store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to receive input and / or control signals related to the operation of the robotic arm 112 and, in response, control the robotic arm 112 to position it in a particular configuration and / or navigate a medical instrument connected to the end effector 1110.

[0120] In some embodiments, the robotic system 110 is configured to engage and / or control a medical instrument, such as a basket retrieval device 120. For example, the robotic arm 112 may be configured to control the position, orientation, and / or tip articulation of a scope (e.g., a sheath and / or a reader of the scope). In some embodiments, the robotic arm 112 may be configured / configurable to manipulate the scope using elongated movement members. The elongated movement members may include one or more pull wires (e.g., pull wires or push wires), cables, fibers, and / or flexible shafts. Illustratively, the robotic arm 112 may be configured to actuate multiple pull wires coupled to the scope to deflect the tip of the scope. The pull wires may comprise any suitable or desirable material, such as metallic and / or non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, etc. In some embodiments, the scope is configured such that the elongated movement members exhibit nonlinear behavior upon application of force. The nonlinear behavior may be based on the stiffness and compressibility of the scope and the variability in sag or stiffness between different elongated movement members.

[0121] Exemplary Control System 12 illustrates example details of a control system 140 according to one or more embodiments. As illustrated, the control system 140 may include one or more of the following components, devices, modules, and / or units (referred to herein as “components”), either separately / individually and / or in combination / collectively: control circuitry 1202, data storage / memory 1204, one or more communication interfaces 1206, one or more power supply units 1208, one or more I / O components 1210, and / or one or more wheels 1212 (e.g., casters or other types of wheels). In some embodiments, the control system 140 may include a housing / enclosure configured and / or dimensioned to house or accommodate at least a portion of one or more of the components of the control system 140. In this example, the control system 140 is illustrated as a cart-based system that is mobile using one or more wheels 1212. In some cases, once in position, one or more wheels 1212 may be locked using wheel locks to hold control system 140 in place, however, control system 140 may be implemented as a fixed system, integrated into another system / device, etc.

[0122] While certain components of control system 140 are illustrated in FIG. 12 , it should be understood that additional components not shown may be included in embodiments consistent with the present disclosure. For example, a graphical processing unit (GPU) or other dedicated embedded chip may be included to execute neural networks. Furthermore, in some embodiments, some of the illustrated components may be omitted. While control circuitry 1202 is illustrated as a separate component in the diagram of FIG. 12 , it should be understood that any or all of the remaining components of control system 140 may be at least partially embodied in control circuitry 1202. That is, control circuitry 1202 may include various devices (active and / or passive), semiconductor materials, and / or regions, layers, areas, and / or portions thereof, conductors, leads, vias, connections, etc., and one or more of the other components of control system 140 and / or portion(s) thereof may be at least partially formed and / or embodied by such circuit components / devices.

[0123] The various components of control system 140 may be electrically and / or communicatively coupled using certain connection circuits / devices / features, which may or may not be part of control circuitry 1202. For example, connection feature(s) may include one or more printed circuit boards configured to facilitate mounting and / or interconnection of at least some of the various components / circuits of control system 140. In some embodiments, two or more of control circuitry 1202, data storage / memory 1204, communication interface 1206, power supply unit 1208, and / or input / output (I / O) components 1210 may be electrically and / or communicatively coupled to one another.

[0124] As illustrated, memory 1204 may include an input device manager 1216 and user interface components 1218 configured to facilitate various functionality discussed herein. In some embodiments, input device manager 1216 and / or user interface components 1218 may include one or more instructions executable by control circuitry 1202 to perform one or more operations. While many embodiments are discussed in the context of components 1216-1218 including one or more instructions executable by control circuitry 1202, any of components 1216-1218 may be implemented at least in part as one or more hardware logic components, such as one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more advanced standard products (ASSPs), one or more complex programmable logic devices (CPLDs), etc. Additionally, although components 1216-1218 are illustrated as being included within control system 140, any of components 1216-1218 may be implemented at least partially within another device / system, such as robotic system 110, table 150, or another device / system. Similarly, any of the other components of control system 140 may be implemented at least partially within another device / system.

[0125] The input device manager 1216 may be configured to receive inputs from the input devices 146 and translate them into actions that can be performed by the robotic system 110. For example, pre-programmed movements such as fast open, fast close, and wiggle movements may be stored in the input device manager 1216. These pre-programmed actions may be assigned to desired inputs (e.g., single or double button presses, voice commands, joystick movements, etc.). In some implementations, the pre-programmed movements are determined by the manufacturer. In other implementations, a user may be able to modify existing pre-programmed movements and / or create new movements.

[0126] The user interface component 1218 may be configured to facilitate one or more user interfaces (also referred to as "one or more graphical user interfaces (GUIs)"). For example, the user interface component 1218 may generate a configuration menu for assigning pre-programmed movements to inputs, or a settings menu for enabling particular modes of operation or disabling selected pre-programmed movements in particular situations. The user interface component 1218 may also provide user interface data 1222 for display to the user.

[0127] The one or more communication interfaces 1206 can be configured to communicate with one or more devices / sensors / systems. For example, the one or more communication interfaces 1206 can send / receive data wirelessly and / or wired over a network. Networks according to embodiments of the present disclosure can include local area networks (LANs), wide area networks (WANs) (e.g., the Internet), personal area networks (PANs), body area networks (BANs), etc. In some embodiments, the one or more communication interfaces 1206 can implement wireless technologies such as Bluetooth, Wi-Fi, near field communication (NFC), etc.

[0128] One or more power supply units 1208 may be configured to manage power for the control system 140 (and / or, in some cases, the robotic system 110). In some embodiments, the one or more power supply units 1208 include one or more batteries, such as lithium-based batteries, lead-acid batteries, alkaline batteries, and / or other types of batteries. That is, the one or more power supply units 1208 may include one or more devices and / or circuits configured to provide a power source and / or provide power management functions. Also, in some embodiments, the one or more power supply units 1208 include a mains power connector configured to couple to an alternating current (AC) or direct current (DC) mains power source.

[0129] The one or more I / O components 1210 may include various components for receiving input and / or providing output, for example, for interfacing with a user. The one or more I / O components 1210 may be configured to receive touch, speech, gestures, or any other type of input. In examples, the one or more I / O components 1210 may be used to provide input for device / system control, such as controlling the robotic system 110, navigating a scope or other medical instrument attached to the robotic system 110, controlling the table 150, controlling the fluoroscopy device 190, etc. As shown, the one or more I / O components 1210 may include one or more displays 142 (sometimes referred to as “one or more display devices 142”) configured to display data. The one or more displays 142 may include one or more liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic LED displays, plasma displays, electronic paper displays, and / or any other type of technology. In some embodiments, the one or more displays 142 may include one or more touchscreens configured to receive input and / or display data. Further, the one or more I / O components 1210 may include one or more input devices 146, which may include a touchscreen, a touchpad, a controller, a mouse, a keyboard, a wearable device (e.g., an optical head-mounted display), a virtual or augmented reality device (e.g., a head-mounted display), etc. Additionally, the one or more I / O components 1210 may include one or more speakers 1226 configured to output sound based on an audio signal and / or one or more microphones 1228 configured to receive sound and generate an audio signal. In some embodiments, the one or more I / O components 1210 include or are implemented as a console.

[0130] 9 , the control system 140 can include and / or control other components, such as one or more pumps, flow meters, valve controls, and / or fluid access components for providing controlled irrigation and / or aspiration capabilities to a medical instrument (e.g., a scope), devices that can be deployed by the medical instrument, etc. In some embodiments, the irrigation and aspiration capabilities can be delivered directly to the medical instrument via separate cables. Additionally, the control system 140 can include voltage and / or surge protectors designed to provide filtered and / or protected power to another device, such as the robotic system 110, thereby avoiding the placement of power transformers and other auxiliary power components within the robotic system 110 and making the robotic system 110 smaller and more mobile.

[0131] The control system 140 may also include support equipment for sensors deployed throughout the medical system 100. For example, the control system 140 may include optoelectronics for detecting, receiving, and / or processing data received from optical sensors and / or cameras. Such optoelectronics may be used to generate real-time images for display on any number of devices / systems included within the control system 140.

[0132] In some implementations, control system 140 may be coupled to medical instruments, such as robotic system 110, table 150, and / or scope and / or basket retrieval device 120, via one or more cables or connections (not shown). In some implementations, support functions from control system 140 may be provided via a single cable, simplifying and cluttering the operating room. In other embodiments, certain functions may be combined in separate cable lines and connections. For example, power may be provided via a single power cable, while support for control, optics, fluidics, and / or navigation may be provided via separate cables.

[0133] The term "control circuitry" is used herein according to its broad and ordinary meaning and can refer to any collection of the following: one or more processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connecting circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, graphics processing units, field programmable gate arrays, programmable logic circuits, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any device that manipulates signals (analog and / or digital) based on hard-coding of circuit and / or operating instructions. The control circuitry can further include one or more storage devices, which can be embodied in a single memory device, multiple memory devices, and / or embedded circuitry of a device. Such data storage devices can include read-only memory, random-access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information. It should be noted that in embodiments where the control circuitry includes a hardware state machine (and / or implements a software state machine) and includes analog, digital, and / or logic circuits, the data storage devices / registers that store any associated operational instructions may be embedded within or external to the circuitry that includes the state machine, analog, digital, and / or logic circuits.

[0134] The term "memory" is used herein according to its broadest and ordinary meaning and can refer to any suitable or desirable type of computer-readable medium, including, for example, one or more volatile, non-volatile, removable, and / or non-removable data storage devices implemented using any technology, layout, and / or data structure / protocol, containing any suitable or desirable computer-readable instructions, data structures, program modules, or other types of data.

[0135] Computer-readable media that can be implemented according to embodiments of the present disclosure include, but are not limited to, phase-change memory, static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transitory medium that can be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media generally may not include communication media such as modulated data signals and carrier waves. Accordingly, computer-readable media should be understood to generally refer to non-transitory media.

[0136] Further embodiments Depending on the embodiment, certain acts, events, or functions of any of the processes or algorithms described herein may occur in a different order, may be added, merged, or omitted entirely, and thus, in some embodiments, not all of the described acts or events are necessary to perform a process.

[0137] In particular, conditional language used herein, such as "can," "could," "might," "may," "eg," and the like, is intended to have its ordinary meaning unless specifically stated otherwise or understood otherwise within the context in which it is used, and is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is not intended to generally imply that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included in or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprising," "including," "having," and the like are used in their ordinary sense and are used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, when the term "or" is used, for example, to connect a list of elements, the term "or" is used in its inclusive sense (and not its exclusive sense) to mean one, some, or all of the listed elements. Unless specifically stated otherwise, connective language such as the phrase "at least one of X, Y, and Z" is understood in the context as it is commonly used to convey that an item, term, element, etc. can be either X, Y, or Z. Thus, such connective language is not generally intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0138] In the foregoing description of embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that any claim requires more features than are expressly recited in that claim. Moreover, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used in conjunction with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Accordingly, it is intended that the scope of the invention(s) disclosed herein and hereinafter claimed should not be limited by the specific embodiments described above, but should be determined solely by a fair reading of the following claims.

[0139] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements such as structures, components, operations, etc., do not necessarily indicate a priority or order of the element relative to any other elements, but rather may generally distinguish the element from other elements having a similar or identical name (apart from the use of the ordinal terminology). Furthermore, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Furthermore, an action performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.

[0140] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It is further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0141] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," "greater than," etc. are intended to encompass the concept of equality. For example, "less" can mean "less than" in the strict mathematical sense as well as "less than or equal to."

[0142] [Embodiment] (1) A robotic system for evaluating an identified stage of a medical procedure to be performed by the robotic system, the robotic system comprising: A video capture device A robot manipulator, one or more sensors configured to detect a configuration of the robotic manipulator; an input device configured to receive one or more user interactions and initiate one or more actions by the robotic manipulator; a data store configured to store metrics associated with stages of a medical procedure; a control circuit communicatively coupled to the input device and the robotic manipulator, the control circuit comprising: determining a first state of the robotic manipulator based on sensor data from the one or more sensors; identifying a first input from the input device to initiate a first action of the robotic manipulator; performing a first analysis of video of the patient site captured by the video capture device; identifying a first stage of the medical procedure based at least in part on the first state of the robotic manipulator, the first input, and the first analysis of the video; generating an assessment of the first stage of the medical procedure based on one or more metrics associated with the first stage; and A robot system comprising: (2) The robotic system of embodiment 1, wherein the first stage includes one of ureteroscopy driving, ureteroscopy laser processing, ureteroscopy basketing, and percutaneous needle insertion. (3) the first step includes ureteroscopy basket processing, and generating the evaluation includes: Counting the number of basket movements; Counting the number of ureteroscope retractions and determining a ratio of the number of basket movements to the number of ureteroscope retractions; and comparing the determined ratio with other ratios from previous ureteroscopy basket procedures. (4) the first step includes a ureteroscopy drive, and generating the assessment includes: Counting the number of times a user manually drives the scope; Counting the number of times the user robotically drives the scope; determining a ratio of the number of times the user manually drives the scope to the number of times the user robotically drives the scope; and comparing the determined ratio with other ratios from previous ureteroscopy basket procedures. (5) the first step includes percutaneous needle insertion, and generating the assessment includes: Counting the number of times the user attempts to insert the needle until the user successfully inserts the needle; and comparing the counted number with recorded needle insertion attempts from previous percutaneous needle insertion operations.

[0143] (6) the first step includes percutaneous needle insertion, and generating the assessment includes: Counting the time it takes to explore the kidney before selecting the target calyx for percutaneous access; and comparing the counted time with the recorded time from a previous percutaneous needle insertion operation. (7) the first step includes percutaneous needle insertion, and generating the assessment includes: Counting the number of times a navigation field generator for tracking the needle is repositioned; and comparing the counted number with a recorded number of repositionings from a previous percutaneous needle insertion operation. (8) the first step includes percutaneous needle insertion, and generating the assessment includes: Counting the number of times automatic alignment between the end effector of the robotic manipulator and the catheter is initiated; and comparing the counted number with a recorded number of automatic alignments from previous operations. (9) the first step includes ureteroscopy laser processing, and generating the assessment includes: Counting the time for laser treatment of stones; determining the size of the stone; comparing the ratio of the laser treatment time to the size of the stone with a previous ratio from another operation. (10) the first step includes ureteroscopy laser processing, and generating the assessment includes: determining the type of stone; aggregating statistics across surgical procedures based on the type of stone.

[0144] (11) The first step includes ureteroscopy laser processing, and generating the assessment includes: Counting the number of times the view of the video capture device is obstructed by dust from the fragmentation of the stone; 2. The robot system of claim 1, further comprising: comparing the counted number of times with a recorded number of times dust is blocked from operation. (12) A method for evaluating an identified stage of a medical procedure performed by a robotic system comprising a video capture device, a robotic manipulator, one or more sensors, and an input device, the method comprising: determining a first state of the robotic manipulator based on sensor data from the one or more sensors; identifying a first input from the input device to initiate a first action of the robotic manipulator; performing a first analysis of video of the patient site captured by the video capture device; identifying a first stage of the medical procedure based at least in part on the first state of the robotic manipulator, the first input, and the first analysis of the video; generating an assessment of the first stage of the medical procedure based on one or more metrics associated with the first stage; A method comprising: (13) The first step includes ureteroscopy basket processing, and generating the evaluation includes: Counting the number of basket movements; Counting the number of ureteroscope retractions and determining a ratio of the number of basket movements to the number of ureteroscope retractions; and comparing the determined ratio with other ratios from previous ureteroscopy basket operations. (14) The first step includes a ureteroscopy drive, and generating the evaluation includes: Counting the number of times a user manually drives the scope; Counting the number of times the user robotically drives the scope; determining a ratio of the number of times the user manually drives the scope to the number of times the user robotically drives the scope; and comparing the determined ratio with other ratios from previous ureteroscopy basket operations. (15) The first step includes percutaneous needle insertion, and generating the assessment includes: Counting the number of times the user attempts to insert the needle until the user successfully inserts the needle; and comparing the counted number with recorded needle insertion attempts from previous percutaneous needle insertion operations.

[0145] (16) The first step includes percutaneous needle insertion, and generating the assessment includes: Counting the time it takes to explore the kidney before selecting the target calyx for percutaneous access; and comparing the counted time with the recorded time since a previous percutaneous needle insertion operation. (17) The first step includes percutaneous needle insertion, and generating the assessment includes: Counting the number of times a navigation field generator for tracking the needle is repositioned; and comparing the counted number of times to a recorded number of repositionings from a previous percutaneous needle insertion operation. (18) The first step includes percutaneous needle insertion, and generating the assessment includes: Counting the number of times automatic alignment between the end effector of the robotic manipulator and the catheter is initiated; comparing the counted number of times to a recorded number of automatic alignments from a previous operation. (19) The first step includes a percutaneous antegrade ureteroscopy laser procedure, and generating the assessment includes: Counting the time for laser treatment of stones; determining the size of the stone; comparing the ratio of the laser treatment time to the size of the stone to a previous ratio from another operation. (20) The first step includes ureteroscopy laser processing, and generating the evaluation includes: determining the type of stone; aggregating statistics across surgical procedures based on the type of stone.

[0146] (21) The first step includes ureteroscopy laser processing, and generating the evaluation includes: Counting the duration that the view of the video capture device is obstructed by debris from the stone fragmentation; comparing the counted duration with a recorded duration from a previous operation. (22) A control system for a robotic device for assessing an identified stage of a medical procedure, the control system comprising: a communication interface configured to receive sensor data, user input data, and video data from the robotic device; a memory configured to store the sensor data, the user input data, and the video data; one or more processors, determining a first state of a manipulator of the robotic device based on sensor data from the one or more sensors; identifying a first input from the user input data to initiate a first action of the manipulator; performing a first analysis of video of the patient site captured by the video capture device; identifying a first stage of the medical procedure based at least in part on the first state of the manipulator, the first input, and the first analysis of the video; generating an assessment of the first stage of the medical procedure based on one or more metrics associated with the first stage; and A control system comprising:

Claims

1. 1. A robotic system for evaluating an identified stage of a medical procedure to be performed by the robotic system, the robotic system comprising: A video capture device A robot manipulator, one or more sensors configured to detect a configuration of the robotic manipulator; an input device configured to receive one or more user interactions and initiate one or more actions by the robotic manipulator; a data store configured to store metrics associated with the stage of the medical procedure; a control circuit communicatively coupled to the input device and the robotic manipulator, the control circuit comprising: determining a first state of the robotic manipulator based on sensor data from the one or more sensors; identifying a first input from the input device to initiate a first action of the robotic manipulator; performing a first analysis of video of a patient site captured by the video capture device; identifying a first stage of the medical procedure based at least in part on the first state of the robotic manipulator, the first input, and the first analysis of the video; generating an assessment of the first stage of the medical procedure based on one or more metrics associated with the first stage; and A robot system comprising:

2. 10. The robotic system of claim 1, wherein the first stage includes one of ureteroscopy actuation, ureteroscopy laser manipulation, ureteroscopy basket manipulation, and percutaneous needle insertion.

3. The first stage includes ureteroscopy basket processing, and generating the assessment includes: Counting the number of basket movements; Counting the number of ureteroscope retractions and determining a ratio of the number of basket movements to the number of ureteroscope retractions; and comparing the determined ratio to other ratios from previous ureteroscopic basket procedures.

4. The first stage includes a ureteroscopy drive, and generating the assessment includes: Counting the number of times a user manually drives the scope; Counting the number of times the user robotically drives the scope; determining a ratio of the number of times the user manually drives the scope to the number of times the user robotically drives the scope; and comparing the determined ratio to other ratios from previous ureteroscopic basket procedures.

5. The first step includes percutaneous needle insertion, and generating the assessment includes: Counting the number of times the user attempts to insert the needle until the user successfully inserts the needle; and comparing the counted number of times to recorded needle insertion attempts from previous percutaneous needle insertion operations.

6. The first step includes percutaneous needle insertion, and generating the assessment includes: Counting the time it takes to explore the kidney before selecting the target calyx for percutaneous access; and comparing the counted time to a recorded time from a previous percutaneous needle insertion operation.

7. The first step includes percutaneous needle insertion, and generating the assessment includes: Counting the number of times a navigation field generator for tracking the needle is repositioned; and comparing the counted number of times to a recorded number of repositionings from a previous percutaneous needle insertion operation.

8. The first step includes percutaneous needle insertion, and generating the assessment includes: Counting the number of times automatic alignment between the end effector of the robotic manipulator and the catheter is initiated; and comparing the counted number of times to a recorded number of automatic alignments from a previous operation.

9. The first step includes a ureteroscopy laser procedure, and generating the assessment includes: Counting the time for laser treatment of stones; determining the size of the stone; and comparing the ratio of the laser treatment time to the size of the stone to a previous ratio from another operation.

10. The first step includes a ureteroscopy laser procedure, and generating the assessment includes: Determining the type of stone; aggregating statistics across surgical procedures based on the type of stone.

11. The first step includes a ureteroscopy laser procedure, and generating the assessment includes: Counting the number of times the view of the video capture device is obstructed by dust from the fragmentation of the stone; and comparing the counted number of times to a recorded number of dust shielding from operation.

12. 1. A method of operating a robotic system comprising a video capture device, a robotic manipulator, one or more sensors, an input device, and control circuitry for evaluating an identified stage of a medical procedure, the method comprising: the control circuitry determining a first state of the robotic manipulator based on sensor data from the one or more sensors; the control circuit identifying a first input from the input device to initiate a first action of the robotic manipulator; the control circuitry performing a first analysis of video of the patient site captured by the video capture device; generating an assessment of a first stage of the medical procedure based on one or more metrics associated with the first stage; Including, The method of operating a robotic system, wherein the first stage is identified by the control circuitry based at least in part on the first state of the robotic manipulator, the first input, and the first analysis of the video.

13. The first step corresponds to a ureteroscopy basket process, and the evaluation comprises: Counting the number of basket movements; Counting the number of ureteroscope retractions and determining a ratio of the number of basket movements to the number of ureteroscope retractions; 13. The method of claim 12, wherein the determined ratio is generated based on comparing the determined ratio with other ratios from previous ureteroscopy basket operations.

14. The first stage corresponds to a ureteroscopy drive, and the evaluation comprises: Counting the number of times a user manually drives the scope; Counting the number of times the user robotically drives the scope; determining a ratio of the number of times the user manually drives the scope to the number of times the user robotically drives the scope; 13. The method of claim 12, wherein the determined ratio is generated based on comparing the determined ratio with other ratios from previous ureteroscopy basket operations.

15. The first step corresponds to percutaneous needle insertion, and the evaluation includes: Counting the number of times the user attempts to insert the needle until the user successfully inserts the needle; 13. The method of claim 12, wherein the count is generated based on comparing the count with recorded needle insertion attempts from previous percutaneous needle insertion operations.

16. The first step corresponds to percutaneous needle insertion, and the evaluation includes: Counting the time it takes to explore the kidney before selecting the target calyx for percutaneous access; 13. The method of claim 12, wherein the counted time is generated based on comparing the counted time with a recorded time from a previous percutaneous needle insertion operation.

17. The first step corresponds to percutaneous needle insertion, and the evaluation includes: Counting the number of times a navigation field generator for tracking the needle is repositioned; and comparing the counted number of times to a recorded number of repositionings from a previous percutaneous needle insertion operation.

18. The first step corresponds to percutaneous needle insertion, and the evaluation includes: Counting the number of times automatic alignment between the end effector of the robotic manipulator and the catheter is initiated; and comparing the counted number of times with a recorded number of automatic alignments from a previous operation.

19. The first step corresponds to a percutaneous antegrade ureteroscopy laser procedure, and the evaluation comprises: Counting the time for laser treatment of stones; determining the size of the stone; 13. The method of claim 12, wherein the ratio of the laser treatment time to the size of the stone is generated based on comparing the ratio to previous ratios from other operations.

20. 1. A control system for a robotic device for assessing an identified stage of a medical procedure, the control system comprising: a communication interface configured to receive sensor data, user input data, and video data from the robotic device; a memory configured to store the sensor data, the user input data, and the video data; one or more processors, determining a first state of a manipulator of the robotic device based on sensor data from one or more sensors; identifying a first input from the user input data to initiate a first action of the manipulator; performing a first analysis of a video of the patient site captured by a video capture device; identifying a first stage of the medical procedure based at least in part on the first state of the manipulator, the first input, and the first analysis of the video; generating an assessment of the first stage of the medical procedure based on one or more metrics associated with the first stage; and A control system comprising:

Images