AI-assisted workflow segmentation

Abstract

A robotic system configured to automatically identify a stage of a medical procedure includes a video capture device, a robotic manipulator, one or more sensors, an input device, and control circuitry. The control circuitry is configured to: determine a first position of the robotic manipulator based on sensor data; determine a first procedure from a set of procedures based on user input and the sensor data; narrow the set of procedure steps to a subset of procedure steps based on the determined first procedure; perform a first analysis of the captured video of the patient site; identify a first stage of the medical procedure from the subset of procedure steps based on the first position of the robotic manipulator and the first analysis of the video; and generate a first video marker indicating the start of the first stage of the medical procedure.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 116,798, filed on November 20, 2020, and U.S. Patent Application No. 63 / 132,850, filed on December 31, 2020, and each of these patent documents is hereby incorporated by reference in its entirety.

[0002] (Field of the Invention) The present disclosure relates to the fields of medical devices and procedures, and artificial intelligence - assisted data processing.

Background Art

[0003] Various medical procedures involve the use of robotic systems that assist in the use of one or more medical instruments configured to reach a treatment site through the human anatomical structure. A particular surgical process may involve inserting one or more medical instruments through the patient's skin and openings to reach a treatment site and removing an object such as a urinary calculus from the patient.

Summary of the Invention

Means for Solving the Problems

[0004] Described herein are one or more systems, devices, and / or methods for assisting a physician or medical professional in controlling a medical instrument for accessing an object such as a urinary calculus located within the anatomical structure of the human body.

[0005] For the purpose of summarizing the present disclosure, certain aspects, advantages, and novel features are described. It should be understood that not all such advantages may necessarily be achieved by any particular embodiment. Thus, the disclosed embodiments may be implemented in a manner that achieves or optimizes the realization of one advantage or group of advantages taught herein without necessarily achieving any other advantages that may be taught or suggested herein.

[0006] One or more computer systems can be configured to perform a specific operation or action by having software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform an action while it is running. One or more computer programs can be configured to perform a specific operation or action by including instructions that cause the device to perform an action when executed by a data processing device. One common embodiment includes a robotic system for automatically identifying stages of medical procedures. The robotic system also includes a video capture device, a robotic manipulator, one or more sensors for determining the configuration of the robotic manipulator, an input device configured to receive one or more user interactions and to initiate one or more actions by the robotic manipulator, and a control circuit communicatively coupled to the input device and the robotic manipulator. The control circuit is configured to determine a first position of the robotic manipulator based on sensor data from one or more sensors; determine a first procedure from a set of procedures based on user input and at least one of the sensor data from one or more sensors; narrow the set of procedure stages to a subset of procedure stages based on the determination of the first procedure; perform a first analysis of a video of a patient site captured by a video capture device; identify a first stage of a medical procedure from the subset of procedure stages based at least in part on the first position of the robotic manipulator and the first analysis of the video; and generate a first video marker indicating the start of the first stage of a medical procedure. Other embodiments of this aspect include a corresponding computer system, apparatus, and computer program recorded in one or more computer storage devices, each configured to perform the operation of the method.

[0007] The implementation may include one or more of the following features: The set of procedures performed by the robotic system may include at least ureteroscopy, percutaneous nephrolithotomy (PCNL), and mini-PCNL. Sensor data for determining a first procedure from one or more sensors may include radio frequency identification (RFID) data of one or more medical devices used by the robotic manipulator. The first procedure is determined based on user input, which may include at least the selection of a first UI screen associated with the first procedure. The control circuit of the robotic system may be further configured to determine a second position of the robotic manipulator, perform a second analysis of video captured by a video capture device, and generate a second video marker for the video indicating the completion of a first stage of a medical procedure, based at least in part on the second position of the robotic manipulator and the second analysis of the video. The robotic manipulator is configured to operate medical devices, which may include a ureteroscope. The first stage may include a laser processing stage, the second stage may include a basket processing stage, and the third stage may include a percutaneous access stage, with the second and third stages indicated by additional video markers on the video. The control circuit of the robotic system may be further configured to trigger an automated movement of the robotic manipulator to a second position in response to identifying the start of the first stage. The automated movement of the robotic manipulator to the second position may include moving the medical instrument to the patient's insertion site and aligning the medical instrument along a predetermined insertion trajectory to the insertion site. The control circuit of the robotic system may be further configured to determine whether the medical instrument is in a target location in response to the movement of the medical instrument by the robotic manipulator, and to indicate the success of the first stage on the user interface of the robotic system in response to the medical instrument reaching the target location. The control circuit of the robotic system may be further configured to aggregate data on the success and failure of the first stage across multiple medical procedures and to determine the success rate of the first stage.The control circuit of the robotic system may be further configured to associate the results of the first stage with a medical professional operating the robotic system. The robotic system may include sensors for determining the location of a medical device being operated by the robotic manipulator. Embodiments of the described technology may include mechanical equipment, methods or processes, or computer software on a computer-accessible medium.

[0008] One general embodiment includes a method for automatically identifying stages of a medical procedure using a robotic system which may include a video capture device. The method also includes determining a first position of a robotic manipulator based on sensor data from one or more sensors; determining a first procedure from a set of procedures based on user input and at least one of the sensor data from one or more sensors; narrowing the set of procedure stages to a subset of procedure stages based on the determination of the first procedure; performing a first analysis of a video of a patient site captured by the video capture device; identifying a first stage of a medical procedure from the subset of procedure stages based at least in part on the first position of the robotic manipulator and the first analysis of the video; and generating a first video marker indicating the start of the first stage of a medical procedure. Other embodiments of this embodiment include a corresponding computer system, apparatus, and computer program recorded in one or more computer storage devices, each configured to perform the operation of the method.

[0009] An implementation may include one or more of the following features. The method may include determining a second position of a robotic manipulator, performing a second analysis of a video, and generating a second video marker for the video indicating the completion of a first stage of a medical procedure, based at least in part on the second position of the robotic manipulator and the second analysis of the video. In some implementations, the first procedure is a ureteroscopy, the first stage may include an investigation stage, the second stage may include a laser processing stage, and the third stage may include a basket processing stage. The method may include generating markers indicating the second and third stages, based at least in part on the position of the robotic manipulator, the video analysis, and user input from an input device. The method may include triggering an automated movement of the robotic manipulator to a second position in response to identifying the start of a first stage. The automated movement of the robotic manipulator to a second position may include moving a medical instrument to the patient's insertion site and aligning the medical instrument along a predetermined insertion. This method may include determining whether a medical device is at a target location in response to the movement of the medical device by a robotic manipulator, and indicating the success of the first stage on the user interface of the robotic system in response to the medical device reaching the target location. This method may also include aggregating data on the success and failure of the first stage across multiple medical procedures, and determining the success rate of the first stage. Implementations of the described technology may include mechanical equipment, methods or processes, or computer software on a computer-accessible medium.

[0010] One general embodiment includes a control system for automatically identifying stages of medical procedures performed by a robotic device. The control system also includes a communication interface configured to receive sensor data, user input data, and video data from the robotic device; a memory configured to store the sensor data, user input data, and video data; and one or more processors configured to: determine a first position of the robotic device from sensor data; determine a first procedure from a set of procedures based on user input and at least one of the sensor data from one or more sensors; narrow down a set of identifiable procedure stages to a subset of procedure stages based on the determination of the first procedure; perform a first analysis of a video of a patient body part captured by a video capture device; identify a first stage of a medical procedure from the subset of procedure stages based at least in part on the first position of the robotic device and the first analysis of the video; and generate a first video marker indicating the start of a first stage of a medical procedure. Other embodiments of this embodiment include a corresponding computer system, apparatus, and computer program recorded in one or more computer storage devices, each configured to perform the operation of the method. [Brief explanation of the drawing]

[0011] Various embodiments are shown in the accompanying drawings for illustrative purposes, but should not be construed as limiting the scope of this disclosure. In addition, various features of different disclosed embodiments can be combined to form further embodiments that are part of this disclosure. Throughout the drawings, reference numbers may be reused to indicate correspondences between reference elements. [Figure 1] This figure illustrates an example of a medical system for performing or supporting medical procedures according to a specific embodiment. [Figure 2A] This is a perspective view of the medical system during a urinary tract stone removal procedure, according to a specific embodiment. [Figure 2B]This is a perspective view of the medical system during a urinary tract stone removal procedure, according to a specific embodiment. [Figure 3] This is a block diagram of a control system for a medical system, with associated inputs and outputs, according to a specific embodiment. [Figure 4A] This is a block diagram of a control system configured to generate output from video data using machine learning, according to a specific embodiment. [Figure 4B] This is a block diagram of a control system configured to generate output from several types of data using machine learning, according to a specific embodiment. [Figure 5] This is a flowchart of a step-by-step identification process according to a specific embodiment. [Figure 6] This is a flowchart of the trigger process for automated robot action according to a specific embodiment. [Figure 7] This figure shows different types of triggered actions of a robot system according to a specific embodiment. [Figure 8] This is a flowchart of the evaluation process for tasks performed during identified stages, according to a specific embodiment. [Figure 9] This is a flowchart of a scoring process for medical tasks according to a specific embodiment. [Figure 10] This is a flowchart of another scoring process for a medical task, according to a specific embodiment. [Figure 11] This figure shows exemplary details of a robotic system according to a specific embodiment. [Figure 12] This figure shows exemplary details of a control system according to a specific embodiment. [Modes for carrying out the invention]

[0012] The direction of proceedings provided herein is for convenience only and does not necessarily affect the scope or meaning of the disclosure. Certain preferred embodiments and examples are disclosed below, but the subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as their modifications and equivalents. Therefore, the claims that may arise from this specification are not limited to any of the specific embodiments described below. For example, in any method or process disclosed herein, the actions or operations of the method or process may be performed in any preferred order and are not necessarily limited to any particular disclosed order. Various operations may be described sequentially as a number of separate operations in a manner that may be helpful in understanding a particular embodiment. However, the order of description should not be construed as meaning that these operations are order-dependent. Furthermore, the structures, systems, and / or devices described herein may be embodied as integrated components or separate components. For the purpose of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not all such aspects or advantages are necessarily realized by any particular embodiment. Therefore, for example, various embodiments can be implemented in a manner that realizes or optimizes one or more advantages or groups of advantages as taught herein, without necessarily realizing other embodiments or advantages that may also be taught or suggested herein.

[0013] The specific standard anatomical term "location" may be used herein to refer to an animal, i.e., human, anatomical structure in relation to a preferred embodiment. Certain spatially relative terms, such as "lateral," "medial," "super," "subordinate," "downward," "upper," "vertical," "horizontal," "apex," "base," and similar terms, are used herein to describe the spatial relationship of one device / element or anatomical structure to another; however, these terms are used herein to simplify the description of the positional relationships between elements / structures, as illustrated in the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of elements / structures during use or operation, in addition to the orientations shown in the drawings. For example, when an element / structure is described as being "above" another element / structure, it may mean a position below or beside such other element / structure with respect to the patient or an alternative orientation of the element / structure, and vice versa.

[0014] Overview This disclosure relates to technologies and systems for collecting and analyzing data from robot-assisted medical procedures, such as those performed by a robotic system for stone management (e.g., retrieval of urinary tract stones, aspiration of stone fragments, etc.) or other medical procedures. A medical procedure may proceed through several different stages. For example, in a ureteroscopy, the stages may include percutaneous insertion of a medical instrument into the body, movement to the location of the urinary tract stone, laser treatment of the urinary tract stone, and / or basket treatment of the fragmented stone. The robotic system typically has several sensors and input devices, enabling the generation of a large amount of data during the medical procedure. This procedure data can be used to automatically determine the different stages of the procedure. By identifying these stages, the robotic system can predict and prepare the actions of the medical professional operating the robotic system during the medical procedure.

[0015] A medical system that includes a robotic system may also be able to annotate video footage of a procedure using metadata that identifies different stages. This enables the video footage to be reviewed more easily by the user and allows for more advanced analysis of the video footage using artificial intelligence (AI). This can make it easier to evaluate and score actions by comparing the actions performed by the user or operator to similar actions from corresponding stages performed during other procedures. For example, the video footage and associated data can be analyzed by an AI system to generate statistics related to the actions, such as the number of attempts before success per stage or per entire procedure, the time for each stage, the number of joint movement commands provided by the operator, the accuracy of needle insertion, etc. Additionally, the data can be aggregated over several actions and used to generate statistics related to the type of action in general, such as the success rate, average action time per stage or per entire procedure. Such a medical system can also provide additional benefits, such as by generating a case summary.

[0016] In one exemplary scenario, there are distinct stages during percutaneous kidney access or other procedures. In an exemplary workflow, the user drives the scope to the desired renal calyx, marks the papilla, withdraws the scope to view the target papilla. Then, the user holds the needle, selects the insertion site, and aligns the needle trajectory with the target papilla using a graphical user interface (“GUI”). Finally, the user inserts the needle according to the graphical user interface to gain access to the kidney through the target papilla. To improve procedure efficiency and assess user skill, the medical system can label the start and end of these events and obtain ground truth data regarding whether a percutaneous access (“perc”) attempt was successful.

[0017] Case data can be split into separate stages, and after generating a state transition chart showing these stages, the transition chart can be used to evaluate the treatment. For example, one exemplary transition chart may show that the physician selected a target and insertion site but did not proceed in the needle alignment step and instead moved to a different renal calyx to select a new target. The chart may show that the physician did not obtain a visual confirmation of access in the first percutaneous access attempt and drove the scope to position the needle. The chart may show that the physician made another percutaneous access attempt using the same target and this time obtained a visual confirmation. Such a chart can be displayed on the GUI of the medical system, as a digital or printed report, on a mobile application, and / or as an output of a similar type.

[0018] Another potential benefit is the provision of ground truth (success / failure) annotations. Stage segmentation allows for prediction of whether a particular percutaneous access attempt was successful or not, thereby serving as a ground truth for the case. The medical system can track a set of feature descriptors during the needle insertion stage to determine whether the percutaneous access was successful or not. Feature descriptors can include various quantities or metrics measured by the medical system, such as needle and scope velocity, as well as the relative orientation of the needle to the scope. It can also include scope joint movement commands and features detected by a computer vision algorithm, which detects whether the needle is visible in the camera view and quantifies how much anatomical movement is present. For example, a direct correlation may exist between visual confirmation and success. In one scenario, if a computer vision algorithm detects the needle in the endoscopic view, the percutaneous access attempt can be annotated as successful or otherwise indicated. In another scenario, the distance between the needle and the scope may be very small, but there is no visual confirmation of the needle in the scope. If the scope begins to move, it implies that the percutaneous access attempt has failed and the user is searching for a needle or driving to another renal calyx to select a new target. Therefore, detection of scope movement in that situation can be used to annotate or otherwise indicate that the percutaneous access attempt has failed.

[0019] Another potential benefit is providing skill assessment. Stage segmentation can enable stage-specific data analysis to assess physician skills and calculate intraoperative or postoperative case statistics. The table below shows postoperative metrics for some percutaneous access stages. For example, by knowing when needle insertion begins (identified, e.g., via video capture, sensor data, etc.), the medical system can determine the entry point on the skin (e.g., using kinematic data, video analysis, etc.) and calculate site selection metrics such as tract length (e.g., distance from skin to nipple).

[0020] [Table 1]

[0021] For example, during the scope-driven phase, the user's skill can be evaluated based on the number of joint movement commands received by the system. Fewer commands received indicate that the movement was performed smoothly and demonstrate higher skill. More commands received indicate that multiple attempts must be made and suggest room for improvement. These metrics can also provide information about parts of anatomical structures that the user is having difficulty navigating. The number of joint movement commands may be recorded and / or displayed for this surgery or for multiple surgeries (all cases, all cases over a period of time, all cases performed by the user, etc.). For example, a medical system can generate metrics that compare a given case, the entire physician, and / or the same physician by location over time across multiple surgeries.

[0022] In another embodiment, the user's skill can be evaluated during the needle insertion phase based on the success rate and / or needle insertion accuracy. The success rate can be calculated more specifically based on the location of the kidney, such as the lower, middle, or upper pole. Needle insertion accuracy can be compared to the average value of a professional. Needle insertion accuracy may be recorded and / or displayed for this surgery or a series of surgeries (e.g., all cases, all cases over a certain period, all cases performed by the user, etc.).

[0023] In a further example, during the site selection phase, user skill may be evaluated based on site selection time or the time it took the user to select the site and average tract length. Site selection time can be compared to the average value of a professional. Site selection time may be recorded and / or displayed for this surgery or multiple surgeries (all cases, all cases over a period of time, all cases performed by the user, etc.). Average tract length can be calculated more specifically based on the location of the kidney, such as the inferior, middle, or superior pole. Patient tract length may be used as an indicator of the patient's body mass index (BMI). This allows case outcomes to be aggregated based on patient population characteristics such as BMI values ​​or ranges.

[0024] The table above shows only a few examples of possible metrics that can be evaluated. Furthermore, the table above shows only some of the specificities that may apply to those metrics. For example, some specificities that apply to one metric may apply to another. In some embodiments, needle insertion accuracy can be further classified based on kidney location. Success rates can be shown with higher specificity by comparing with the average of experts or across multiple surgeries (e.g., all cases, all cases over a certain period, all cases performed by the user, etc.).

[0025] Another potential benefit of such a healthcare system is that it provides optimization of skill assessment workflows. Workflow analysis can reveal correlations between the sequence of workflow steps and the success and efficiency of percutaneous access. For example, an algorithm can compare cases where site selection is performed before target selection versus when target selection is performed before site selection to assess the impact on percutaneous access time and accuracy.

[0026] Such medical systems can be used in several types of procedures, including ureteroscopy. Nephrolithiasis, also known as urolithiasis, is a relatively common medical condition involving the formation of solid fragments of material in the urinary tract, referred to as “nephroliths,” “urinary tract stones,” “kidney stones,” “kidney stones,” or “nephrolithiasis.” Urinary tract stones can form and / or be found in the kidneys, ureters, and bladder (referred to as “bladder stones”). Such urinary tract stones result from the concentration of minerals and can cause significant abdominal pain when they reach a size large enough to obstruct the flow of urine through the ureters or urethra. Urinary tract stones can form from calcium, magnesium, ammonia, uric acid, cysteine, or other compounds.

[0027] To remove urinary tract stones from the bladder and ureters, a surgeon may insert a ureteroscope into the urinary tract through the urethra. Typically, the ureteroscope includes an endoscope at its distal end configured to allow visualization of the urinary tract. The ureteroscope may also include a stone extraction mechanism, such as a basket retrieval device for capturing or fragmenting urinary tract stones. During a ureteroscopy, one physician / technician may control the position of the ureteroscope, while another physician / technician may control the stone extraction mechanism.

[0028] In many embodiments, the techniques and systems are discussed in relation to minimally invasive procedures. However, it should be understood that the techniques and systems can be implemented in relation to any medical procedure, including percutaneous surgery, non-invasive procedures, therapeutic procedures, diagnostic procedures, non-percutaneous procedures, or other types of procedures, for example, in which access to a target site is obtained by making a puncture and / or small incision in the body to insert a medical instrument. For example, such techniques can be used in tumor biopsies or ablations for urological and bronchoscopy, where an automated biopsy action can be triggered when the system detects proximity to a suspected site. Endoscopic procedures may include bronchoscopy, ureteroscopy, gastroscopy, pelvic ureteroscopy, and nephrolithotomy. Furthermore, in many embodiments, the techniques and systems are described as being performed as robot-assisted procedures. However, it should be understood that the techniques and systems can also be performed in other procedures, such as fully robotic medical procedures.

[0029] For the sake of illustration and discussion, the technique and system will be discussed in the context of removing urinary tract stones, such as kidney stones, from the kidneys. However, as mentioned above, this technique and system can be used to perform other procedures.

[0030] Healthcare system Figure 1 illustrates an example of a medical system 100 for performing or assisting in medical procedures according to one or more embodiments. Embodiments of the medical system 100 may be used for surgical and / or diagnostic procedures. The medical system 100 includes a robotic system 110 configured to engage with and / or control a medical instrument 120 and to perform the procedure on a patient 130. The medical system 100 also includes a control system 140 configured to interface with the robotic system 110, provide information about the procedure, and / or perform various other actions. For example, the control system 140 may include a display 142 that presents a user interface 144 to assist a physician 160 in using the medical instrument 120. Furthermore, the medical system 100 may include a table 150 configured to hold the patient 130, and / or imaging sensors 180 such as a camera, X-ray, computed tomography (CT), magnetic resonance imaging (MRI), or positron emission tomography (PET) device.

[0031] In some embodiments, a physician performs a minimally invasive medical procedure, such as ureteroscopy. The physician 160 interacts with a control system 140 to control a robotic system 110, which can navigate a medical instrument 120 (e.g., a basket retrieval device and / or scope) from the urethra into the kidney 170 where the stone 165 is located. The control system 140 can provide information about the medical instrument 120 via a display 142, such as real-time images from the medical instrument 120 or an imaging sensor 180, to assist the physician 160 in navigation. Once the site of the kidney stone is reached, the medical instrument 120 can be used to break up and / or capture the urinary tract stone 165.

[0032] In some implementations using the medical system 100, a physician 160 can perform percutaneous procedures. For example, if a patient 130 has a kidney stone 165 in their kidney 170 that is too large to be removed through the urinary tract, the physician 160 can perform a procedure to remove the kidney stone through a percutaneous access point on the patient 130. For instance, the physician 160 can interact with a control system 140 to control a robotic system 110 and navigate a medical instrument 120 (e.g., a scope) from the urethra into the kidney 170 where the stone 165 is located. The control system 140 can provide information about the medical instrument 120 via a display 142, such as real-time images from the medical instrument 120 or an imaging sensor 180, to assist the physician 160 in navigating the medical instrument 120. Once the site of the kidney stone is reached, the medical instrument 120 can be used to specify a target location (e.g., a desired point for accessing the kidney) for a second medical instrument (not shown) to access the kidney percutaneously. To minimize damage to the kidney, physician 160 may designate a specific papilla as a target site for entry into the kidney with a second medical device. However, other target sites may be designated or determined. Once the second medical device reaches the target site, physician 160 may use the second medical device and / or another medical device to remove the kidney stone from patient 130 through a percutaneous access point, etc. Although the above percutaneous procedure is considered in the context of using medical device 120, in some implementations, the percutaneous procedure may be performed without the assistance of medical device 120. Furthermore, the medical system 100 may be used to perform various other procedures.

[0033] Minimally invasive surgery offers the possibility of video recording of the surgery, as a camera (e.g., a scope of medical instrument 120) can be inserted into the body during the procedure. Additional cameras and sensors placed outside the body can be used to capture video and / or data of the patient and medical system 100. For example, operating room (OR) cameras(or more) can capture video of activities in the operating room, such as the movements of the operator or physician's hands, the location of needles, the change of fluid bags, and patient bleeding. Details such as the number of contrast agent injections during fluoroscopy may also be captured by the OR camera and used to estimate the amount of radiation exposure to the patient. Audio recorded in the video can also be used to help identify stages. For example, some robotic systems emit a beep or other audible noise when laser processing is occurring. The video can be archived and used later for reasons such as cognitive training, skill assessment, and workflow analysis.

[0034] Computer vision, a form of artificial intelligence (AI), enables the quantitative analysis of video by computers for object and pattern identification. For example, in endoscopic surgery, AI video systems can be used for gesture / task classification, skill assessment, tool type recognition, shot / event detection, and retrieval. The AI ​​system can view video of surgical procedures and track the movement and timing of instruments used during the procedure. The AI ​​system can track tool timing, such as which instruments were used, when, and for how long, using metrics. In addition, the AI ​​system can track the trajectory of instruments, which can be useful for evaluating the procedure or identifying stages in the procedure. The AI ​​system can determine how far a tool is from the surgical field, which may correlate with the quality of surgery, as better surgeons tend to handle instruments in focused areas. The AI ​​system can also determine metrics to evaluate multiple aspects of the medical professional's performance, including the economics of the medical professional's movements, how often the medical professional switched between instruments, and the medical professional's efficiency at each step of the procedure.

[0035] In the example in Figure 1, the medical device 120 is implemented as a basket retrieval device. Therefore, for ease of explanation, the medical device 120 is also referred to as the “basket retrieval device 120”. However, the medical device 120 can be implemented as various types of medical devices, including, for example, a scope (sometimes called an “endoscope”), a needle, a catheter, a guidewire, a lithotripter, forceps, a vacuum, a scalpel, or a combination of the above. In some embodiments, the medical device is a controllable device, and in other embodiments, the medical device is a non-controllable device. In some embodiments, surgical tools refer to devices configured to puncture or be inserted through human body structures, such as needles, surgical scalpels, and guidewires. However, surgical tools can also refer to other types of medical devices. In other embodiments, multiple medical devices may be used. For example, an endoscope can be used with the basket retrieval device 120. In some embodiments, the medical device 120 may be a composite device incorporating several instruments, such as a vacuum, a basket retrieval device, a scope, or various combinations of instruments.

[0036] In some embodiments, the medical device 120 may include a radio frequency identification (RFID) chip for identifying the medical device 120. The medical system 100 may include an RFID reader for reading the RFID chip within the medical device to help identify the device. Such information can be used to facilitate the identification of procedures and stages. For example, if the RFID data identifies the device as a needle, the stage may relate to needle insertion, but determining the exact stage may require combining the RFID data with additional data such as video, device status, and telemetry (e.g., magnetic tracking, robotic data, fluid dynamics data, etc.).

[0037] The robotic system 110 can be configured to facilitate medical procedures. The robotic system 110 can be positioned in various ways depending on the specific procedure. The robotic system 110 may include one or more robotic arms 112 (robotic arms 112(a), 112(b), 112(c)) configured to engage with and / or control a medical instrument 120 and perform the procedure. As shown, each robotic arm 112 may include multiple jointed arm segments, thereby providing multiple degrees of mobility. In the embodiment of Figure 1, the robotic system 110 is positioned close to the lower torso of the patient 130, and the robotic arms 112 are actuated to engage with and position the medical instrument 120 for access to an access point such as the urethra of the patient 130. Once the robotic system 110 is properly positioned, the medical instrument 120 can be inserted into the patient 130 using the robotic arms 112 under robotic control, manually by the physician 160, or a combination thereof.

[0038] The robot system 110 may also include a base 114 coupled to one or more robot arms 112. The base 114 may include various subsystems such as control electronics, power supply, pneumatics, light source, actuators (e.g., motors for moving the robot arms), control circuits, memory, and / or communication interfaces. In some embodiments, the base 114 includes an input / output (I / O) device 116 configured to receive inputs such as user input for controlling the robot system 110 and to provide outputs such as patient status, location of medical equipment, etc. The I / O device 116 may include a controller, mouse, keyboard, microphone, touchpad, other input devices, or a combination thereof. The I / O device may include output components such as a speaker, display, haptic feedback device, other output devices, or a combination thereof. In some embodiments, the robot system 110 is movable (e.g., the base 114 includes wheels), and as a result, the robot system 110 may be positioned in a location appropriate or desirable for a procedure. In other embodiments, the robot system 110 is a fixed system. Furthermore, in some embodiments, the robot system 110 is integrated with the table 150.

[0039] The robot system 110 can be coupled to any component of the medical system 100, such as the control system 140, the table 150, the imaging sensor 180, and / or the medical instrument 120. In some embodiments, the robot system is communicatively coupled to the control system 140. In one embodiment, the robot system 110 can receive control signals from the control system 140 to perform actions such as positioning the robot arm 112 in a specific manner or operating a scope. In response, the robot system 110 can control its components and perform actions. In another embodiment, the robot system 110 can receive images from the scope depicting the internal anatomical structures of the patient 130 and / or transmit the images to the control system 140 (the images may then be displayed on the control system 140). Furthermore, in some embodiments, the robot system 110 is coupled to components of the medical system 100, such as the control system 140, to receive data signals, power, etc. Other medical devices, such as IV bags and blood packs, can also be coupled to other components of the robotic system 110 or the medical system 100, depending on the medical procedure being performed.

[0040] The control system 140 can be configured to provide various functions to assist in the performance of medical procedures. In some embodiments, the control system 140 can be coupled to and operate in cooperation with the robot system 110 to perform medical procedures on the patient 130. For example, the control system 140 can communicate with the robot system 110 via wireless or wired connections (for example, to control the robot system 110, a basket retrieval device 120, or to receive images captured by a scope, etc.), control the flow of fluid through the robot system 110 via one or more fluid channels, provide power to the robot system 110 via one or more electrical connections, or provide optical signals to the robot system 110 via one or more optical fibers or other components, etc. Furthermore, in some embodiments, the control system 140 can communicate with a scope for receiving sensor data. Furthermore, in some embodiments, the control system 140 can communicate with a table 150 to position the table 150 in a specific orientation or otherwise control the table 150.

[0041] As shown in Figure 1, the control system 140 includes various I / O devices configured to assist a physician 160 or another person in performing a medical procedure. In some embodiments, the control system 140 includes an input device 146 employed by the physician 160 or another user to control the basket retrieval device 120. For example, the input device 146 can be used to navigate the basket retrieval device 120 within the patient 130. The physician 160 can provide input via the input device 146, and in response, the control system 140 can send control signals to the robotic system 110 to operate the medical instrument 120.

[0042] In some embodiments, the input device 146 is a controller similar to a game controller. The controller may have multiple axes and buttons that can be used to control the robotic system 110. Although the input device 146 is shown as a controller in the embodiment of Figure 1, the input device 146 can be implemented as various types of I / O devices or combinations of various types of I / O devices, such as a touchscreen / pad, mouse, keyboard, microphone, or smart speaker. Also, as shown in Figure 1, the control system 140 may include a display 142 that provides various information about the procedure. For example, the control system 140 may receive real-time images captured by a scope and display the real-time images via the display 142. In addition, or alternatively, the control system 140 may receive signals (e.g., analog, digital, electrical, acoustic / sonic, pneumatic, tactile, hydraulic, etc.) from a medical monitor and / or sensors associated with the patient 130, and the display 142 may present information about the patient's health and / or the patient's environment. Such information may include, for example, information displayed via a medical monitor, such as heart rate (e.g., electrocardiogram (ECG), heart rate variability (HRV)), blood pressure / blood flow velocity, muscle biosignals (e.g., electromyography (EMG)), body temperature, oxygen saturation (e.g., SpO2), carbon dioxide (CO2), electroencephalogram (EEG), and ambient temperature.

[0043] Figure 1 also shows various anatomical structures of patient 130 relating to a particular aspect of the present disclosure. Specifically, patient 130 includes a kidney 170 fluidly connected to the bladder 171 via a ureter 172, and a urethra 173 fluidly connected to the bladder 171. As shown in the enlarged depiction of the kidney 170, the kidney includes renal calyces 174 (e.g., major and minor calyces), renal papillae (including renal papillae 176, also referred to as “papillae 176”), and renal pyramids (including renal pyramids 178). In these embodiments, the kidney stone 165 is located in close proximity to the papillae 176. However, kidney stones may be located elsewhere within the kidney 170.

[0044] As shown in Figure 1, in an exemplary minimally invasive procedure to remove a kidney stone 165, a physician 160 may position a robotic system 110 at the foot of a table 150 to initiate delivery of a medical instrument 120 to the patient 130. Specifically, the robotic system 110 may be positioned close to the lower abdomen of the patient 130 and aligned for direct, linear access to the patient's urethra 173. From the foot of the table 150, a robotic arm 112(B) may be controlled to provide access to the urethra 173. In this embodiment, the physician 160 inserts the medical instrument 120 at least partially into the urethra along this direct, linear access path (sometimes referred to as a “virtual rail”). The medical instrument 120 may include a lumen configured to receive a scope and / or basket retrieval device, thereby assisting in the insertion of these devices into the anatomical structure of the patient 130.

[0045] Once the robotic system 110 is properly positioned and / or the medical instrument 120 is at least partially inserted into the urethra 173, the scope can be inserted into the patient 130 by robotic control, manually, or a combination thereof. For example, a physician 160 can connect the medical instrument 120 to a robotic arm 112(C). The physician 160 can then interact with a control system 140, such as an input device 146, to navigate the medical instrument 120 within the patient 130. For example, the physician 160 can control the robotic arm 112(C) by providing input via the input device 146 to navigate the basket retrieval device 120 through the urethra 173, bladder 171, ureter 172 to the kidney 170.

[0046] The control system 140 may include various components (sometimes called “subsystems”) to facilitate its function. For example, the control system 140 may include various subsystems such as control electronics, power supply, pneumatics, light source, actuator, control circuit, memory, and / or communication interface. In some embodiments, the control system 140 includes a computer-based control system that stores executable instructions that, when executed, cause various actions to be performed. In some embodiments, the control system 140 is mobile, as shown in Figure 1, but in other embodiments, the control system 140 is a fixed system. Various functions and components have been discussed as being implemented by the control system 140, but any of these functions and / or components may be integrated into and / or implemented by other systems and / or devices, such as the robot system 110 and / or the table 150.

[0047] The medical system 100 can provide various benefits, such as providing guidance to assist physicians in performing procedures (e.g., instrument tracking, patient status), enabling physicians to perform procedures from ergonomic positions without requiring awkward arm movements and / or positions, enabling one physician to perform procedures using one or more medical instruments, avoiding radiation exposure (e.g., associated with fluoroscopy techniques), enabling procedures to be performed in a single surgical setting, and providing continuous suction for more efficient removal of objects (e.g., removal of kidney stones). Furthermore, the medical system 100 can provide non-radiation-based navigation and / or localization techniques to reduce physicians' radiation exposure and / or reduce the amount of equipment in the operating room. In addition, the medical system 100 can be divided into a control system 140 and a robotic system 110, each of which may be independently mobile. Such functional and / or mobile divisions can allow the control system 140 and / or robotic system 110 to be positioned in locations that are optimal for a particular medical procedure, thereby maximizing the work area around the patient and / or providing an optimized location for the physician to perform the procedure. For example, many aspects of a procedure may be performed by the robotic system 110 (positioned relatively close to the patient), while the physician manages the procedure from a distance, away from the control system 140 (which may be positioned further away).

[0048] In some embodiments, the control system 140 can function even if it is located in a different geographical location from the robot system 110. For example, in the implementation of telemedicine, the control system 140 is configured to communicate with the robot system 110 via a wide-area network. In one scenario, a physician 160 may be located in one hospital with the control system 140, while the robot system 110 is located in a different hospital. The physician can then perform medical procedures remotely. This can be beneficial when remote hospitals, such as rural hospitals, have limited expertise in certain procedures. These hospitals can rely on more experienced physicians located elsewhere. In some embodiments, the control system 140 can be paired with various robot systems 110 by, for example, selecting a specific robot system and forming a secure network connection (e.g., using a password, encryption, authentication token, etc.). Thus, a physician in one location may be able to perform medical procedures in various different locations by establishing connections with robot systems 110 located in each of various different locations.

[0049] In some embodiments, the robot system 110, the table 150, the medical instrument 120, the needle and / or the imaging sensor 180 are connected in a way that allows them to communicate with one another via a network that may include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), one or more local area networks (LANs), one or more wide area networks (WANs), one or more Internet area networks (IANs), one or more cellular networks, the Internet, and the like. Furthermore, in some embodiments, the control system 140, the robot system 110, the table 150, the medical instrument 120, and / or the imaging sensor 180 are connected via one or more support cables for communication, fluid / gas exchange, power exchange, and the like.

[0050] Although not shown in Figure 1, in some embodiments, the medical system 100 includes and / or is associated with a medical monitor configured to monitor the health of patient 130 and / or the environment in which patient 130 is located. For example, the medical monitor may be located in the same environment in which the medical system 100 is situated, such as in an operating room. The medical monitor may be physically and / or electrically coupled to one or more sensors configured to detect or determine one or more physical, physiological, chemical, and / or biological signals, parameters, characteristics, states, and / or conditions related to patient 130 and / or the environment. For example, one or more sensors may be configured to determine / detect any type of physical characteristic, including temperature, pressure, vibration, force / tactile characteristics, sound, optical level or characteristics, load or weight, flow rate (e.g., of a target gas and / or liquid), amplitude, phase, and / or orientation of magnetic and electric fields, component concentrations of substances in gaseous, liquid, or solid form, etc. One or more sensors can provide sensor data to a medical monitor, which can then present information about the patient 130's health and / or the patient 130's environment. Such information may include, for example, heart rate (e.g., ECG, HRV), blood pressure / blood flow velocity, muscle biosignals (e.g., EMG), body temperature, oxygen saturation (e.g., SpO2), CO2, electroencephalogram (e.g., EEG), and ambient temperature, which are displayed via the medical monitor. In some embodiments, the medical monitor and / or one or more sensors are coupled to a control system 140, which is configured to provide information about the patient 130's health and / or the patient 130's environment.

[0051] Urinary stone capture Figures 2A and 2B show perspective views of the medical system 100 during a urinary tract stone removal procedure. In these embodiments, the medical system 100 is positioned in an operating room to remove kidney stones from a patient 130. In many examples of such procedures, the patient 130 is positioned in a modified supine position with the patient 130 slightly tilted to the side to access the posterior or lateral side of the patient 130. The urinary tract stone removal procedure may also be performed on a patient in a normal supine position, as shown in Figure 1. While Figures 2A and 2B illustrate the use of the medical system 100 to perform a minimally invasive procedure to remove kidney stones from a patient 130, the medical system 100 may be used to remove kidney stones in other ways and / or to perform other procedures. Furthermore, the patient 130 may be positioned in other positions desired for the procedure. Various actions are described in Figures 2A and 2B and throughout this disclosure as being performed by a physician 160. It should be understood that these actions may be performed directly by physician 160, indirectly by physician with the help of medical system 100, by user under the direction of physician, by another user (e.g., technician), and / or any other user.

[0052] While specific robotic arms of the robotic system 110 are illustrated in the context of Figures 2A and 2B as performing specific functions, those functions can be performed using any of the robotic arms 112. Furthermore, any additional robotic arms and / or systems can be used to perform the procedure. In addition, the robotic system 110 can be used to perform other parts of the procedure.

[0053] As shown in Figure 2A, the basket retrieval device 120 is maneuvered into the kidney 170 to approach the urinary tract stone 165. In some scenarios, a physician 160 or other user directly controls the movement of the basket retrieval device 120 using an input device 146. Such directly controlled movement may include insertion / retraction, bending the basket retrieval device 120 to the left or right, rotation, and / or regular opening / closing of the basket. Using a variety of movements, the basket retrieval device 120 is positioned near the stone.

[0054] In some embodiments, a laser, shock wave device, or other device is used to break up the stone. The laser or other device may be incorporated into the basket retrieval device 120 or may be a separate medical device. In some situations, the stone 165 is small enough that it does not need to be broken down into smaller pieces.

[0055] As shown in Figure 2B, the opened basket is maneuvered to surround the urinary tract stone 165 or a fragment of the urinary tract stone. The basket retrieval device 120 is then withdrawn from the kidney 170 and subsequently removed from the patient's body.

[0056] If further stones (or larger fragments of the fragmented stone 165) are present, the basket retrieval device 120 may be reinserted into the patient to capture the remaining larger fragments. In some embodiments, a vacuum device may be used to facilitate the removal of smaller fragments. In some situations, the stone fragments may be small enough to pass through the patient spontaneously.

[0057] Step-by-step segmentation and step-by-step recognition Automated surgical workflow analysis can be used to detect different stages in a procedure and to evaluate surgical skills and procedure efficiency. Data collected during a procedure (e.g., video data) can be segmented into multiple sections using machine learning methods, including but not limited to, hidden Markov models (HMMs) and long-term-short-memory (LTSM) networks.

[0058] In surgical stage segmentation, captured medical procedure data is automatically segmented into stages using input data from the operating room to identify the stages. Segmentation may be performed in real time during the procedure or postoperatively on recorded data. In one embodiment, surgical data can be preprocessed using dynamic time warping to divide stages into equivalent segments. Input data may consist of instrument signals, annotations, instrument (e.g., EM) tracking, or information obtained from video.

[0059] Recognition of surgical workflows can be performed at different granular levels depending on the procedure. This can be done at the stage and step level (higher level) or at the gesture and activity level (lower level). Surgical stage recognition can be performed on time-series, kinematic, and video data using machine learning techniques such as HMMs, Gaussian Mixture Models (GMMs), and Support Vector Machines (SVMs), as well as deep learning-based techniques for stage recognition from video data using Convolutional Neural Networks (CNNs). For surgical gesture and activity recognition, similar methods (SVMs, Markov models) can be used, as well as more recent deep learning-based methods such as CNNs, which can be used primarily for video data or combinations of video and kinematic data, and for recognizing the presence of tools, tasks, and activities in video data. Stage segmentation can be performed by segmenting case data into different subtasks using multiple data sources, as shown in Figure 3, or by classifying the current stage using a single data source such as video, as shown in Figure 4. In Figure 4, additional data (e.g., sensor data or UI data) can then be incorporated to further refine the output generated by the control system 140.

[0060] In Figure 3, the control system 140 receives various input data from the medical system 100. Such inputs may include video data 305 captured by the imaging sensor 180, robot sensor data 310 from one or more sensors of the robot system 110, and user interface (UI) data received from the input device 146.

[0061] Video data 305 may include video captured from a scope deployed in the patient, video captured from a camera in the operating room, and / or video captured by a camera of the robotic system 110. Robot sensor data 310 may include kinematic data from the robotic system 110 (e.g., using vibration, accelerometer, positioning, and / or gyroscope sensors), device status, temperature, pressure, vibration, force / tactile features, sound, optical level or characteristics, load or weight, flow rate (e.g., of a target gas and / or liquid), amplitude, phase, and / or orientation of magnetic and electric fields, component concentrations of substances in gaseous, liquid, or solid form, etc. UI data 315 may include button presses, menu selections, page selections, gestures, voice commands, and / or similar actions performed by the user and captured by input devices of the medical system 100. Patient sensor data, as described in Figure 1 above, can also be used as input to the control system 140.

[0062] The control system 140 can identify the stages of a medical procedure by analyzing video data 305 (for example, using a machine learning algorithm) and using robot sensor data 310 and UI data 315. In one embodiment, a medical procedure such as ureteroscopy includes several tasks (e.g., Task 1 to Task 5). Each task may be performed in one or more stages of the medical procedure. In the embodiment shown in Figure 3, Task 1 is performed in Stage 1. Task 2 is performed in Stages 2 and 4. Task 3 is performed in Stages 3 and 5. Task 4 is performed in Stages 6 and 8. Task 5 is performed in Stage 7. Time 1 (T1) indicates the time taken to complete Stage 1, Time 2 (T2) indicates the time taken to complete Stage 2, and Time 3 (T3) indicates the time taken to complete Stage 3. Other procedures may have a different number of tasks and / or a different number of stages.

[0063] In robotic procedures where manual and automated tasks exist, surgical stage detection can be used to automatically and seamlessly transition between manual and automated tasks. For example, T1 may correspond to a manual task, T2 may be an automated task, and T3 may be a manual task again. In one embodiment, when the target selection stage is active, the target selection step may be performed autonomously by the robot driving the scope. Alternatively, the user can perform site selection by selecting a point on the skin using an EM marker, and the robot can autonomously position the needle in the target insertion trajectory.

[0064] Figure 4A shows a block diagram of a control system 140 configured to generate output from video data from a medical procedure using machine learning, according to a particular embodiment. In some embodiments, the control system 140 is configured to first process video data 305 using a machine learning algorithm. In one embodiment, the video data 305 is processed by a CNN 405 to produce output 412, which identifies features recorded in the video, such as surgical tools, stones, and anatomical structures (e.g., nipples). Such identified features 415, along with the original video, may be provided as input to a recurrent neural network (RNN) 410. The RNN 410 can then process the video data 305 and the identified features 415 to generate output 412 for identifying a stage 420 in a medical procedure.

[0065] Next, supplementary data such as robot sensor data 310 or UI data 315 can be used to further refine the identified features 415 and identified stages 420 (e.g., to increase accuracy or increase the number of identifications). In other embodiments, robot sensor data 310 and / or UI data 315 may be used before the control system 140 processes the video data 305 to narrow down the possible options considered by the control system 140. For example, the supplementary data may be used to identify a specific action, which narrows the range of possible tasks and stages to those corresponding to that specific action. The control system 140 can then limit the identified features 415 and identified stages 420 to those corresponding to that specific action. For example, if a task is initially identified by the control system 140 in the video data 305 but that task is not associated with a specific action, the control system 140 can reprocess the video until the task is re-identified as a task corresponding to a specific action.

[0066] After completing the processing of the video data 305, the control system 140 can generate an annotated video containing the identified features 415 and / or identified stages 420. Such annotations may be stored as part of the video (e.g., in the same video file), in metadata stored in the database along with the video, and / or in other data formats.

[0067] Creating metadata-enhanced videos makes it easier to use videos for reviewing medical procedures. For example, viewers can jump forward or backward to specific stages of interest, rather than manually searching for when a particular stage occurred. Furthermore, multiple videos can be processed more easily to aggregate data and generate metrics. For instance, multiple videos can be searched and analyzed for instances of a specific stage (e.g., needle insertion or lithotripsy) to generate metrics related to that stage (e.g., success rate, average attempt, number of attempts, etc.).

[0068] Figure 4A shows video data 305 being processed by the control system 140, but other types of data may be processed by the control system 140 sequentially or in parallel with each other. For example, such data may include robot system 110 data such as instrument positioning measured by an electromagnetic tracking sensor, how far the scope is inserted, how much articulation the scope is performing, whether the basket is open or closed, how far the basket is inserted, and / or the connection status of the robot system. The data may be provided as input to a single neural network or multiple neural networks. For example, each different type of sensor (e.g., video, device state, telemetry, e.g., magnetic tracking, robot data, and / or fluid data) may have its own network, and the outputs of the networks may be concatenated before a final stage classification layer to obtain a single stage prediction.

[0069] Figure 4B shows one such embodiment in which different types of data from different devices and / or sensors are processed by different neural networks. Video data 305 can be processed by a first neural network 425 (e.g., a CNN and / or RNN as described in Figure 4A), robot sensor data 310 can be processed by a second neural network 430, and UI data can be processed by a third neural network 435. The outputs from the different neural networks can then be combined to generate an output 412 (e.g., a stage prediction) for the medical system 100.

[0070] Step-by-step identification process Figure 5 is a flowchart of the step identification process 500 according to a particular embodiment. The step identification process 500 can be performed by the control system 140 or by another component of the medical system 100 in Figure 1. The following describes one possible sequence for the process, but other embodiments may perform the process in a different order, or may include additional steps, or may omit one or more of the steps described below.

[0071] In block 505, the control system 140 identifies input from the UI of the robot system. For example, input may be received from an input device 146 such as a controller or a touchscreen. Possible inputs may include the selection of a treatment stage or the selection of a UI screen associated with a particular treatment stage. For example, a first screen may enumerate options for a first treatment, while a second screen may enumerate options for a second treatment. If the user makes a selection on the first screen, these selections indicate that the user is performing a first treatment. If the user makes a selection on the second screen, these selections indicate that the user is performing a second treatment. Thus, by organizing the UI screens to correspond to specific stages, the control system 140 can obtain stage information based on the user's selection. In another embodiment, one embodiment of the medical system 100 may include a UI having a first screen showing selectable stone management treatments such as ureteroscopy, percutaneous access, or mini-percutaneous nephrolithotomy (PCNL). If the user selects ureteroscopy, the control system 140 can determine that the step is related to ureteroscopy (e.g., basket treatment, laser treatment, and / or examination within the kidney). Similarly, selecting other stone management procedures indicates that the step is related to the corresponding procedure.

[0072] In block 510, the control system 140 determines a procedure from a set of procedures based on at least one of the UI inputs and sensor data. As described above, inputs from the UI interface can be used to identify the current possible procedure stage. In addition, robot sensor data can also be used to identify a procedure. For example, if it is determined that the arm of the robot system 110 is approaching the patient while holding a surgical instrument, the control system 140 may determine that the current procedure involves the insertion of a medical instrument.

[0073] In block 515, the control system 140 can narrow down the set of identifiable procedure stages to a subset of procedure stages based on the determined procedure. For example, laser treatment may be associated with tasks or stages such as starting or stopping the laser. Basket treatment may be associated with tasks or stages such as capturing a stone or retracting the basket. Insertion of the medical instrument 120 may be associated with aligning the instrument with the target and inserting the instrument into the target site. In one embodiment, if the control system 140 determines that the current procedure is a basket treatment during ureteroscopy, the control system 140 can narrow down the possible stages to capturing a stone or retracting the basket.

[0074] In block 520, the control system 140 can determine the position of the robot manipulator (e.g., robot arm 112) from the sensor data of the robot system 110. As illustrated in Figure 3, various types of sensors can be used to generate sensor data, which can then be used to determine the position.

[0075] In block 525, the control system 140 can perform analysis of the captured video. In some embodiments, such as those shown in Figure 4, machine learning algorithms are used to perform the analysis and generate outputs such as identified features and provisional identification of stages. The outputs may include the identification of physical objects, such as surgical tools or parts of anatomical structures. For example, if the control system 140 identifies a ureter in the captured video, it indicates that the stage is not related to percutaneous access. Similarly, identifying a papilla indicates that the stage is not related to basket processing. Identification of other types of anatomical structures can be used in a similar manner to rule out the possibility of a particular stage.

[0076] In block 530, the control system 140 can identify a stage from a subset of treatment stages based on at least the position of the robotic manipulator and the analysis performed. For example, if the control system 140 receives a basket insertion input via the controller, the control system 140 can determine that the stage is one of the basket processing stages. In addition, if the analysis performed identifies that the captured video shows the basket approaching a fragmented kidney stone, the control system 140 can determine that the current stage is capturing the stone. In another embodiment, if the analysis performed identifies that the captured video shows the basket being withdrawn from the fragmented kidney stone, the control system 140 can determine that the current stage is retracting the basket into the sheath. In a further embodiment, kinematic data from the robotic system 110 may indicate that the medical instrument is being withdrawn from the patient, and the control system 140 may determine that the current stage is retracting the basket into the sheath.

[0077] In block 535, the control system 140 can generate video markers for identified stages of the captured video. The video markers may be embedded as metadata in the same file as the video, as a separate file associated with the video file, or as metadata stored in a database for video annotations.

[0078] In some embodiments, the video file is annotated so that viewers can jump to specific stages within the video. For example, the video may be divided into chapters or segments corresponding to different stages. In one embodiment, the video's seek bar may be marked with colored segments corresponding to different stages, each stage being indicated by a different color.

[0079] In block 550, the control system 140 can determine whether the end of the video has been reached. If yes, process 500 can terminate. If no, process 500 can loop back to block 520 and continue identifying additional stages. For example, process 500 can loop once, twice, three times, or more times to identify the first stage, the second stage, the third stage, or more stages. The captured video may then terminate with one or more video markers, depending on the number of stages identified.

[0080] Automatic action triggers Figure 6 is a flowchart of a trigger process 600 for automated robot action according to a particular embodiment. The trigger process 600 can be performed by the control system 140 or by another component of the medical system 100 in Figure 1. The following describes one possible sequence for the process, but other embodiments may perform the process in a different order, or may include additional steps, or may omit one or more of the steps described below.

[0081] In block 605, the control system 140 can determine the state of the robot manipulator (e.g., robot arm 112) from sensor data (e.g., kinematic data) of the robot system 110. As shown in Figure 3, various types of sensors can be used to generate sensor data, which can then be used to determine the position or other state of the robot manipulator.

[0082] In block 610, the control system 140 can determine an input to initiate an action of the robot manipulator. For example, the input may come from a user operating a controller to control a basket device. In another embodiment, the input may be a screen selection or menu selection on the UI of the medical system 100.

[0083] In block 615, the control system 140 can perform analysis of the captured video. In some embodiments, such as those shown in Figure 4, machine learning algorithms are used to perform the analysis and generate outputs such as identified features and provisional identification of stages.

[0084] In block 620, the control system 140 can identify the stage of a medical procedure based at least on the state of the manipulator, the identified input, and the analysis performed. For example, if the control system 140 receives a basket insertion input via the controller, the control system 140 can determine that the stage is one of the basket processing stages. In addition, if the analysis performed identifies that the captured video shows the basket approaching a fragmented kidney stone, the control system 140 can determine that the current stage is capturing the stone. In another embodiment, if the analysis performed identifies that the captured video shows the basket being withdrawn from the fragmented kidney stone, the control system 140 can determine that the current stage is retracting the basket into the sheath. In a further embodiment, kinematic data from the robotic system 110 may indicate that the medical instrument is being withdrawn from the patient, and the control system 140 may determine that the current stage is retracting the basket into the sheath.

[0085] In block 625, the control system 140 can trigger an automatic action of the robot system 110 based on an identified stage. The triggered action may vary depending on the type of procedure being performed. Several possible actions are shown in blocks 630, 635, and 640. In block 630, the robot system 110 performs an action during ureteroscopy laser processing. In block 635, the robot system 110 performs an action during the insertion of a medical instrument such as a needle. In block 635, the robot system 110 performs an action during ureteroscopy basket processing. After the robot system 110 has triggered an action, the trigger process 600 can be terminated. Figure 7 illustrates additional details regarding specific actions that may be triggered.

[0086] Figure 7 shows different types of triggered actions of the robotic system 110 according to one embodiment. Actions may be triggered in response to identifying the current stage of an operation or identifying a user action. In some embodiments, actions may be fully automatic and performed without requiring additional input from the user. In other embodiments, actions may be partially automated and require confirmation from the user before being performed by the robotic system 110. Different combinations of stages may be performed based on the procedure being performed by the robotic system 110. Some exemplary procedures include (retrograde) ureteroscopy, percutaneous nephrolithotomy (PCNL), mini-PCNL, etc. For example, ureteroscopy may include an investigation stage (not shown), a laser processing stage, and a basket processing stage. PCNL may include a percutaneous access stage, an investigation stage, a laser processing stage, and a basket processing stage. Mini-PCNL may include an additional alignment and / or aspiration stage.

[0087] For example, during laser processing 705, triggerable actions include applying the laser to the stone 710 and stopping the laser when it is not directed at the stone 715. In one scenario, the robotic system 110 can use various sensors (e.g., a camera) to detect when the laser is directed at the stone. It may then determine the size of the stone, for example, by using a machine learning algorithm trained with recordings of previous ureteroscopy procedures, or by using a conventional computer vision algorithm (e.g., by comparing the known size of a basket with the size of the stone). Based on the determined size, the robotic system 110 can then determine an initial laser processing time based on laser processing times recorded for stones of similar size and / or type. The robotic system 110 can then stop the laser after the determined laser processing time, or when it detects that the stone has been fragmented. In other scenarios, the user can provide additional inputs, such as setting the laser processing time or providing permission for the robotic system to activate the laser.

[0088] In another scenario, the application of the laser may be triggered by the user, and the stopping of the laser may be automatically triggered by the robotic system 110. For example, the robotic system 110 can use its sensors to detect when the laser target drifts away from the stone or is otherwise not centered on the stone, and stop the laser in response.

[0089] In another embodiment, during basket processing 725, triggerable actions include capturing the stone inside the basket 730 and retracting the basket into the sheath 735. In one scenario, the robotic system 110 can trigger the operation of the basket 730 when it detects that the basket 730 is aligned with the stone and within a specified distance. The basket 730 can then be operated to capture the stone. The robotic system 110 can then use its sensors (e.g., a camera or pressure sensor) to determine whether the stone is captured inside the basket 730 and trigger the retraction of the basket into the sheath 735. The user may then retract the sheath from the patient's body, thereby removing the stone. In another embodiment, during percutaneous access 740, triggerable actions include target (renal calycere) selection 745, insertion site selection 750, and needle insertion 755 into the target site. In one scenario, the robotic system 110 can determine the target and the insertion site on the target (e.g., marked by the user or identified by the system). The robotic system 110 may then wait for confirmation from the user. After receiving confirmation, the robotic system 110 may then insert the needle (or other instrument) into the target site.

[0090] In another embodiment, during a mini-PCNL procedure, additional steps may include robotic alignment with the PCNL sheath 765 and laser treatment of the stone by active irrigation and suction 770. Actions triggered in these steps may include aligning the instrument with the PCNL sheath and increasing suction. For example, if the robotic system 110 detects an increase in stone fragments during laser treatment, or larger dust particles that would otherwise limit visibility, the robotic system 110 may increase suction or suction force to remove more stone fragments. If visibility or field of view increases, the robotic system 110 may reduce suction.

[0091] The above describes some examples and scenarios of automated actions of the robotic system 110 that can be triggered based on identified stages, but the triggerable actions are not limited to those described above. The robotic system 110 may be programmed to perform other triggerable actions based on the needs of the user and the patient.

[0092] Evaluation of tasks performed during this period Figure 8 is a flowchart of the evaluation process 800 of a task performed during an identified stage, according to a particular embodiment. The evaluation process 800 can be performed by the control system 140 or by another component of the medical system 100 in Figure 1. The following describes one possible sequence for the process, but other embodiments may perform the process in a different order, or may include additional steps, or may omit one or more of the steps described below.

[0093] In block 805, the control system 140 can determine the state of the robot manipulator (e.g., robot arm 112) from sensor data of the robot system 110. As shown in Figure 3, various types of sensors can be used to generate sensor data, which can then be used to determine the position or other state of the robot manipulator.

[0094] In block 810, the control system 140 can determine an input to initiate an action of the robot manipulator. For example, the input may come from a user operating a controller to control a basket device. In another embodiment, the input may be a screen selection or menu selection on the UI of the medical system 100.

[0095] In block 815, the control system 140 can perform analysis of the captured video. In some embodiments, such as those shown in Figure 4, machine learning algorithms are used to perform the analysis and generate outputs such as identified features and provisional identification of stages.

[0096] In block 820, the control system 140 can identify the stage of a medical procedure based at least on the state of the manipulator, the identified input, and the analysis performed. For example, if the control system 140 receives a basket insertion input via the controller, the control system 140 can determine that the stage is one of the basket processing stages. In addition, if the analysis performed identifies that the captured video shows the basket approaching a fragmented kidney stone, the control system 140 can determine that the current stage is capturing the stone. In another embodiment, if the analysis performed identifies that the captured video shows the basket being withdrawn from the fragmented kidney stone, the control system 140 can determine that the current stage is retracting the basket into the sheath. In a further embodiment, kinematic data from the robotic system 110 may indicate that the medical instrument is being withdrawn from the patient, and the control system 140 may determine that the current stage is retracting the basket into the sheath.

[0097] In block 825, the control system 140 can generate an evaluation of identified stages based on one or more metrics. The stages to be evaluated may vary based on the type of procedure being performed. Several possible stages are shown in blocks 830, 835, and 840. In block 830, the control system 140 evaluates the ureteroscopy laser processing stage. In block 835, the control system 140 evaluates the medical device insertion stage. In block 840, the control system 140 evaluates the ureteroscopy basket processing stage. Several specific examples of various evaluations are shown below.

[0098] Figure 9 is a flowchart of a scoring process 900 for a medical task according to a particular embodiment. The scoring process 900 can be performed by the control system 140 or by another component of the medical system 100 in Figure 1. The following describes one possible sequence for the process, but other embodiments may perform the process in a different order, or may include additional steps, or may omit one or more of the steps described below.

[0099] In block 905, the control system 140 counts the number of times the first treatment task is performed. In block 910, the control system 140 counts the number of times the second treatment task is performed. In block 915, the control system 140 determines the ratio of the count for the first treatment task to the count for the second treatment task. In block 920, the control system 140 can compare the determined ratio with a historical ratio. For example, the historical ratio may be generated by analyzing the historical records of the same treatment to determine the mean or median ratio.

[0100] In one embodiment, during ureteroscopy basket processing, the control system 140 can count the number of basket movements and the number of ureteroscope retractions. The control system 140 can then determine the ratio of the number of basket movements to the number of ureteroscope retractions and compare the determined ratio with other ratios from previous ureteroscopy basket procedures.

[0101] In one embodiment of ureteroscopy-driven procedures, the control system 140 can count the number of times the user manually drives the scope and the number of times the user robotically drives the scope. Manual driving is generally used to examine the kidneys. The scope, on the other hand, is typically docked to a robotic system to perform basket processing. The control system 140 can then determine the ratio of the number of times the user manually drives the scope to the number of times the user robotically drives the scope, and compare the determined ratio with other recorded ratios from previous ureteroscopy procedures. This ratio can measure the level of the user's adaptation to robotic ureteroscopy.

[0102] In another embodiment, during ureteroscopy laser treatment, the control system 140 can count the laser treatment time of the stone and determine the size and / or type of the stone. Next, the control system 140 can determine the ratio of the laser treatment time to the size of the stone and compare the determined ratio with previous ratios from other operations. By determining the type of stone (e.g., uric acid, calcium oxalate monohydrate, struvite, cysteine, bruscheit, etc.), the control system 140 can aggregate the statics of the entire surgical procedure based on the type of stone. For example, the laser treatment duration and the treatment duration can be differentiated by the type of stone.

[0103] In block 925, the control system 140 can generate comparative outputs. Such outputs may be reports, visual indicators, guides, scores, graphs, etc. For example, the control system 140 may show that the user is performing below or above a recorded ratio from a previous action, by the median or mean of the ratio. In some embodiments, the output may track the user's personal performance by comparing the current user to a record of the user's previous actions. In some embodiments, the output may compare the user to other healthcare professionals.

[0104] In one embodiment, the output may include a real-time indicator showing how the user's current performance compares to previous actions. Such an output can assist the user during surgery, for example, by providing user input on how long to run the laser treatment based on the size of the stone. Other outputs may provide the user with other relevant information.

[0105] Various types of procedural tasks can be evaluated using the scoring process 900. For example, some ratios may include the number of basket movements relative to the number of ureteroscope retractions, the number of times the user manually drives the scope relative to the number of times the user robotically drives the scope, and the laser processing time of the stone relative to the size of the stone.

[0106] Figure 10 is a flowchart of another scoring process for a medical task according to a particular embodiment. The scoring process 1000 can be performed by the control system 140 or by another component of the medical system 100 in Figure 1. The following describes one possible sequence for the process, but other embodiments may perform the process in a different order, or may include additional steps, or may omit one or more of the steps described below.

[0107] In block 1005, the control system 140 can count the first procedure task. In block 1010, the control system 140 can compare the first procedure task with a history count of the first procedure. For example, during ureteroscopy, the control system 140 can count the number of times the user attempts to insert the needle until successful, and compare that count with recorded needle insertion attempts from previous percutaneous needle insertion operations.

[0108] In another embodiment, during percutaneous needle insertion, the control system 140 may count the time taken to examine the kidney before selecting a target renal calyce for percutaneous access, and compare the counted time with the recorded time since a previous percutaneous needle insertion operation. The control system 140 may also count the number of times the automatic alignment of the robotic manipulator with the catheter is initiated during percutaneous needle insertion, and compare the counted number with the recorded number of automatic alignments since a previous operation.

[0109] During mini-PCNL alignment, the control system 140 may count the number of times automatic alignment of the robotic manipulator's end effector with the catheter or sheath is initiated, and compare the counted number with the number of recorded automatic alignments from previous operations. In another embodiment, during ureteroscopy laser processing, the control system 140 may count the number of times the view of the video capture device is obstructed by dust from stone fragmentation, and compare the counted number with the number of recorded dust obstructions from previous operations.

[0110] In block 1015, the control system 140 can generate comparative outputs. Such outputs may be reports, visual indicators, guides, scores, graphs, etc. For example, the control system 140 may show that a user is performing below or above a median or mean compared to a recorded metric from previous actions. In some embodiments, the output may track the user's personal performance by comparing the current user to a record of the user's previous actions. In some embodiments, the output may compare the user to other users.

[0111] In one embodiment, the output may include a real-time indicator showing how the user's current performance compares to previous behavior. Such an output can assist the user during surgery, for example, by indicating whether the amount of dust from the fragment is abnormal. Other outputs may provide the user with other relevant information.

[0112] Various types of procedural tasks can be evaluated using the scoring process 1000. For example, some tasks may include counting the number of times the user attempts to insert the needle until successful insertion, counting the time taken to examine the kidney before selecting a target calyce for percutaneous access, counting the number of times the navigation field generator for tracking the needle is repositioned, counting the number of times automated alignment between the robotic manipulator and the catheter is initiated, and counting the number of times the view of the video capture device is obstructed by dust from stone fragmentation.

[0113] Exemplary robotic system Figure 11 illustrates exemplary details of a robot system 110 according to one or more embodiments. In this embodiment, the robot system 110 is exemplified as a mobile, cart-based, robot-controllable system. However, the robot system 110 can be implemented as a fixed system, integrated into a table, or the like.

[0114] The robotic system 110 may include a support structure 114 comprising an elongated section 114(A) (sometimes referred to as “column 114(A)”) and a base 114(B). Column 114(A) may include one or more carriages, such as carriages 1102 (alternatively referred to as “arm supports 1102”), for supporting the deployment of one or more robotic arms 112 (three are shown in the figure). The carriage 1102 may include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robotic arm 112 for positioning relative to a patient. The carriage 1102 may also include a carriage interface 1104 that allows the carriage 1102 to translate vertically along column 114(A). The carriage interface 1104 is connected to column 114(A) through slots, such as slots 1106, which are positioned on both sides of column 114(A) to guide the vertical translation of the carriage 1102. Slot 1106 includes a vertical translation interface for positioning and holding the carriage 1102 at various vertical heights relative to the base 114(B). Vertical translation of the carriage 1102 allows the robot system 110 to adjust the reach of the robot arm 112 to accommodate various table heights, patient sizes, physician preferences, etc. Similarly, individually configurable arm mounts on the carriage 1102 allow the robot arm base 1108 of the robot arm 112 to be angled in various configurations. Column 114(A) may internally house mechanisms such as gears and / or motors, designed to use vertically aligned lead screws to translate the carriage 1102 in a mechanized manner in response to control signals generated in response to user input, such as input from an I / O device 116.

[0115] In some embodiments, the slot 1106 may be complemented by a slot cover that is coplanar and / or parallel to the slot surface to prevent dirt and / or fluid from entering the internal chamber and / or vertical translation interface of the column 114(A) as the carriage 1102 translates vertically. The slot cover may be deployed through a pair of spring spools positioned near the vertical top and bottom of the slot 1106. The cover may be coiled within the spools until it is deployed to extend and retract from a coiled state as the carriage 1102 translates vertically up and down. The spring load of the spools provides a force to retract the cover into the spools as the carriage 1102 translates toward the spools, and at the same time maintains a tight seal when the carriage 1102 translates away from the spools. The cover may be connected to the carriage 1102 using, for example, a bracket in the carriage interface 1104 to ensure proper extension and retraction of the cover as the carriage 1102 translates.

[0116] The base 114(B) can balance the weight of the column 114(A), carriage 1102, and / or arm 112 on a surface such as the floor. Thus, the base 114(B) can accommodate one or more heavy components such as electronics, motors, power supplies, and components that enable the movement and / or fixing of the robot system 110. For example, the base 114(B) may include rolling wheels 1116 (also referred to as "casters 1116") that enable the robot system 110 to move around the room for treatment. After reaching a suitable position, the casters 1116 can be locked using wheel locks to hold the robot system 110 in place during treatment. As illustrated, the robot system 110 also includes handles 1118 to assist in maneuvering and / or stabilizing the robot system 110.

[0117] The robotic arm 112 may generally comprise a robotic arm base 1108 and an end effector 1110 separated by a series of linkage mechanisms 1112 connected by a series of joints 1114. Each joint 1114 may include an independent actuator, and each actuator may include an independently controllable motor. Each independently controllable joint 1114 represents an independent degree of freedom available to the robotic arm 112. For example, each arm 112 may have seven joints and thus seven degrees of freedom. However, any number of joints can be implemented with any degree of freedom. In some embodiments, a large number of joints can provide a large number of degrees of freedom, enabling "redundant" degrees of freedom. Redundant degrees of freedom allow the robotic arm 112 to position its individual end effectors 1110 in a specific position, orientation, and / or trajectory in space using different linkage mechanism positions and / or joint angles. In some embodiments, the end effectors 1110 may be configured to engage with and / or control medical instruments, devices, objects, etc. The degrees of freedom of motion of the arm 112 allow the robotic system 110 to position and / or orient a medical device from a desired point in space, and / or allow a physician to move the arm 112 to a clinically convenient position away from the patient, thereby avoiding collisions with the arm and providing access to the device.

[0118] As shown in Figure 11, the robotic system 110 may also include an I / O device 116. The I / O device 116 may include a display, touchscreen, touchpad, projector, mouse, keyboard, microphone, speaker, controller, camera (for example, to receive gesture input), or another I / O device for receiving input and / or providing output. The I / O device 116 may be configured to receive touch, speech, gesture, or any other type of input. The I / O device 116 may be positioned at the vertical end of column 114(A) (for example, the top of column 114(A)) and / or provide a user interface for receiving user input and / or providing output. For example, the I / O device 116 may include a touchscreen (for example, a multi-purpose device) for receiving input and providing preoperative and / or intraoperative data to a physician. Exemplary preoperative data may include preoperative planning, navigation, and / or mapping data derived from preoperative computerized tomography (CT) scans, and / or notes from preoperative patient interviews. Exemplary intraoperative data may include tools / instruments, optical information provided by sensors, and / or coordinate information from sensors, as well as important patient statistics such as respiration, heart rate, and / or pulse. The I / O device 116 can be positioned and / or tilted to allow the physician to access the I / O device 116 from various positions, such as on the opposite side of the carriage 1102 of the column 114(A). From this position, the physician can view the I / O device 116, the robotic arm 112, and / or the patient, while simultaneously operating the I / O device 116 from behind the robotic system 110.

[0119] The robot system 110 may include various other components. For example, the robot system 110 may include one or more control electronic devices / circuits, a power supply, pneumatics, a light source, actuators (e.g., motors for moving the robot arm 112), memory, and / or a communication interface (e.g., for communicating with another device). In some embodiments, the memory may store computer-executable instructions, which, when executed by the control circuit, cause the control circuit to perform one of the operations considered herein. For example, the memory may store computer-executable instructions, which, when executed by the control circuit, cause the control circuit to receive inputs and / or control signals relating to the operation of the robot arm 112, and in response to this, control the robot arm 112 to position itself in a particular arrangement and / or navigate a medical device connected to the end effector 1110.

[0120] In some embodiments, the robotic system 110 is configured to engage with and / or control a medical device such as a basket retrieval device 120. For example, the robotic arm 112 may be configured to control the position, orientation, and / or end-articular movement of a scope (e.g., the scope sheath and / or leader). In some embodiments, the robotic arm 112 may be configured to operate the scope using elongated moving members. Examples of elongated moving members include one or more pull wires (e.g., pull wires or push wires), cables, fibers, and / or flexible shafts. To illustrate, the robotic arm 112 may be configured to drive a plurality of pull wires coupled to the scope to deflect the end of the scope. The pull wires may include any preferred or desired material such as metallic materials and / or non-metallic materials such as stainless steel, Kevlar, tungsten, and carbon fiber. In some embodiments, the scope is configured to exhibit nonlinear behavior in response to applied forces on the elongated moving members. The nonlinear behavior may be based on the stiffness and compressibility of the scope, as well as variability in slack or stiffness between different elongated moving members.

[0121] Exemplary control system Figure 12 illustrates exemplary details of the control system 140 according to one or more embodiments. As illustrated, the control system 140 may include, separately / individually and / or in combination / together, one or more of the following components, devices, modules, and / or units (hereinafter referred to as “Components”), namely, a control circuit 1202, a data storage / memory 1204, one or more communication interfaces 1206, one or more power supply units 1208, one or more I / O components 1210, and / or one or more wheels 1212 (e.g., casters or other types of wheels). In some embodiments, the control system 140 may include a housing / enclosure configured to house or accommodate at least one or more of the components of the control system 140 and / or a dimensioned housing / enclosure. In this embodiment, the control system 140 is illustrated as a cart-based system that is movable using one or more wheels 1212. In some cases, after reaching the appropriate position, one or more wheels 1212 can be secured using wheel locks to hold the control system 140 in place. However, the control system 140 can be implemented as a fixed system or integrated with another system / device, etc.

[0122] While certain components of the control system 140 are illustrated in Figure 12, it should be understood that additional components not shown may be included in embodiments of this disclosure. For example, a graphical processing unit (GPU) or other dedicated embedded chip may be included to run a neural network. Furthermore, in some embodiments, some of the exemplary components may be omitted. Although the control circuit 1202 is illustrated as a separate component in Figure 12, it should be understood that any or all of the remaining components of the control system 140 may be at least partially embodied in the control circuit 1202. That is, the control circuit 1202 may include various devices (active and / or passive), semiconductor materials, and / or regions, layers, areas, and / or parts thereof, conductors, leads, vias, connections, etc., and one or more and / or parts of the other components of the control system 140 may be at least partially formed and / or embodied by such circuit components / devices.

[0123] Various components of the control system 140 may be electrically and / or communicatively coupled using certain connection circuits / devices / features, and these components may or may not be part of the control circuit 1202. For example, the (multiple) connection function units may include one or more printed circuit boards configured to facilitate the mounting and / or interconnection of at least some of the various components / circuits of the control system 140. In some embodiments, two or more of the control circuit 1202, data storage / memory 1204, communication interface 1206, power supply unit 1208, and / or input / output (I / O) components 1210 may be electrically and / or communicatively coupled to one another.

[0124] As illustrated, the memory 1204 may include an input device manager 1216 and a user interface component 1218 configured to facilitate the various functionalities considered herein. In some embodiments, the input device manager 1216 and / or the user interface component 1218 may include one or more instructions that can be executed by the control circuit 1202 to perform one or more operations. While many embodiments are considered in the context of components 1216-1218 that include one or more instructions that can be executed by the control circuit 1202, any of the components 1216-1218 may be at least partially implemented as one or more hardware logic components, such as one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more program-specific standard products (ASSPs), or one or more composite programmable logic devices (CPLDs). Furthermore, although components 1216-1218 are exemplified as being included within the control system 140, any of components 1216-1218 may be at least partially implemented within another device / system, such as the robot system 110, the table 150, or another device / system. Similarly, any of the other components of the control system 140 may be at least partially implemented within another device / system.

[0125] The input device manager 1216 may be configured to receive inputs from the input device 146 and convert them into actions that can be performed by the robot system 110. For example, pre-programmed movements such as fast opening, fast closing, and small jerky movements can be stored in the input device manager 1216. These pre-programmed movements can be assigned to desired inputs (e.g., single or double button presses, voice commands, joystick movements, etc.). In some implementations, the pre-programmed movements are determined by the manufacturer. In other implementations, it may be possible for the user to modify existing pre-programmed movements and / or create new movements.

[0126] The user interface component 1218 can be configured to facilitate one or more user interfaces (also referred to as "one or more graphical user interfaces (GUIs)"). For example, the user interface component 1218 can generate a configuration menu for assigning pre-programmed movements to inputs, or a settings menu for enabling a specific operating mode or disabling a pre-programmed movement selected in a specific situation. The user interface component 1218 can also provide user interface data 1222 for display to the user.

[0127] One or more communication interfaces 1206 can be configured to communicate with one or more devices / sensors / systems. For example, one or more communication interfaces 1206 can send / receive data wirelessly and / or via a wired method over a network. Networks according to embodiments of this disclosure include local area networks (LANs), wide area networks (WANs) (e.g., the Internet), personal area networks (PANs), body area networks (BANs), and the like. In some embodiments, one or more communication interfaces 1206 can implement wireless technologies such as Bluetooth, Wi-Fi, and near-field communication (NFC).

[0128] One or more power units 1208 may be configured to manage the power of the control system 140 (and / or in some cases, the robot system 110). In some embodiments, one or more power units 1208 include one or more batteries, such as lithium-ion batteries, lead-acid batteries, alkaline batteries, and / or other types of batteries. That is, one or more power units 1208 may include one or more devices and / or circuits configured to provide a power source and / or power management functions. Also in some embodiments, one or more power units 1208 include a mains power connector configured to couple to an alternating current (AC) or direct current (DC) mains power supply.

[0129] One or more I / O components 1210 may include various components for receiving inputs and / or providing outputs, for example, for interface connection with a user. One or more I / O components 1210 may be configured to receive touch, speech, gestures, or any other type of input. In embodiments, one or more I / O components 1210 may be used to provide input for controlling a device / system, such as controlling a robotic system 110, navigating a scope or other medical instrument attached to the robotic system 110, controlling a table 150, or controlling a fluoroscopy device 190. As illustrated, one or more I / O components 1210 may include one or more displays 142 (sometimes referred to as "one or more display devices 142") configured to display data. One or more displays 142 may include one or more liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic LED displays, plasma displays, electronic paper displays, and / or any other type of technology. In some embodiments, one or more displays 142 may include one or more touchscreens configured to receive inputs and / or display data. Furthermore, one or more I / O components 1210 may include one or more input devices 146, which may include touchscreens, touchpads, controllers, mice, keyboards, wearable devices (e.g., optical head-mounted displays), virtual or augmented reality devices (e.g., head-mounted displays), etc. Additionally, one or more I / O components 1210 may include one or more speakers 1226 configured to output sound based on an audio signal, and / or one or more microphones 1228 configured to receive sound and generate an audio signal. In some embodiments, one or more I / O components 1210 may include or be implemented as a console.

[0130] Although not shown in Figure 9, the control system 140 includes and / or can be controlled one or more pumps, flow meters, valve control devices, and / or other components such as fluid access components, or devices that can be placed by the medical device, for providing controlled irrigation and / or suction capabilities to a medical device (e.g., a scope). In some embodiments, the irrigation and suction capabilities can be delivered directly to the medical device via a separate cable. Furthermore, the control system 140 may include a voltage protector and / or surge protector designed to provide filtered and / or protected power to another device such as a robotic system 110, thereby avoiding the placement of a power transformer and other auxiliary power components within the robotic system 110, making the robotic system 110 smaller and more mobile.

[0131] The control system 140 may also include support equipment for sensors deployed throughout the medical system 100. For example, the control system 140 may include optoelectronic equipment for detecting, receiving, and / or processing data received from optical sensors and / or cameras. Such optoelectronic equipment can be used to generate real-time images for display on any number of devices / systems included within the control system 140.

[0132] In some implementations, the control system 140 may be coupled to the robot system 110, the table 150, and / or medical instruments such as the scope and / or basket retrieval device 120 via one or more cables or connections (not shown). In some implementations, support functions from the control system 140 can be provided via a single cable, simplifying and tidying up the operating room. In other embodiments, specific functions can be coupled via separate cables and connections. For example, power may be provided via a single power cable, while support for control, optics, fluid mechanics, and / or navigation may be provided via separate cables.

[0133] The term “control circuit” is used herein in accordance with its broad and ordinary meanings and can refer to any set of: one or more processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connection circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, graphics processing units, field-programmable gate arrays, programmable logic circuits, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any device that manipulates signals (analog and / or digital) based on hardcoding of circuits and / or operation instructions. A control circuit may further include one or more storage devices, which can be embodied in a single memory device, multiple memory devices, and / or embedded circuits of a device. Examples of such data storage devices include read-only memory, random-access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information. In embodiments in which the control circuit includes a hardware state machine (and / or implements a software state machine) and includes analog circuits, digital circuits, and / or logic circuits, note that data storage devices / registers for storing any associated operation instructions can be embedded inside or outside the circuit including the state machine, analog circuits, digital circuits, and / or logic circuits.

[0134] The term “memory” is used herein in accordance with its broad and ordinary meanings and can refer to any suitable or desirable type of computer-readable medium. For example, computer-readable mediums include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, and / or non-removable data storage devices, which are implemented using any technology, layout, and / or data structure / protocol, and which include any suitable or desirable computer-readable instructions, data structures, program modules, or other types of data.

[0135] Computer-readable media that can be implemented by embodiments of this disclosure include, but are not limited to, phase-change memory, static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other non-temporary media that can be used to store information for access by a computing device. In the specific contexts used herein, computer-readable media may not generally include communication media such as modulated data signals and carrier waves. Therefore, computer-readable media should be understood to generally refer to non-temporary media.

[0136] Further Embodiments Depending on the embodiment, any particular action, event, or function among the processes or algorithms described herein may be performed in a different order, added, merged, or completely excluded. Therefore, in some embodiments, not all of the described actions or events are necessary for the execution of the process.

[0137] In particular, conditional language used herein, such as “can,” “could,” “might,” “may,” “eg,” and equivalents, is intended in its ordinary sense unless otherwise specifically described or understood in the context in which it is used, and is generally intended to convey that a particular embodiment includes a particular feature, element, and / or step, but other embodiments do not. Therefore, such conditional language is not generally intended to suggest that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included in or implemented in any particular embodiment, with or without author input or prompting. Terms such as “comprising,” “including,” “having,” and equivalents are used in their ordinary sense, in a non-restrictive and comprehensive manner, and do not exclude further elements, features, actions, behaviors, etc. Furthermore, the term “or,” when used, for example to connect an enumeration of elements, is used in its inclusive sense (and not its exclusive sense), meaning one, some, or all of the enumerated elements. Unless otherwise specifically stated, connecting language such as “at least one of X, Y, and Z” is understood in the context in which it is commonly used to convey that an item, term, element, etc., may be any of X, Y, or Z. Thus, such connecting language is not intended in general to imply that a particular embodiment requires the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0138] In the above description of embodiments, various features are sometimes grouped together in a single embodiment, figure, or description for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various aspects of the invention. However, the method of the disclosure should not be construed as reflecting an intention that any claim requires more features than expressly described in that claim. Furthermore, any component, feature, or step illustrated and / or described in a particular embodiment of this specification may be applied to or used in conjunction with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is required or essential for any particular embodiment. Accordingly, the scope of the invention disclosed herein and claimed below is not intended to be limited by the particular embodiments described above, but should be determined solely by a fair reading of the following claims.

[0139] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply any physical characteristics or ordering. Therefore, when used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements such as structure, components, and behavior do not necessarily indicate the priority or order of an element relative to any other element, but rather, generally, they may distinguish an element from another element having a similar or identical name (apart from the use of ordinal terms). Furthermore, when used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Additionally, actions performed "on the basis of" a condition or event may also be performed on the basis of one or more other conditions or events not explicitly listed.

[0140] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as those generally understood by those skilled in the art to which the embodiments pertain. Terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meanings in the context of the relevant art, and should not be interpreted in an ideal or overly formal sense unless expressly defined herein.

[0141] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," and "greater" are intended to encompass the concept of equality. For example, "less" can mean not only "less" in the strict mathematical sense, but also "less than or equal to."

[0142] [Implementation Method] (1) A robotic system for automatically identifying the stage of a medical procedure, wherein the robotic system is Video capture device and Robot manipulator and One or more sensors for determining the configuration of the robot manipulator, An input device configured to receive one or more user interactions and to initiate one or more actions by the robot manipulator, A control circuit that is communicatively coupled to the input device and the robot manipulator, wherein the control circuit comprises: Based on sensor data from one or more of the aforementioned sensors, the first position of the robot manipulator is determined. Based on user input and at least one of the sensor data from one or more sensors, a first action is determined from a set of actions. Based on the aforementioned determination of the first treatment, the set of treatment stages is narrowed down to a subset of the treatment stages, Performing a first analysis of the video of the patient body part captured by the video capture device, Identifying the first stage of the medical procedure from the subset of the procedure stages, at least in part, based on the first position of the robotic manipulator and the first analysis of the video, A control circuit is configured to generate a first video marker indicating the start of the first stage of the medical procedure, A robotic system equipped with the following features. (2) The robotic system according to Embodiment 1, wherein the set of procedures includes at least ureteroscopy, percutaneous nephrolithotomy (PCNL), and mini-PCNL. (3) The robotic system according to Embodiment 1, wherein the sensor data for determining the first action from one or more sensors includes radio frequency identification (RFID) data of one or more medical devices used by the robotic manipulator. (4) A UI further comprising a user interface (UI) screen associated with the set of procedures, The robot system according to Embodiment 1, wherein the first action is determined based on the user input, and the user input includes at least the selection of a first UI screen associated with the first action. (5) The control circuit is Determining the second position of the robot manipulator, Performing a second analysis of the video captured by the video capture device, The robotic system according to Embodiment 1, further configured to generate a second video marker for the video indicating the completion of a first stage of the medical procedure, based at least in part on the second position of the robotic manipulator and the second analysis of the video.

[0143] (6) The robotic system according to Embodiment 1, wherein the robotic manipulator is configured to operate a medical instrument comprising a ureteroscope. (7) The robotic system according to Embodiment 1, wherein the first stage includes a laser processing stage, the second stage includes a basketing phase, and the third stage includes a transcutaneous access stage, the second and third stages being indicated by additional video markers relative to the video. (8) The control circuit is, The robotic system according to Embodiment 1, further configured to trigger an automated movement of the robotic manipulator to a second position in response to identifying the start of the first stage. (9) The automated movement of the robot manipulator to the second position is Moving the medical device to the patient's insertion site, The robotic system according to Embodiment 8, comprising aligning the medical device along a predetermined insertion trajectory to the insertion site. (10) The control circuit is In response to the movement of the medical device by the robotic manipulator, it is determined whether or not the medical device is in the target location. The robotic system according to Embodiment 9, further configured to indicate the success of the first stage on the user interface of the robotic system in response to the medical device reaching the target location.

[0144] (11) The control circuit is, To aggregate data on the success and failure of the first stage across multiple medical procedures, The robotic system according to Embodiment 10, further configured to determine the success rate of the first stage. (12) The control circuit is The robotic system according to Embodiment 1, further configured to associate the results of the first step with a medical professional operating the robotic system. (13) The robotic system according to Embodiment 1, further comprising a sensor for determining the location of a medical device operated by the robotic manipulator. (14) A method for automatically identifying the stage of a medical procedure using a robotic system comprising a video capture device, a robotic manipulator, one or more sensors, and an input device, the method being Based on sensor data from one or more of the aforementioned sensors, the first position of the robot manipulator is determined. Based on user input and at least one of the sensor data from one or more sensors, a first treatment is determined from a set of treatments. Based on the aforementioned determination of the first treatment, the set of treatment stages is narrowed down to a subset of the treatment stages, Performing a first analysis of the video of the patient body part captured by the video capture device, Identifying the first stage of the medical procedure from the subset of the procedure stages, at least in part, based on the first position of the robotic manipulator and the first analysis of the video, To generate a first video marker indicating the start of the first stage of the medical procedure, Methods that include... (15) Determining the second position of the robot manipulator, Perform a second analysis of the aforementioned video, The method of Embodiment 14, further comprising generating a second video marker for the video indicating the completion of a first stage of the medical procedure, based at least in part on the second position of the robotic manipulator and the second analysis of the video.

[0145] (16) The method according to Embodiment 14, wherein the first procedure is ureteroscopy, the first step includes an investigation step, the second step includes a laser processing step, and the third step includes a basket processing step. (17) The above method is The method according to Embodiment 16, further comprising generating markers indicating the second and third steps based at least in part on the position of the robot manipulator, the analysis of the video, and user input from the input device. (18) The method according to Embodiment 14, further comprising triggering an automated movement of the robot manipulator to a second position in response to identifying the start of the first step. (19) The automated movement of the robot manipulator to the second position is Moving the medical device to the patient's insertion site, The method according to Embodiment 18, comprising aligning the medical device along a predetermined insertion. (20) In response to the movement of the medical device by the robot manipulator, determine whether the medical device is in the target location, The method according to Embodiment 19, further comprising indicating the success of the first stage on the user interface of the robotic system in response to the medical device reaching the target location.

[0146] (21) Aggregating data on the success and failure of the first stage across multiple medical procedures, The method according to Embodiment 20, further comprising determining the success rate of the first step. (22) A control system for automatically identifying the stage of a medical procedure performed by a robotic device, wherein the control system is A communication interface configured to receive sensor data, user input data, and video data from the robot device, A memory configured to store the sensor data, the user input data, and the video data, One or more processors, The first position of the robot device is determined from the aforementioned sensor data, Based on user input and at least one of the sensor data from one or more sensors, a first action is determined from a set of actions. Based on the aforementioned determination of the first treatment, the set of identifiable treatment stages is narrowed down to a subset of the treatment stages, Performing a first analysis of the video of the patient body part captured by the video capture device, Identifying the first stage of the medical procedure from the subset of the procedure stages, at least partially based on the first position of the robotic device and the first analysis of the video, One or more processors configured to generate a first video marker indicating the start of the first stage of the medical procedure, A control system equipped with the following features.

Claims

1. A robotic system for identifying the stage of a medical procedure, wherein the robotic system is Video capture device and Robot manipulator and One or more sensors configured to generate sensor data indicating the position of the robot manipulator, A control circuit is communicatively coupled to the robot manipulator, wherein the control circuit comprises: Based on the sensor data from one or more of the aforementioned sensors, it is determined that the robot manipulator is in a first position. Identifying the stage of the medical procedure from the set of provisional identification stages, at least in part on a machine learning algorithm configured to identify a set of provisional identification stages from the first position of the robotic manipulator and the video captured by the video capture device when the robotic manipulator is in the first position, Annotating the video with a first video marker associated with the identified stage of the medical procedure, The number of times the first treatment task was performed during the identified stage, The number of times the second treatment task was performed during the identified stage, Determine the ratio of the number of times the first treatment task was performed to the number of times the second treatment task was performed, The ratio is compared with the historical ratio generated from the historical record associated with the identified stage, To generate an output based on the above comparison, A control circuit is configured to perform the following: A robotic system equipped with [the necessary components].

2. The robotic system according to claim 1, wherein the medical procedure includes at least ureteroscopy, percutaneous nephrolithotomy (PCNL), and mini-PCNL.

3. The robotic system according to claim 1, wherein the sensor data includes radio frequency identification (RFID) data of one or more medical devices used by the robotic manipulator.

4. A UI including multiple user interface (UI) screens associated with the aforementioned medical procedure, The system further comprises an input device configured to receive one or more user inputs and to initiate one or more actions by the robot manipulator based at least partially on the one or more user inputs, The robot system according to claim 1, wherein the one or more user inputs include the selection of a UI screen from among the plurality of UI screens associated with the medical procedure.

5. The first video marker indicates the start of the identified stage of the medical procedure. The aforementioned control circuit is The robot manipulator is determined to be in a second position based on the sensor data from one or more of the aforementioned sensors. The robotic system according to claim 1, further configured to annotate the video with a second video marker indicating the end of the identified stage of the medical procedure, based at least in part on the second position of the robotic manipulator and the video captured by the video capture device while the robotic manipulator is in the second position.

6. The robotic system according to claim 1, wherein the video is annotated with a plurality of video markers associated with different stages of the medical procedure, the plurality of video markers including the first video marker.

7. The aforementioned control circuit is Determining whether a medical device is located at a target location associated with the identified stage of the medical procedure, The robotic system according to claim 1, further configured to indicate the success of the identified stage of the medical procedure on the user interface of the robotic system in response to determining that the medical device is in the target location.

8. The aforementioned control circuit is The robotic system according to claim 7, further configured to determine the success rate of the identified stage of the medical procedure based on the number of times the medical device is determined to be in the target location during the identified stage across multiple cases of the medical procedure.

9. A method performed by a control system comprising one or more processors for identifying the stage of a medical procedure, The one or more processors receive sensor data indicating the position of the robot manipulator through a communication interface, The one or more processors identify the stage of the medical procedure from the set of provisional identification stages, at least in part on a machine learning algorithm configured to identify a set of provisional identification stages from the position of the robot manipulator and video captured by a video capture device when the robot manipulator is in the position. The one or more processors annotate the video with video markers associated with the identified stage of the medical procedure, The one or more processors determine the number of times the first action task was performed during the identified stage, The one or more processors determine the number of times the second action task was performed during the identified stage, The one or more processors determine the ratio of the number of times the first treatment task is executed to the number of times the second treatment task is executed, The one or more processors compare the ratio with the historical ratio generated from the historical record associated with the identified stage, The one or more processors generate an output based on the comparison, Methods that include...

10. A control system for identifying the stage of medical treatment, A communication interface configured to receive sensor data indicating the position of a robotic device, and video data captured by a video capture device, A memory configured to store the sensor data and the video data, One or more processors, Identifying the stage of the medical procedure from the set of provisional identification stages, at least in part on a machine learning algorithm configured to identify a set of provisional identification stages from the position of the robot device and the video data captured by the video capture device when the robot device is in the position, Annotating the video data with video markers associated with the identified stage of the medical procedure, The number of times the first treatment task was performed during the identified stage, The number of times the second treatment task was performed during the identified stage, Determine the ratio of the number of times the first treatment task was performed to the number of times the second treatment task was performed, The ratio is compared with the historical ratio generated from the historical record associated with the identified stage, To generate an output based on the above comparison, One or more processors configured to perform the following: A control system equipped with the following features.

Images