Method and system for automatically tracking and managing surgical tool inventory in an operating room

Abstract

The present invention is entitled "Method and system for automatically tracking and managing surgical tool inventory in an operating room". The embodiments described herein provide various examples of automatically processing surgical videos to detect surgical tools and tool-related events and extract surgical tool usage information. In one aspect, a method for automatically detecting new surgical tool occupancy during a recorded surgical procedure is disclosed. This method may begin by receiving a surgical procedure video and then segmenting the surgical video into a sequence of video frames. Next, for each sequence of video frames, the video frames are processed to detect one or more surgical tools and one or more surgical tool occupancy associated with the detected surgical tools. If a surgical tool occupancy is detected in the sequence of video frames, the method then determines whether the detected surgical tool associated with the detected surgical tool occupancy is associated with a previously identified surgical tool occupancy. If not, the method identifies the detected surgical tool occupancy as a new surgical tool occupancy.

Description

Technical Field

[0001] The present disclosure relates generally to building surgical video analysis tools, and more specifically to systems, devices, and techniques for automatically segmenting surgical videos, detecting surgical tools and tool-related events, and extracting surgical tool usage information to guide or make recommendations to surgeons. Background Art

[0002] Tracking the use of surgical tools in the operating room (OR) and managing their inventory plays a very important role in improving surgical results and providing quality patient care. For example, some surgical tools have a limited storage life in terms of the number of times they can be fired. Therefore, it is necessary to track and record the use of these surgical tools. Currently, tracking and recording the use of surgical tools in the OR involves surgical staff using computers or manually performing these tasks by taking notes on a whiteboard. As another example, for minimally invasive surgery (MIS) such as laparoscopic surgery, detecting which tools are occupied is useful for tool inventory purposes. Today, information about which tools are occupied during surgical procedures is also collected manually by surgical staff. In fact, most surgical tool-related measurements in the OR are collected manually by surgical staff. Unfortunately, manually counting or tracking the surgical tool-related measurements for each surgical procedure is highly labor-intensive and prone to error. Summary of the invention

[0003] The embodiments described herein provide various examples of surgical tool management systems for automatically processing surgical videos to detect surgical tools and tool-related events, and extracting surgical tool usage information to guide surgical personnel or make suggestions to surgical personnel. In one aspect, a method for automatically detecting new surgical tool occupancy during a recorded surgical procedure is disclosed. This method may begin with receiving a surgical video captured for the surgical procedure. The method then segments the surgical video into a plurality of video frame sequences. Next, for each of the plurality of video frame sequences, the video frame sequence is processed to detect one or more surgical tools and one or more surgical tool occupancy associated with the one or more detected surgical tools in the video frame sequence. If surgical tool occupancy is detected in the video frame sequence, the method then determines whether the detected surgical tool associated with the detected surgical tool occupancy is associated with a previously identified surgical tool occupancy in a previously processed video frame sequence. If not, the method identifies the detected surgical tool occupancy as a new surgical tool occupancy.

[0004] In some embodiments, processing the video frame sequence to detect the one or more surgical tools includes applying a machine learning model to the video frame sequence. Specifically, the machine learning model has been trained to detect and identify a set of surgical tools associated with the surgical procedure.

[0005] In some embodiments, the method detects surgical tool occupation by determining whether the detected surgical tool continues to be detected in the sequence of video frames for a predetermined number of consecutive video frames.

[0006] In some embodiments, the predetermined number of consecutive video frames is determined based on a minimum time period required to occupy the detected surgical tool in at least the surgical procedure.

[0007] In some embodiments, the surgical video is one of: an open surgical video; an endoscopic surgical video; and a robotic surgical video.

[0008] In some embodiments, the method does not consider the detected surgical tool occupancy as a new surgical tool occupancy if the detected surgical tool associated with the detected surgical tool occupancy is associated with the previously identified surgical tool occupancy.

[0009] In some embodiments, after the detected surgical tool occupancy is identified as a new surgical tool occupancy, the method records the new surgical tool occupancy for tool inventory purposes.

[0010] In some embodiments, for a detected surgical tool, the method determines the number of times the detected surgical tool has been associated with a recorded surgical tool occupation. If the determined number of times has reached a predetermined number of uses, the method then marks the given surgical tool as expired and generates a warning to communicate to the surgical staff.

[0011] On the other hand, a method for automatically tracking the number of uses of a limited-use surgical tool is disclosed. The method may begin by receiving a surgical video containing one or more applications of a limited-use surgical tool to tissue. The method then processes a sequence of video frames in the surgical video to detect that the limited-use surgical tool is initially occupied. Next, during the time period when the detected limited-use surgical tool remains occupied, the method counts the number of uses of the limited-use surgical tool. The method then determines whether there is an existing use count of the limited-use surgical tool from one or more previous occupations of the limited-use surgical tool in the surgical video. If so, the method updates the total use count of the limited-use surgical tool by combining the use count during the current occupied time period with the existing use count from these previous occupations.

[0012] In some embodiments, the method detects that the limited-use surgical tool is initially occupied by applying a first machine learning model to the sequence of video frames, wherein the first machine learning model is trained to detect and identify the limited-use surgical tool in a given video frame.

[0013] In some embodiments, the method counts the number of uses of the limited-use surgical tool using a second machine learning model trained to detect events associated with use of the limited-use surgical tool.

[0014] In some embodiments, the limited-use surgical tool is a surgical stapler, and the second machine learning model is trained to detect a single firing of the surgical stapler each time a firing knob of the surgical stapler is pushed from a first position to a second position.

[0015] In some embodiments, the method further includes inferring a length of tissue passed by the surgical stapler during the time period by multiplying the determined firing count during the time period by a length of a cartridge of the surgical stapler.

[0016] In some embodiments, for each counted firing of the surgical stapler, the method extracts the color of the cartridge used on the surgical stapler from the associated video frame.

[0017] In some embodiments, the limited-use surgical tool is an electrocautery tool, and the second machine learning model is trained to detect a single firing of the electrocautery tool whenever a plume of surgical smoke is generated.

[0018] In some embodiments, the method further comprises the steps of determining whether a maximum number of uses for the limited use surgical tool has been reached; and if so, generating a tool expiration warning for communication to surgical personnel.

[0019] On the other hand, a device for automatically detecting new surgical tool occupancy during a recorded surgical procedure is disclosed. The device includes one or more processors and a memory coupled to the one or more processors. The memory stores instructions that, when executed by the one or more processors, cause the device to: receive a surgical video captured for the surgical procedure; segment the surgical video into a plurality of video frame sequences; and for each of the plurality of video frame sequences: process the video frame sequence to detect one or more surgical tools and one or more surgical tool occupancy associated with the one or more detected surgical tools in the video frame sequence; and if a surgical tool occupancy is detected in the video frame sequence, further determine whether the detected surgical tool associated with the detected surgical tool occupancy is associated with a previously identified surgical tool occupancy in a previously processed video frame sequence, and if not, identify the detected surgical tool occupancy as a new surgical tool occupancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The structure and operation of the present disclosure will be understood by reviewing the following detailed description and accompanying drawings, in which like reference numerals refer to like components, and in which:

[0021] Figure 1 A block diagram of an exemplary surgical tool management system 100 is shown, according to some embodiments described herein.

[0022] Figure 2 A flow chart illustrating an exemplary method for processing surgical video to detect new surgical tool occupancy during a recorded surgical procedure according to some embodiments described herein is presented.

[0023] Figure 3 A flow chart illustrating an exemplary method for processing surgical videos to track the number of uses of limited-use surgical tools according to some embodiments described herein is presented.

[0024] Figure 4 Photographic images are shown depicting a surgical stapler according to some embodiments described herein that includes a ruler printed on a cartridge segment of the surgical stapler.

[0025] Figure 5 A computer system is conceptually illustrated that may be used to implement some embodiments of the subject technology. DETAILED DESCRIPTION

[0026] The specific embodiments listed below are intended to describe various configurations of the subject technology and are not intended to represent the only configuration in which the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the specific embodiments. The specific embodiments include specific details in order to thoroughly understand the subject technology. However, the subject technology is not limited to the specific details listed herein and can be implemented without these specific details. In some cases, structures and components are shown in block diagram form to avoid blurring the concept of the subject technology.

[0027] Recorded videos of medical procedures such as surgical operations contain very valuable rich information for medical education and training, evaluation and analysis of the surgical quality and skills of surgeons, and improvement of surgical results and skills of surgeons. There are many surgical procedures that involve displaying and capturing video images of surgical procedures. For example, almost all minimally invasive surgeries (MIS) such as endoscopy, laparoscopy, and arthroscopy involve using video cameras and video images to assist surgeons. In addition, the most advanced robot-assisted surgery requires capturing intraoperative video images and displaying them on the surgeon's monitor. Therefore, for many of the aforementioned surgical procedures, such as gastric sleeve surgery or cholecystectomy, there are already a large number of surgical videos, and surgical videos are continuously generated due to a large number of surgical cases performed by many different surgeons from different hospitals. The simple fact that there is a large number of (and increasing) surgical videos for specific surgical procedures makes surgical video processing and analysis of given procedures a potential machine learning problem.

[0028] This patent disclosure provides various examples of applying machine learning and / or computer vision-based technologies to automatically perform various surgical tool-related tasks (such as tool detection, tool usage counting and tracking, and result analysis) and automatically generate warnings / guidance to surgical personnel. The disclosed automation technology frees surgical personnel from manually performing the above tasks, thereby saving precious time for other more important tasks, making surgical workflow smoother, and improving surgical results by eliminating manual errors.

[0029] In various embodiments, tool-related data can be mined from all types of surgical video feeds, including but not limited to open surgical videos, endoscopic surgical videos, and robotic surgical videos. For each type of tool-related data, data can be mined from such videos using various machine learning-based techniques, conventional computer vision techniques such as image processing, or a combination of machine learning and computer vision techniques.

[0030] Figure 1 A block diagram of an exemplary surgical tool management system 100 is shown in accordance with some embodiments described herein. Figure 1As can be seen in FIG. 1 , the surgical tool management system 100 includes at least a video source 102 and a video analysis subsystem 104. In some embodiments, the video source 102 may include a live surgical procedure video. In these embodiments, the video source 102 may be coupled to a live video capture module or may be a part thereof, which is configured to capture and record real-time surgical procedure video and / or still images during a live surgical procedure. In addition, this live video capture module may be coupled to a surgical platform such as an open surgical platform, an endoscopic surgical platform, or a robotic surgical platform or may be a part thereof. Therefore, the video source 102 may generate and provide various types of surgical videos, including but not limited to open surgical videos, endoscopic surgical videos, and robotic surgical videos. In some embodiments, the video source 102 may include a stored / archived surgical procedure video. In these embodiments, the video source 102 may be coupled to a storage system or device that stores previously captured surgical procedure videos and / or still images or may be a part thereof. Thus, video source 102 may store and provide various types of surgical videos, including, but not limited to, open surgical videos, endoscopic surgical videos, and robotic surgical videos.

[0031] As in Figure 1 As can be seen in FIG. 1 , the video analysis subsystem 104 is coupled to the video source 102 and is configured to receive live or stored surgical videos and / or still images (hereinafter collectively referred to as “surgical video images 130”) from the video source 102 and perform various surgical video analyses to extract tool-related information. More specifically, the video analysis subsystem 104 includes a set of video processing modules 108, wherein each of the set of video processing modules 108 can be used to process the surgical video images 130 to extract one or more specific types of tool-related information.

[0032] For example, the video processing module 108 may include a tool occupancy detection module 108-1 that processes the surgical video image 130 and detects which surgical tools have been occupied during the recorded surgical procedure. In various embodiments, the tool occupancy detection module 108-1 may include machine learning or computer vision technology trained to detect and identify various surgical tools. The tool occupancy detection module 108-1 may employ this machine learning or computer vision technology to detect various surgical tools captured in the surgical video image 130. Thus, the tool occupancy detection module 108-1 is used to automatically identify different types of tool occupancy captured in the surgical video image 130, and automatically generate an inventory of the detected tools involved in the associated surgical procedure.

[0033] The video processing module 108 may include a tool usage counting module 108-2, which is used to process the surgical video image 130 to determine how many times a particular surgical tool has been fired or used in another way. It should be noted that some surgical tools such as surgical staplers or some power-based surgical tools (e.g., electrocautery tools) have limited use in terms of the number of times they can be fired or occupied / used in other ways. For example, a given surgical stapler or a given electrocautery tool can only be fired a limited number of times before the tool must be replaced. Therefore, it is highly desirable to record the number of firings / uses / applications of a given surgical tool with limited use. Such surgical tools with limited use are also referred to as "surgical tools with limited use" hereinafter. Conventionally, for some surgical procedures such as endoscopic procedures, this usage data is manually tracked by surgical personnel. In various embodiments, the tool usage counting module 108-2 may include machine learning or computer vision technology that is trained to recognize and record each firing of a given surgical tool. The tool usage counting module 108-2 can use this machine learning or computer vision technology to automatically identify each time a given limited-use surgical tool is fired or used in another way, thereby recording the use of a given surgical tool. For example, for a surgical stapler, a machine learning model can be trained to detect each time the firing knob of the surgical stapler is pushed from a first position to a second position to detect a single firing of the surgical stapler. For another example, for an electric cautery tool, a machine learning model can be trained to detect each time a plume of surgical smoke is generated as a single firing of a cautery tool. It should be noted that this usage information is cumulative, and the tool usage counting module 108-2 can combine usage information extracted from surgical video images 130 from multiple surgical videos.

[0034] In some embodiments, when it is determined that the maximum usage has been reached (e.g., the maximum number of firings of a surgical stapler), the tool usage counting module 108-2 can generate a warning or notification to communicate to the surgical staff (described in more detail below). In some embodiments, the tool usage counting module 108-2 can also be used to count events such as the number of staple cartridges used or the number of sponges used. Thus, the tool usage counting module 108-2 is used to automatically determine how many times a particular surgical tool has been used or the number of times a given type of surgical item has been applied without human intervention or manual processing, thereby eliminating human error.

[0035] In some embodiments, the tool usage counting module 108-2 can be configured to infer the length of tissue traversed by a surgical tool. For example, based on the determined number of uses of a surgical tool such as a surgical stapler and the length of the surgical tool, the tool usage counting module 108-2 can quickly estimate the approximate length of tissue traversed along the tissue by the surgical tool. By combining automatic tool usage counting and length estimation, the approximate length of tissue traversed by a surgical tool can be quickly estimated without manually measuring the traversed length.

[0036] The video processing module 108 may also include a tool parameter detection module 108-3, which is used to process the surgical video image 130 to determine a given parameter associated with a given surgical tool. For example, the color of the surgical stapler bin / load is an important parameter to be collected because the different colors of the bin / load represent different types of staples for different tissue thicknesses. The color of the bin / load is also an important parameter for summarizing what type of stapler bin / load has been used on the patient during the procedure during the postoperative summary procedure. Conventionally, the color of the bin / load is visually and manually tracked by surgical personnel. Some future surgical stapler designs are intended to transmit color information as a feedback signal through the surgical stapler.

[0037] In various embodiments, the tool parameter detection module 108-3 may include a machine learning or computer vision model trained to identify the color of each surgical stapler load. The tool parameter detection module 108-3 may employ this machine learning or computer vision model to automatically identify the color of each surgical stapler load captured in the surgical video image 130, thereby greatly simplifying the color recognition / tracking method without requiring design changes to the surgical stapler tool just to collect color information.

[0038] The video processing module 108 may also include a staple line verification module 108-4, which is used to process the surgical video image 130 to verify the stability and uniformity of each staple line formed in the tissue or body part. It should be noted that a defective staple line can cause leakage, especially when multiple defective staples are grouped or aligned in a specific geometric pattern. The staple line verification module 108-4 may include a machine learning or computer vision model trained to identify defective staples and calculate their relative geometry. The staple line verification module 108-4 can use this machine learning or computer vision model to automatically process the surgical video image 130 including the formed staple line, and identify defective staples and determine their relative geometry. The geometry of the defective staples determined can then be used to predict the possibility of leakage.

[0039] The video processing module 108 may also include a loose object detection module 108-5, which is used to process the surgical video image 130 to identify loose or floating objects in the tissue, such as loose or open nails. During the operation of the surgical stapler, the knife in the stapler that cuts through the tissue between the two rows of formed nail lines may sometimes accidentally cut through one of the nail lines formed, resulting in "forked nails", i.e., loose V-shaped nails, which can then float around in the patient's body. If not removed, these loose nails can move around, pierce other tissues and cause bleeding. In various embodiments, the loose object detection module 108-5 may include a machine learning or computer vision model trained to identify "forked nails" or other forms of loose nails. The loose object detection module 108-5 can use this machine learning or computer vision model to automatically process the surgical video image 130 of the formed nail line, and identify "forked nails" and other forms of loose nails. The geometry of the defective nails determined can then be used to predict potential leakage. In some embodiments, when loose nails are detected, the loose object detection module 108-5 can generate a warning or notification to communicate to the surgical staff (described in more detail below). The surgical staff can then take appropriate actions to remove these loose nails to avoid potential injuries.

[0040] It should be noted that the various video processing modules 108-1 to 108-5 are only some examples of surgical video processing modules for automatic surgical tool management. It is understood that over time, the video processing module 108 may include more and more modules to automatically process video images and extract even more types of surgical tool-related information.

[0041] like Figure 1 As shown, the video processing module 108 can generate a warning signal 110 to the surgical staff 140 in real time. The surgical staff 140 can then take appropriate actions based on the nature of the warning signal 110. For example, when the maximum usage of the surgical tool is detected, the tool usage counting module 108-2 can generate a warning signal 110 to warn the surgical staff 140. The surgical staff 140 can then replace the old surgical tool with a new surgical tool. In some embodiments, the warning signal 110 can also directly trigger the prevention of further operation of the surgical tool.

[0042] Tool occupancy detection using the tool occupancy detection module

[0043] Conventionally, tracking surgical tool occupancy during a surgical procedure requires surgical personnel to visually track each new tool occupancy and log the information into a PC or take notes on a whiteboard. Using machine learning, computer vision, or a combination of machine learning and computer vision, a computer can be programmed or trained to detect what is happening in a surgical video and then identify the tool type of different tools. The tool occupancy detection module 108-1 of the present invention can process surgical videos and detect which surgical tools have been occupied during a recorded surgical procedure.

[0044] In some embodiments, the video processing output from the disclosed tool occupation detection module can be corroborated with feedback data collected directly from the surgical system. For example, in some robotic surgical systems, each time a surgical tool is mounted on a robotic arm, tool information, such as tool type and tool ID, can be obtained through the robotic arm's interface, and this tool information can be recorded and stored. In some embodiments, the tool information collected by the surgical robot can be combined with the tool detection output of the disclosed tool occupation detection module to provide a more complete record for each detected tool occupation and to provide more accurate tool tracking.

[0045] Figure 2 A flow chart illustrating an exemplary method 200 for processing surgical video to detect new surgical tool occupancy during a recorded surgical procedure according to some embodiments described herein is presented. In one or more embodiments, the following steps may be omitted, repeated, and / or performed in a different order. Figure 2 Therefore, Figure 2 The specific arrangement of the steps shown should not be understood as limiting the scope of the technology. It should be noted that the method 200 can be implemented in the tool occupancy detection module 108-1 described above.

[0046] Method 200 begins by receiving a surgical video captured for a surgical procedure (step 202). It should be noted that the surgical video may be an open surgical video, an endoscopic surgical video, or a robotic surgical video. Next, method 200 segments the surgical video into a set of video sequences, each of which is composed of a sequence of video frames (step 204). For example, method 200 may first segment the surgical video into a set of surgical stages, and further segment a given surgical stage in the set of surgical stages into a set of surgical sub-stages or surgical tasks. More details of segmenting surgical videos into stages, sub-stages, and / or tasks are described in a related patent application to be granted, the contents of which are incorporated herein by reference. In some embodiments, each of the segmented surgical stages, sub-stages, and tasks may be further segmented into a set of video sequences of equal length, each of which includes a predetermined number of video frames in the sequence.

[0047] For each video frame sequence in the group of video sequences, method 200 then processes the video frame sequence to detect one or more surgical tool occupancy associated with one or more detected surgical tools in the video frame sequence (step 206). In some embodiments, in order to detect tool occupancy, method 200 first uses a machine learning model trained to detect and identify a group of surgical tools to detect surgical tools in a video frame sequence. For example, a machine learning model can be trained to detect and identify each of a set of required surgical tools for a given surgical procedure captured by a surgical video. In various embodiments, this machine learning model may include a regression model, a model based on a deep neural network, a support vector machine, a decision tree, a naive Bayes classifier, a Bayesian network, or a K nearest neighbor (KNN) model. In some embodiments, this machine learning model is built based on a convolutional neural network (CNN) architecture, a recursive neural network (RNN) architecture, or another form of a deep neural network (DNN) architecture.

[0048] It should be noted that processing multiple continuous video frames to detect surgical tool occupancy can be more accurate than processing a single video frame. This is because the meaningful occupancy of surgical tools is usually associated with a minimum time period. Therefore, when the time period in which the surgical tool appears in the video is shorter than the minimum time period, the surgical tool may not actually be occupied, but only presented in the video for different reasons. For example, if a surgical stapler appears in a video frame sequence and disappears in less than one second, the surgical stapler is usually not occupied during the short-term appearance. It should be noted that this minimum time period can be measured by a predetermined number of continuous video frames. Therefore, in some embodiments, after the surgical tool is detected in the video frame sequence being processed, the detected surgical tool is determined as tool occupancy only when the surgical tool remains detected to reach a predetermined number of continuous video frames. Those skilled in the art will know that for different types of surgical tools, this minimum time period can vary significantly. Therefore, after a given surgical tool has been detected, the corresponding minimum time period of the given surgical tool will be applied to determine whether the given surgical tool is actually occupied. Note that because the number of recorded occupancy times for a given surgical tool is positively correlated with the calculation of the remaining shelf life / number of uses for the given surgical tool, it is necessary to distinguish actual tool occupancy from accidental occurrences of a given surgical tool.

[0049] At step 208, method 200 determines whether surgical tool occupancy has been detected. If not, method 200 terminates or returns to step 206 to process the next video frame sequence in the set of video sequences. If surgical tool occupancy is detected in the current video frame sequence, method 200 determines whether the detected surgical tool associated with the detected tool occupancy is associated with a previously identified surgical tool occupancy in a previously processed video frame sequence in the set of video sequences (step 210). If so, method 200 terminates or returns to step 206 to process the next video frame sequence in the set of video sequences. However, if the detected surgical tool occupancy is not detected in the previously processed video frame sequence, method 200 determines that a new surgical tool occupancy has been detected, and then records the detected tool occupancy for tool inventory purposes (step 212).

[0050] It should be noted that the above-mentioned method of obtaining a tool inventory of which surgical tools have been occupied during a surgical procedure can be combined with a tool usage counting method to track the number of times a given surgical tool has been used. More specifically, for a given surgical tool with limited use, each time a surgical tool is detected and associated with a newly identified surgical tool occupation, the tool usage counting method can be activated to track and record the number of separate uses / applications of the given surgical tool to tissue during the newly identified tool occupation. It should be noted that for a given surgical tool with limited use, the usage count is accumulated within each newly identified occupation and previously identified occupation (if any) of the given surgical tool. In some embodiments, when the cumulative number of uses of a given surgical tool has reached a predetermined number of uses (e.g., a maximum number of valid uses or a maximum number of safe uses), the given surgical tool is considered expired and a replacement warning for the given surgical tool can be issued to the surgical staff. The following is combined with Figure 3 An exemplary method of tracking the total number of times a given surgical tool has been used is described.

[0051] Count tool usage using the tool usage counting module

[0052] As described above, some surgical tools, such as surgical staplers, have limited usage in terms of the number of times they can be fired or otherwise used. Conventionally, for some surgical procedures, such as endoscopic procedures, this usage data is tracked visually and manually by surgical personnel. Using machine learning, computer vision, or a combination of machine learning and computer vision, a computer can be programmed or trained to recognize and record each time a target surgical tool is fired or used in another way. The disclosed tool usage counting module 108-2 can use model-based techniques to process surgical videos and recognize and record each time a target surgical tool is fired or used in another way in each new application.

[0053] In some embodiments, the video processing output from the disclosed tool usage count module can be corroborated with feedback data collected directly from the surgical system. For example, in some robotic surgical systems, each time a surgical tool is mounted on a robotic arm, tool information, such as tool type and tool ID, can be obtained through the interface of the robotic arm, and this tool information can be recorded and stored. In these embodiments, the tool information collected by the surgical robot can be processed to determine the number of times a target surgical tool has been mounted on one or more surgical robots. This data can then be combined with the tool tracking output for the same surgical tool from the disclosed tool usage count module 108-2 to provide a more accurate surgical tool usage count.

[0054] Figure 3 A flow chart is presented showing an exemplary method 300 for processing surgical videos to track the number of uses of limited-use surgical tools according to some embodiments described herein. In one or more embodiments, the following steps may be omitted, repeated, and / or performed in a different order. Figure 3 Therefore, Figure 3 The specific arrangement of the steps shown should not be understood as limiting the scope of the technology. Note that method 300 can be implemented in the tool use counting module 108-2 described above. Note that the limited-use surgical tools described herein may include surgical staplers, power-based cautery tools, or some other power-based surgical tools.

[0055] Method 300 begins by receiving a surgical video containing one or more applications of a limited-use surgical tool to tissue (step 302). It should be noted that the surgical video may be an open surgical video, an endoscopic surgical video, or a robotic surgical video. Method 300 then processes video frames in the surgical video to detect that the limited-use surgical tool is initially occupied (step 304). For example, a trained machine learning model based on a set of images of a limited-use surgical tool as training data can be used to detect and identify the limited-use surgical tool in step 304. In various embodiments, this machine learning model may include a regression model, a model based on a deep neural network, a support vector machine, a decision tree, a naive Bayesian classifier, a Bayesian network, or a K nearest neighbor (KNN) model. In some embodiments, this machine learning model is constructed based on another form of a CNN architecture, an RNN architecture, or a DNN architecture. It should be noted that during the surgical procedure captured in the surgical video, there may be multiple occupancy of the same limited-use surgical tool separated by other surgical tasks. Thus, step 304 should be interpreted as detecting each new occupancy of a limited-use surgical tool, rather than just the first occupancy of a limited-use surgical tool in a surgical video.

[0056] Next, during the time period when the detected limited-use surgical tool remains occupied, method 300 detects the limited-use surgical tool and tracks its number of uses (step 306). For example, after initially detecting the occupation of the limited-use surgical tool, a machine learning model for detecting a single use of the limited-use surgical tool may be activated. In some embodiments, for a surgical stapler, a machine learning model may be trained to detect each time the firing knob of the surgical stapler is pushed from a first position to a second position, which may be counted as a single use. As another example, for an electric cautery tool, this machine learning model may be trained to detect each time a plume of surgical smoke is generated as a single use of the cautery tool. It should be noted that the operation of a single use of a surgical tool of limited use of one type may be significantly different from the operation of a single use of a surgical tool of limited use of another type.

[0057] After determining the number of uses of the limited-use surgical tool during the current surgical tool occupation, the method next determines whether there is an existing use count of the limited-use surgical tool from the previously detected surgical tool occupation in the surgical video (step 308). If so, the method 300 combines the new count of the number of uses of the limited-use surgical tool during the current occupation with the existing use count of the limited-use surgical tool (step 310). In some embodiments, if the limited-use surgical tool is operated within a robotic surgical system, when the limited-use surgical tool is used, the limited-use surgical tool is mounted on the robotic arm of the robotic surgical system. In these embodiments, the existing use count of the limited-use surgical tool can be stored in a storage device of the robotic surgical system, such as a tool memory. Therefore, when the limited-use surgical tool is initially engaged with the robotic arm, the tool controller of the robotic surgical system can read tool information such as tool ID and the existing use count of the limited-use surgical tool from the tool memory. Next, after the total number of uses of the limited-use surgical tool has been updated with the new count of uses, the tool controller can write the updated total number of uses of the limited-use surgical tool into the tool memory of the robotic surgical system to replace the existing use count of the limited-use surgical tool.

[0058] After the total number of uses of the surgical tool of limited use is updated, the method 300 next determines whether the maximum number of uses of the surgical tool of limited use has been reached (step 312). For example, the maximum number of uses of a surgical stapler can be determined based on the statistically effective number of uses of the surgical stapler before the surgical stapler becomes unstable. In the robotic surgical system example, the maximum number of uses of the tool can also be stored in the tool memory of the robotic surgical system, and retrieved from the memory by the tool controller for the method 300 to perform step 312. In some embodiments, once the maximum number of uses is reached, the surgical tool of limited use can be considered as expired, and the method 300 generates an expired warning or notification to communicate to the surgical staff (step 314). Otherwise, the method 300 terminates or returns to step 304 to continue tracking the use of the same limited-use surgical tool (not shown).

[0059] As described above, the use / fire count of a surgical stapler during a given tool occupation period can be used to infer the length of tissue that the surgical stapler has passed through. For example, the approximate length of tissue that a surgical stapler has passed through during a single occupation period can be estimated as the count of firings multiplied by the length of the cartridge segment of the surgical stapler. In some embodiments, the length of the cartridge can be directly extracted from an image of the surgical stapler that shows a ruler representing the length of the cartridge. For example, Figure 4 A photographic image depicting a surgical stapler 400 according to some embodiments described herein is shown, the surgical stapler including a scale 402 printed on a cartridge segment of the surgical stapler. Figure 4 As shown, the scale 402 indicates that each cartridge is approximately 6 cm long for the surgical stapler 400. Therefore, the disclosed tool usage counting module 108-2 may also include a length estimation function to infer the length of tissue passed by the surgical stapler based on the number of firings detected for a given occupancy and the extracted length information of the stapler cartridge.

[0060] It should be noted that, similar to extracting bin length information from the video image of the surgical stapler, other tool-related parameters can also be extracted from the video image of the identified surgical tool such as the surgical stapler. For example, the color of the stapler bin / load of the surgical stapler is an important parameter to be collected, because the different colors of the bin / load represent different types of staples for different tissue thicknesses. The color of the bin / load is also an important parameter for summarizing what type of stapler load has been used for the patient during the surgical procedure during the postoperative summary procedure. In some embodiments, the disclosed tool usage counting module may include an image processing mechanism that is integrated with the method 300 to determine the color of each stapler load for each detected firing of the surgical stapler. The determined color information can be combined with other tool usage information (such as the length of the tissue passed by the stapler and the number of firings of the surgical stapler extracted from the surgical video). Then, the combined tool usage information can be used for postoperative analysis.

[0061] Figure 5 Conceptually illustrates a computer system that can be used to implement some embodiments of the subject technology. Computer system 500 can be a client, server, computer, smart phone, PDA, laptop, or tablet computer with one or more processors embedded therein or coupled thereto, or any other type of computing device. Such a computer system includes various types of computer-readable media and interfaces for various other types of computer-readable media. Computer system 500 includes bus 502, processing unit 512, system memory 504, read-only memory (ROM) 510, permanent storage device 508, input device interface 514, output device interface 506, and network interface 516. In some embodiments, computer system 500 is part of a robotic surgical system.

[0062] Bus 502 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of computer system 500. For example, bus 502 communicatively connects processing unit 512 with ROM 510, system memory 504, and permanent storage device 508.

[0063] Processing unit 512 retrieves instructions to be executed and data to be processed from these various memory units in order to perform the various methods described in this patent disclosure, including the above-mentioned combined Figures 1 to 3The described method of detecting which surgical tools have been occupied during the recorded surgical procedure, the method of tracking how many times the limited-use surgical tools have been fired, and the method of evaluating the quality of the newly formed nail line. The processing unit 512 may include any type of processor, including but not limited to a microprocessor, a graphics processing unit (GPU), a tensor processing unit (TPU), an intelligent processor unit (IPU), a digital signal processor (DSP), a field programmable gate array (FPGA), and an application-specific integrated circuit (ASIC). The processing unit 512 may be a single processor or a multi-core processor in different specific implementations.

[0064] ROM 510 stores static data and instructions required by processing unit 512 and other modules of the computer system. On the other hand, permanent storage device 508 is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when computer system 500 is turned off. Some specific implementations of the subject disclosure use a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) as permanent storage device 508.

[0065] Other implementations use removable storage devices (such as floppy disks, flash drives and their corresponding disk drives) as permanent storage devices 508. Similar to permanent storage devices 508, system memory 504 is a read-write memory device. However, unlike storage devices 508, system memory 504 is a volatile read-write memory, such as random access memory. System memory 504 stores some of the instructions and data that the processor needs at run time. In some implementations, the present patent disclosure is combined with Figures 1 to 3 Various methods are described, including methods for detecting which surgical tools have been occupied during a recorded surgical procedure and methods for tracking the number of uses of limited-use surgical tools, stored in system memory 504, permanent storage device 508, and / or ROM 510. From these various memory units, processing unit 512 retrieves commands to be executed and data to be processed in order to perform some of the specifically implemented methods.

[0066] The bus 502 is also connected to an input device interface 514 and an output device interface 506. The input device interface 514 enables a user to communicate information to the computer system and select commands. Input devices used with the input device interface 514 include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 506 enables, for example, the display of images generated by the computer system 500. Output devices used with the output device interface 506 include, for example, a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD). Some specific implementations include devices that are used as both input and output devices, such as a touch screen.

[0067] Finally, if Figure 5 As shown, bus 502 also couples computer system 500 to a network (not shown) via network interface 516. Thus, the computer may be part of a computer network, such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet or a network of networks, such as the Internet. Any or all components of computer system 500 may be used in conjunction with the subject disclosure.

[0068] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed in this patent disclosure may be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above according to their functions. Whether such functions are implemented as hardware or software depends on the specific application and design restrictions imposed on the overall system. The technician can implement the described functions in an alternative manner for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0069] The hardware for implementing various exemplary logics, logic blocks, modules, and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of receiver devices, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuits specific to a given function.

[0070] In one or more exemplary aspects, the function may be implemented by hardware, software, firmware, or any combination thereof. If implemented by software, the function may be stored as one or more instructions or codes on a non-transient computer-readable storage medium or a non-transient processor-readable storage medium. The steps of the method or algorithm disclosed herein may be implemented by processor-executable instructions that may reside on a non-transient computer-readable storage medium or a processor-readable storage medium. A non-transient computer-readable or processor-readable storage medium may be any storage medium that can be accessed by a computer or a processor. By way of example but not limitation, such non-transient computer-readable storage medium or processor-readable storage medium may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage device, disk storage device or other magnetic storage device, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, disks and discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and blue discs, wherein "disks" typically reproduce data magnetically, and "disks" reproduce data optically by lasers. The above combination should also be included within the scope of non-transient computer-readable media and processor-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable storage medium and / or a computer-readable storage medium, which may be incorporated into a computer program product.

[0071] Although this patent document contains many specific details, these details should not be understood as limiting the scope of any disclosed technology or what may be claimed, but rather as descriptions of features of specific embodiments that may be specific to a specific technology. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as acting in a particular combination, and even initially protected by claims, one or more features from a claimed combination may be removed from the combination in some cases, and a claimed combination may be directed to a sub-combination or a variation of a sub-combination.

[0072] Similarly, although operations are shown in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all of the illustrated operations be performed, in order to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0073] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this patent document.

Claims

1. A computer-implemented method for automatically detecting new surgical tool occupancy during a recorded surgical procedure, the method comprising: receiving a surgical video captured for the surgical procedure; Segmenting the surgical video into a plurality of video frame sequences; as well as For each video frame sequence in the plurality of video frame sequences: processing the sequence of video frames to detect one or more surgical tools in the sequence of video frames and one or more surgical tool engagements associated with the one or more detected surgical tools; and If surgical tool occupation is detected in said sequence of video frames: Then determining whether the detected surgical tool associated with the detected surgical tool occupancy is associated with a previously identified surgical tool occupancy in a previously processed video frame sequence in the multiple video frame sequences; and if not, identifying the detected surgical tool occupancy as a new surgical tool occupancy and recording the new surgical tool occupancy for tool inventory purposes.

2. The computer-implemented method of claim 1 , wherein processing the sequence of video frames to detect the one or more surgical tools comprises: A machine learning model is applied to the sequence of video frames, wherein the machine learning model is trained to detect and identify a set of surgical tools associated with the surgical procedure.

3. The computer-implemented method of claim 1 , wherein detecting surgical tool occupancy comprises: A determination is made as to whether the detected surgical tool continues to be detected in the sequence of video frames for a predetermined number of consecutive video frames.

4. A computer-implemented method according to claim 3, wherein the predetermined number of consecutive video frames is determined based on a minimum time period required to occupy the detected surgical tool in at least the surgical procedure.

5. The computer-implemented method of claim 1, wherein the surgical video is one of: an open surgical procedure video; Endoscopic surgical videos; and Robotic surgery video.

6. The computer-implemented method of claim 1 , wherein if the detected surgical tool associated with the detected surgical tool occupancy is associated with the previously identified surgical tool occupancy, the method further comprises: Detected surgical tool occupancy is not considered a new surgical tool occupancy.

7. The computer-implemented method of claim 1 , wherein the method further comprises: determining, for a detected surgical tool, a number of times the detected surgical tool has been associated with a recorded surgical tool occupation; as well as If the determined number of times has reached a predetermined number of uses, the given surgical tool is marked as expired and a warning is generated for communication to surgical personnel.

8. A computer-implemented method for automatically tracking the number of uses of a limited-use surgical tool, the method comprising: receiving a surgical video of a single surgical procedure, the surgical video comprising one or more applications of a limited-use surgical tool to tissue; processing a sequence of video frames in the surgical video to detect that the limited-use surgical tool is initially occupied; counting a number of uses of the limited-use surgical tool during a period of time when the detected limited-use surgical tool remains occupied; Determine whether there is an existing usage count of the limited-use surgical tool from one or more previous occupancy of the limited-use surgical tool in the surgical video; and if the existing usage count exists, update the total usage count of the limited-use surgical tool by combining the usage count during the current occupied time period with the existing usage count from the previous occupancy.

9. The computer-implemented method of claim 8, wherein detecting that the limited-use surgical tool is initially occupied comprises: A first machine learning model is applied to the sequence of video frames, wherein the first machine learning model is trained to detect and identify the limited-use surgical tool in a given video frame.

10. The computer-implemented method of claim 8, wherein counting the number of uses of the limited-use surgical tool comprises: A second machine learning model is used that is trained to detect events associated with use of the limited-use surgical tool.

11. A computer-implemented method according to claim 10, wherein the limited-use surgical tool is a surgical stapler, and wherein the second machine learning model is trained to detect a single firing of the surgical stapler each time a firing knob of the surgical stapler is pushed from a first position to a second position.

12. The computer-implemented method of claim 11, wherein the method further comprises: The length of tissue passed by the surgical stapler during the time period is inferred by multiplying the determined firing count during the time period by the length of the cartridge of the surgical stapler.

13. The computer-implemented method of claim 12, wherein the method further comprises: For each counted firing of the surgical stapler, a color of the cartridge used on the surgical stapler is extracted from an associated video frame.

14. The computer-implemented method of claim 10, wherein the limited-use surgical tool is an electrocautery tool, and wherein the second machine learning model is trained to detect a single firing of the electrocautery tool whenever a plume of surgical smoke is generated.

15. The computer-implemented method of claim 8, wherein the method further comprises: determining whether a maximum number of uses of the limited-use surgical tool has been reached; and If so, a tool expiration warning is generated for communication to surgical staff.

16. The computer-implemented method of claim 8, wherein the surgical video is one of: an open surgical procedure video; Endoscopic surgical videos; and Robotic surgery video.

17. An apparatus for automatically detecting new surgical tool occupancy during a recorded surgical procedure, the apparatus comprising: one or more processors; a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the apparatus to: receiving a surgical video captured for the surgical procedure; Segmenting the surgical video into a plurality of video frame sequences; and For each video frame sequence in the plurality of video frame sequences: processing the sequence of video frames to detect one or more surgical tools in the sequence of video frames and one or more surgical tool engagements associated with the one or more detected surgical tools; and If surgical tool occupation is detected in said sequence of video frames: Then determining whether the detected surgical tool associated with the detected surgical tool occupancy is associated with a previously identified surgical tool occupancy in a previously processed video frame sequence in the multiple video frame sequences; and if not, identifying the detected surgical tool occupancy as a new surgical tool occupancy and recording the new surgical tool occupancy for tool inventory purposes.

18. A device according to claim 17, wherein the memory also stores instructions that, when executed by the one or more processors, cause the device to: detect the one or more surgical tools by applying a machine learning model to the video frame sequence, wherein the machine learning model is trained to detect and identify a set of surgical tools associated with the surgical procedure.

19. The apparatus of claim 17, wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to: For a given surgical tool, determining a number of times the given surgical tool has been associated with the identified surgical tool occupancy; and If the determined number of times has reached a predetermined number of uses, the given surgical tool is marked as expired and a warning is generated for communication to surgical personnel.

Images