Systems and methods for tracking surgical instruments - Patents.com

Abstract

Described herein are methods and systems for tracking surgical instruments. The method can be executed by one or more processors and can include receiving a first tally of surgical instruments, receiving one or more images, each image being a field of view including one or more surgical instruments, determining a second tally of surgical instruments based at least in part on the one or more received images, and providing a notification based on a comparison of the first tally of surgical instruments and the second tally of surgical instruments.

Description

[Technical field]

[0001] Priority claim This application claims priority to U.S. Provisional Patent Application No. 62 / 856,701, filed June 3, 2019, and entitled "SYSTEMS AND METHODS FOR TRACKING SURGICAL INSTRUMENTS," which is incorporated by reference in its entirety.

[0002] The subject matter disclosed herein relates generally to the medical arts, and more specifically to new and useful systems and methods for tracking surgical instruments, including dedicated machines that facilitate tracking of surgical instruments, software configured computerized variants of such dedicated machines and improvements to such variants, and technological improvements that improve such dedicated machines over other dedicated machines that facilitate tracking of surgical instruments. [Background technology]

[0003] Tracking surgical instruments during medical procedures can be important to reduce the risk of instruments being inadvertently retained within a patient. Current methods for ensuring that surgical instruments are not retained within a patient revolve around manual tracking and counting of surgical instruments. However, manual counting is often inaccurate due to fatigue of medical staff, complex surgeries that make tracking of surgical instruments difficult, poor counting systems, and instruments sticking together or otherwise obscuring. Furthermore, compliance with established protocols may be incomplete, especially in long or complex procedures. Thus, retained surgical instruments remain a significant problem during medical procedures. Instruments left within a patient during surgery pose significant health risks to the patient and increase the overall time and cost of medical care. Thus, new and improved methods and systems for tracking surgical instruments during medical procedures may be desirable. [Brief description of the drawings]

[0004] [Figure 1]FIG. 1 is a schematic block diagram illustrating a method for tracking a surgical instrument, according to some exemplary embodiments. [Diagram 2] FIG. 1 is a schematic block diagram illustrating a method for tracking a surgical instrument, according to some exemplary embodiments. [Diagram 3] FIG. 13 illustrates a graphical user interface showing information related to a first tally of surgical instruments, according to some exemplary embodiments. [Figure 4] FIG. 1 is a schematic diagram illustrating a container configured to receive surgical instruments for counting, according to some exemplary embodiments. [Diagram 5] 1A-1C are schematic diagrams illustrating mounting structures configured to hold imaging devices, according to some exemplary embodiments. [Figure 6] FIG. 1 is a schematic diagram illustrating a protocol for counting surgical instruments, according to some exemplary embodiments. [Figure 7] 1 is a schematic diagram illustrating a user operating a mobile device to capture an image of a container configured to receive a surgical instrument, according to some exemplary embodiments. [Figure 8] FIG. 1 is a schematic block diagram illustrating components of a method for verifying the location of a surgical instrument, according to some exemplary embodiments. [Figure 9] 1 illustrates a display of information generated by a method for tracking surgical instruments, according to some exemplary embodiments. [Figure 10] 1 illustrates a display of information generated by a method for tracking surgical instruments, according to some exemplary embodiments. [Figure 11] 13A-13C illustrate graphical user interfaces prompting user verification of surgical fiber location, according to some exemplary embodiments. [Figure 12] 1 is a schematic diagram illustrating a method for analyzing surgical fabric, according to some exemplary embodiments. [Figure 13]FIG. 1 illustrates a graphical user interface showing information related to tracking of a surgical instrument, according to some exemplary embodiments. [Figure 14] 1A-1C are diagrams illustrating graphical user interfaces showing information related to tracking of surgical instruments at intermediate points during a surgical procedure, in accordance with some exemplary embodiments. [Figure 15] FIG. 13 illustrates a graphical user interface including a surgical instrument location summary report, according to some exemplary embodiments. [Figure 16] FIG. 1 illustrates a summary report, according to some exemplary embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Exemplary methods (e.g., algorithms) facilitate tracking of surgical instruments, and exemplary systems (e.g., specialized machines configured with specialized software) are configured to facilitate tracking of surgical instruments. The examples are merely representative of possible variations. Unless otherwise indicated, structures (e.g., structural components such as modules) are optional and may be combined or subdivided. Also, operations (e.g., procedures, algorithms, other functions, etc.) may be reordered, combined, or subdivided. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various exemplary embodiments. However, it will be apparent to one skilled in the art that the present subject matter may be practiced without these specific details.

[0006] Any one or more of the exemplary methods and systems described herein may be used to track (e.g., identify, locate, monitor, etc.) one or more surgical instruments. For example, such methods and systems may be used to compare the number of surgical instruments at various times to determine whether all surgical instruments are located. Such methods and systems may utilize computational techniques, such as computer vision, machine learning, deep learning, or any suitable combination thereof, to provide accurate identification and counting of surgical instruments. The use of advanced computational techniques may, for example, improve the accuracy of surgical instrument counting, improve compliance with counting protocols, streamline the process of surgical instrument location, reduce the overall cost and time associated with the risk of retaining surgical instruments, or any suitable combination thereof. Accurate location of surgical instruments introduced into a medical procedure may help to avoid inadvertent retention of a patient's surgical instruments after the medical procedure. Tracking of surgical instruments may also provide useful information regarding the use of surgical instruments during a procedure, compliance with established protocols, or both, and may help users trigger various responses when certain indications are present.

[0007] The methods and systems described herein can be used in a variety of settings, including hospital or clinic settings (e.g., surgical or clinical settings), military settings (e.g., battlefields), or other suitable care settings. Although the methods and systems described herein are primarily described with respect to tracking surgical instruments such as surgical fabrics (e.g., surgical sponges or surgical gauze pads), in other variations, such methods and systems may additionally or alternatively be used to track other instruments (e.g., surgical supplies or equipment).

[0008] Although exemplary methods are described herein primarily with respect to tracking surgical fabrics, such methods may be applied to any instrument used in a medical procedure. For example, such methods may include identifying, locating, or tracking other surgical instruments, such as scalpels, scissors, sutures, needles, gauze, bandages, or any other suitable instrument. Locating various types of surgical instruments may also include the use of multi-compartment containers used to tally surgical instruments, such as surgical fabrics. Such multi-compartment containers may be referred to as "tally bags." However, in some exemplary embodiments, methods for tracking surgical instruments may not utilize tally bags. Computational techniques may be applied to tally instruments without the use of tally bags and may be used to tally instruments grouped or organized in any suitable manner.

[0009] Overview of the Example Described herein are, among other things, exemplary methods for tracking surgical instruments. These methods may be executed by one or more processors. In some exemplary embodiments, a method for tracking surgical instruments includes receiving a first tally of surgical instruments, receiving one or more images, each image depicting a field of view including one or more surgical instruments, and determining a second tally of the surgical instruments based, at least in part, on the one or more received images. Some exemplary embodiments of the methods described herein may also include providing a notification based on the first tally of the surgical instruments, the second tally of the surgical instruments, or both. For example, the notification may be provided based on a comparison between the first tally of the surgical instruments and the second tally of the surgical instruments. In some exemplary embodiments, the first tally may generally be determined prior to the medical procedure, and the second tally may generally be determined during or after the medical procedure.

[0010] Receiving the first tally of surgical instruments may include identifying one or more surgical instruments depicted in the images. For example, identifying the one or more surgical instruments may include using computer vision, machine learning, other computational techniques, or any suitable combination thereof. In some exemplary embodiments, receiving the first tally of surgical instruments may be based, at least in part, on detecting a weight of the one or more surgical instruments. In certain exemplary embodiments, the first tally of surgical instruments may be determined, at least in part, based on user input. Determining the second tally of surgical instruments may include identifying one or more surgical instruments depicted in the one or more received images. One or more of the received images may be, for example, a depth image (e.g., an infrared depth image), a color image, or other suitable type of image. In some exemplary embodiments, identifying the one or more surgical instruments depicted in the one or more received images includes identifying and characterizing a container, including at least one compartment, depicted in the one or more images. Characterizing the container may include determining one or more parameters of the container. Determining the one or more parameters of the container may include, for example, quantifying one or more compartments within the container, determining a geometric arrangement of one or more compartments within the container, determining the presence or absence of surgical instruments in at least some (e.g., each) of the one or more compartments, or any suitable combination thereof. In various exemplary embodiments, determining the second tally of surgical instruments may additionally or alternatively be based, at least in part, on detecting a change in weight of a container configured to hold one or more surgical instruments.

[0011] Determining the second tally of the surgical instruments may include updating a current index value of the surgical instruments. Some exemplary embodiments of the methods described herein include displaying the current index value of the surgical instruments on a display (e.g., on a graphical user interface of a mobile computing device). Certain exemplary embodiments of such methods include prompting a user to verify accuracy of the first tally of the surgical instruments, the second tally of the surgical instruments, or both tally of the surgical instruments. Various exemplary embodiments of these methods include displaying the first tally of the surgical instruments, the second tally of the surgical instruments, a notification, or any suitable combination thereof on a display (e.g., a display screen). For example, some exemplary embodiments of the methods described herein include providing a summary report of the tracked surgical instruments. In certain exemplary embodiments, the method may include providing a notification based on a comparison between the first tally and the second tally and may include generating an alert in response to the first tally not matching the second tally. The alert may follow a predetermined protocol. For example, the alert may display a predetermined protocol, guide the user through the protocol, or both.

[0012] Also described herein are exemplary systems for use in, among other things, identifying, tracking, and locating surgical instruments. The system for tracking surgical instruments may include one or more processors. For example, the one or more processors may be configured to receive a first tally of surgical instruments, receive one or more images, each image depicting a field of view including one or more surgical instruments, and determine a second tally of surgical instruments based, at least in part, on the one or more received images. The one or more processors may also provide a notification based on the first tally of surgical instruments, the second tally of surgical instruments, or both. For example, the notification may be provided based on a comparison of the first tally of surgical instruments and the second tally of surgical instruments. The one or more processors may also be configured to identify one or more surgical instruments depicted in the one or more received images. The one or more surgical instruments may be a surgical fabric (e.g., a surgical sponge or a piece of surgical gauze). At least one of the received images may be a depth image. In some exemplary embodiments, the depth image is an infrared image. At least one of the received images may be a color image. The one or more processors may also be configured to identify and characterize a container including at least one compartment and depicted in the one or more images. In some exemplary embodiments, the one or more processors are configured to determine one or more parameters of the container. Example parameters include a geometric arrangement of one or more compartments within the container, the presence or absence of a surgical instrument in each of the one or more compartments, or both. The container may be a flexible container including a lining and a plurality of pockets arranged in a grid (e.g., a rectangular grid). The pockets of the container may be transparent. The lining of the container may be composed of an opaque material, a colored material, or any suitable combination thereof.

[0013] Certain exemplary embodiments of the system described herein further include an optical sensor configured to generate images received by the one or more processors. In some exemplary embodiments, the system includes a display screen configured to display a summary report of the tracked surgical instruments. The display screen may also be configured to display a notification, for example, based on a comparison between the first tally and the second tally. According to some exemplary embodiments, the notification includes an alert according to a predetermined protocol if the first tally does not match the second tally. According to certain exemplary embodiments, at least one of the one or more processors is included in a mobile computing device. The mobile computing device may be removably mounted to a stand or other suitable mounting device.

[0014] Certain exemplary embodiments of the methods described herein include determining the presence of one or more surgical instruments, and such methods may be executed by one or more processors. For example, tracking the surgical instruments may include receiving an image of a field of view, the field of view including one or more surgical instruments, determining the presence of the one or more surgical instruments in the image, and providing an indication of the determined presence of the one or more surgical instruments in the image. Certain exemplary methods further include quantifying the one or more surgical instruments in the image based on at least one machine learning model. In some exemplary embodiments, the machine learning model may incorporate one or more deep learning techniques. Other computational techniques may additionally or alternatively be used to determine the presence of the one or more surgical instruments in the image. Exemplary methods may include detecting instruments in each of one or more compartments of a container. Determining the presence of the one or more surgical instruments may include detecting the presence of a container in the image, the container including multiple compartments, each compartment configured to receive at least one respective surgical instrument. The one or more surgical instruments may include one or more surgical fibers, one or more surgical equipment, or any suitable combination thereof. According to some example embodiments, the methods described herein include classifying each compartment of the container based on the presence or absence of a surgical instrument. Providing an indication of the determined presence of a surgical instrument may include providing an indication of the determined presence of at least one surgical instrument depicted in the image.

[0015] Example of a method for tracking surgical instruments Described herein are exemplary methods for tracking (e.g., identifying, detecting, locating, detecting objects over time, etc.) surgical instruments during a medical procedure or the like. Such methods for tracking surgical instruments may use various computational components, such as processors, optical detection systems, computer vision, machine learning, deep learning algorithms, or any suitable combination thereof, to automate and streamline the process of identifying, tracking, and locating surgical instruments. FIG. 1 provides a schematic diagram of several exemplary embodiments of a method 100 for tracking surgical instruments in a medical procedure. The method 100 may include determining a first tally of one or more surgical instruments S110, determining a second tally of one or more surgical instruments S120, and providing a notification S130 based on at least one of the first tally of the one or more surgical instruments and the second tally of the one or more surgical instruments. For example, a notification may be provided based on a comparison between the first tally and the second tally (e.g., verifying a match between the first tally and the second tally, tracking progress toward incrementally increasing the tally until the first tally and the second tally match, alerting a user if the first tally and the second tally fail to match (e.g., do not match), or any suitable combination thereof).

[0016] An additional exemplary embodiment of a method 200 for tracking surgical instruments is shown in Figure 2. The method 200 may include receiving, with one or more processors, a first tally of surgical instruments S220, receiving one or more images S230, each image being a field of view including one or more surgical instruments, and determining a second tally of surgical instruments S240 based at least in part on the one or more received images. The method 200 may include providing a notification S250 based on a comparison between the first tally of surgical instruments and the second tally of surgical instruments.

[0017] In some exemplary embodiments, a first tally of surgical instruments may be associated with a first time, while a second tally of surgical instruments may be associated with a second time after the first time. For example, the first tally of surgical instruments may be an "initial tally" and corresponds to the number of surgical instruments introduced into an operating room, removed from packaging, or prepared for use during a medical procedure. The second tally of surgical instruments may correspond to the number of surgical instruments used during a medical procedure. As described herein, the term "used" refers to instruments intended for disposal, and used instruments may be soiled or unsoiled (e.g., blood, other liquids, other contents, or any suitable combination thereof). The second tally may be measured at an intermediate time point, or may be measured as a "final tally" at the end of the procedure (e.g., after the procedure is completed and before the end of surgery) as a "final tally."

[0018] Maintaining (e.g., updating) and comparing the number of surgical instruments at various points in a medical procedure may facilitate accurate location of the surgical instruments. This may provide the advantage of preventing instances of surgical instruments from being retained within a patient. In some exemplary embodiments, the method of comparing the tally may include matching an initial tally of surgical instruments with a final tally of surgical instruments (e.g., checking whether the initial tally matches the final tally). This matching may provide the advantage of determining whether all surgical instruments have been located prior to the end of the procedure. According to certain exemplary embodiments, the methods described herein may further include providing a notification to a user based on whether the first tally matches the second tally. If the first tally and the second tally do not match, such methods may include notifying or otherwise alerting a user to take one or more predetermined actions (e.g., searching for unlocated surgical instruments).

[0019] As noted above, any one or more of the methods described herein for tracking (e.g., identifying and / or detecting) a surgical instrument can be implemented using one or more computational techniques. In some exemplary embodiments, such methods for tracking a surgical instrument include the use of computer vision, machine learning, deep learning algorithms, or any suitable combination thereof. For example, computer vision may include one or more computational techniques for identifying objects in an image, a sequence of images, or a frame of a video. As another example, machine learning or deep learning techniques may include the use of training and computational mathematical models to perform a specific task by identifying complex patterns associated with a surgical instrument in an image (e.g., depicted in a still image, a sequence of images, or a frame of a video). In some machine learning techniques, computer algorithms "learn" to perform complex tasks by being trained on various data sets that include a large number of labeled images (e.g., labeled with features associated with a surgical instrument). The labeled data sets can be used to train a machine learning model to recognize specific patterns. This allows the model to perform the programmed task with a high level of accuracy. In some implementations, even when a model is used, the algorithm can be updated based on actual data received over time, and the parameters can be continually updated to improve object tracking, detection performance, or both, thereby continuing to learn. For example, machine learning can be used to generate a model that identifies one or more instruments, aggregates one or more instruments, or both. Deep learning is a type of machine learning, and can be based, for example, on artificial neural networks that use multiple layers or analyses to extract relevant features from input data to detect, identify, or track surgical instruments. As used herein, the term "computational technique" refers to any one or more of the methods described herein and should not be limited to describing one technique. Additionally, in some implementations, aspects of multiple computational techniques can be combined.For example, one or more computer vision techniques may be implemented via machine learning, deep learning, or both. For example, a computer vision used to identify a surgical instrument may utilize machine learning algorithms, deep learning algorithms, or both to perform the identification of the surgical instrument.

[0020] According to various exemplary embodiments, a mobile computing device, such as a mobile phone or tablet, is used to perform various aspects of one or more of the methods described herein for tracking surgical instruments. For example, a software application executable by one or more processors on the mobile device can be used to perform one or more operations including determining a first tally, determining a second tally, comparing the first tally to the second tally, and notifying a user based on the comparison. The one or more processors on the mobile device can be configured to analyze the image. The one or more processors of the mobile device can be configured to perform machine learning algorithms, computer vision techniques, deep learning algorithms, or any suitable combination thereof in association with any one or more of the methods described herein. In some exemplary embodiments, the mobile application is configured to perform all stages of the method of tracking surgical instruments and display information to one or more users based on one or more operations (e.g., stages) of the method that have been performed. Implementing one or more of such methods of tracking surgical instruments on a mobile device can provide the advantage of automating and streamlining the process of locating surgical instruments as well as providing an additional virtual presence that guides the user through the appropriate locating protocol that facilitates accurate locating of the surgical instruments. Additionally, implementing computational techniques on a mobile device can help make the improved method of tracking surgical instruments highly accessible and intuitive to use. In other exemplary embodiments, some or all of the operations of the method can be performed using one or more external processors. For example, the mobile device can be used to acquire one or more images, receive information, and display information, and the external processor can be used to perform analysis of one or more images using one or more machine learning algorithms.

[0021] Example of determining the first aggregate Some exemplary embodiments of methods for tracking surgical instruments include receiving a first tally of surgical instruments. In some implementations, the first tally may establish a corresponding number of one or more surgical instruments that have been introduced or prepared (e.g., introduced into an operating room, removed from packaging, or both) with the intent of being used in a medical procedure. The first tally may be established once at any point during the procedure. In some implementations, the first tally is or includes the number of surgical instruments introduced before or at the beginning of a surgical procedure. In some implementations, the first tally is then updated throughout the course of the procedure (e.g., when surgical instruments are introduced into an operating room, removed from packaging, or both). For example, one package of surgical instruments may be introduced into an operating room and removed from its packaging at the beginning of the procedure (e.g., before the patient's surgical phase begins). Subsequently, at some point during the medical procedure, such as while the surgery is being performed, a second package of surgical instruments may be retrieved and introduced into the operating room and the surgical instruments removed from their packaging. Thus, the first tally may be updated (e.g., incremented) with the number of surgical instruments in the second package. Any number of surgical instruments may be introduced at any time during the procedure, and the first tally may be updated each time a new set of surgical instruments is introduced.

[0022] Establishing the first tally of instruments may provide the benefit of facilitating location verification of all surgical instruments at the end of the procedure. For example, the methods described herein may include comparing a tally of instruments present at the end of the procedure to the first tally to determine whether all surgical instruments are located (e.g., as opposed to being held within the patient). Various suitable methods may be used to establish the first tally of surgical instruments, as described below. In some implementations, establishing the first tally of instruments may be performed manually. For example, the first tally of surgical instruments may be received as a manual input or other user input. As another example, the first tally of surgical instruments may additionally or alternatively be established by one or more processors (e.g., via one or more computational techniques, as described below). Computational techniques for establishing the first tally may provide the benefit of streamlining the process of establishing the first tally and determining the first tally with increased accuracy and reduced user error, as compared to manual methods, for example.

[0023] In some exemplary embodiments, as described above, the first tally of the instruments may be received as a user input. For example, establishing the first tally of the surgical instruments may include a manual tally technique. A user may manually tally a plurality of surgical instruments to determine a first tally and provide the first tally via a user interface. For example, a user may receive a package containing one or more surgical instruments, retrieve the instruments from the package, and tally the surgical fibers as the user separates the fibers from one another, such as while placing them on a surface such as a table. In some implementations, the manual tally is entered into the computing system by a user (e.g., manually entered into a keypad or touch screen, verbally, etc.). For example, a user may enter the manual tally into the computing system via a mobile application executable on a mobile computing device. In some implementations, the mobile application may prompt the user to enter the manual tally.

[0024] In some exemplary embodiments, determining the first tally of surgical instruments may additionally or alternatively include the use of computer vision, machine learning, deep learning, other computational techniques, or any suitable combination thereof. For example, establishing the first tally may include analyzing the images to identify one or more surgical instruments, tally the one or more surgical instruments, or both. The images of the one or more surgical instruments to be tally in the first tally may be acquired using any suitable imaging device. For example, an imaging system such as on a mobile computing device (e.g., a mobile phone, tablet, etc.) may be used to capture images of the one or more surgical instruments. The images may be still images or frames from a video feed. In some implementations, multiple images (e.g., multiple frames from a video feed) are processed to determine the first tally of surgical instruments. In some implementations, the images are or include color images, while in other implementations, the images are or include depth images (e.g., infrared, stereoscopic, ultrasound, etc.), hyperspectral images, or other suitable types of images. Additionally, the multiple images of the one or more surgical instruments may be acquired simultaneously, near simultaneously, or sequentially, and may be of different types (e.g., color images combined with depth images). In general, one or more processors may be configured to receive the one or more images, analyze the one or more images, or both. Additionally, the one or more processors may implement computational techniques (e.g., using computer vision, machine learning, deep learning, or any suitable combination thereof) to identify the one or more surgical instruments, tabulate the one or more surgical instruments, or both. The images received by the one or more processors depict the one or more surgical instruments. Such a processor may be implemented in a computing device, such as a mobile computing device.In some implementations, a mobile application executable on a computing device can be used in combination with any of the variations described below to acquire, receive, and process data related to the first tally of surgical instruments.

[0025] In some exemplary embodiments, determining a first tally of surgical instruments using image analysis includes the following operations: A user obtains a package including one or more surgical instruments, such as a surgical garment, and brings the package into a surgical room. The user may remove the one or more surgical instruments from the package, separate the surgical instruments, and place them on a surface, such as a table. The user may then use an imaging system, such as a camera on a mobile computing device, to obtain one or more images of the surgical instruments on the table. A processor of the mobile computing device then analyzes the images to identify individual surgical instruments in the images and tally and / or otherwise quantify the number of surgical instruments in the images. For example, the first tally of surgical instruments may be determined using computational techniques, such as computer vision, machine learning, deep learning techniques, or any suitable combination thereof, including any one or more of those described in further detail below with respect to determining a second tally of surgical instruments. The processor may store information regarding the detection and tally of the surgical instruments in a memory. In some implementations, determining the first tally includes updating the first tally by adding the number of surgical instruments counted by the processor in the image to a previously counted number of surgical instruments (e.g., surgical instruments depicted in an image captured at a prior time point). Additionally or alternatively, determining the first tally may include counting the number of surgical instruments in an image captured at a time point (e.g., if no surgical instruments have been counted previously).

[0026] In some exemplary embodiments, the one or more computational techniques include detecting a characteristic arm motion of the user to determine the first tally of surgical instruments. The characteristic arm motion may be characterized in any suitable manner. For example, detecting the characteristic arm motion may include identifying a "handling" motion as the user separates the surgical instruments as the user removes the surgical fibers from the packaging, as medical personnel often "handle" the surgical fibers on a surface during a surgical procedure and tally them. An imaging system with a camera may be positioned such that the system views the user's arm motion (e.g., a front view, a side view, an overhead view, or any suitable combination thereof), and a processor may receive still frame images or videos from the imaging system. Sweeping (e.g., handing) arm motions as the user removes the surgical fibers from the packaging and places the surgical fibers on a surface may be detected using the imaging system and computational analysis (e.g., performed by the processor). Thus, each arm motion detected by the processor may correspond to a tabulated surgical instrument and may contribute to the first tally of surgical instruments.

[0027] In certain exemplary embodiments, the one or more computational techniques utilize the weight information to determine the first tally. For example, surgical instruments can be sequentially placed on the scale, and incremental weight changes can be detected by one or more processors to determine the first tally of the surgical instruments. That is, the processor may be able to detect the presence of additional surgical instruments based on incremental weight changes that occur when additional surgical instruments are placed on the scale. Each time the processor detects an incremental weight change, the processor may increment the first tally.

[0028] Additionally or alternatively, according to some exemplary embodiments, multiple surgical instruments can be placed on the scale and information regarding the weight of each individual surgical instrument can be used to determine a first tally of the surgical instruments. For example, a user can be prompted via a user interface to input the type of surgical instrument to be weighed. The type of surgical instrument can be input, for example, by searching a database of existing surgical instrument types, by navigating a hierarchical menu of surgical instrument types, or by inputting the surgical instrument type. The types of surgical instruments can be classified or otherwise differentiated, for example, based on function (e.g., fabric, type of surgical equipment, etc.) or instrument-specific characteristics such as brand, size, material, etc. Additionally or alternatively, the surgical instrument type can be determined based on other identifications (e.g., scanning a bar code or radio frequency identification (RFID) tag associated with the surgical instrument type). A processor in communication with the scale can receive information regarding the weight of a single surgical instrument based on the type of surgical instrument received. Thus, information regarding the weight of the surgical instruments may be manually entered by a user, stored in memory in a look-up table, or obtained by scanning an electronic tag, such as a barcode or RFID tag, on the surgical instrument. Based on the information regarding the weight of a single surgical instrument, the processor may determine the number of surgical instruments on the scale by dividing the total weight of the multiple surgical instruments by the known weight of a single surgical instrument.

[0029] As described above, the mobile application can be used in combination with one or more weight-based methods of determining the first tally. FIG. 3 illustrates an exemplary embodiment of a graphical user interface 300 illustrating information related to determining the first tally through the use of a scale. The graphical user interface of FIG. 3 displays the type of surgical instrument 310 being tallied, and a current determination 320 of the first tally. The graphical user interface 300 prompts the user to place a new surgical instrument on the scale 330. Additionally, the graphical user interface 300 provides selectable icons 340a and 340b to allow the user to manually override the first tally (340a) to provide a manual tally, and to end the first tally (340b) (e.g., indicating that all surgical instruments of the selected type have, at least for now, been tallied into the first tally).

[0030] In some exemplary embodiments, the one or more computational techniques for additionally or alternatively determining the first tally include the use of scannable tags (e.g., RFID, bar codes, or any suitable combination thereof). For example, surgical instruments may include individual RFID tags that may be scanned when the surgical instruments are introduced into the operating room, when they are removed from their packaging, or at some other time prior to use. In other situations, the packaging of surgical instruments may include scannable tags configured to allow location of multiple surgical instruments with a single scan of the scannable tag. For example, the packaging of surgical instruments may include one scannable tag that communicates to a processor the number of instruments in the packaging for inclusion in the first tally. Thus, a method of determining the first tally may include scanning tags on the surgical instruments or packaging of the surgical instruments to determine the first tally.

[0031] The determination of the first tally may include one or more of the methods described above. In some exemplary embodiments, multiple methods of determining the first tally are used to verify the accuracy of the first tally. The methods of determining the first tally may be combined in any suitable manner. For example, a weight-based method, such as detecting incremental weight changes on a scale or utilizing known weight information for a single surgical instrument, may be combined with an image-based method, such as using one or more computational techniques to identify and tally the surgical instruments in an image. As another example, an image-based method may be combined with one or more computational techniques including recognition of a characteristic movement of a user. One or more manual tallying techniques may be used to verify any one or more of the methods described herein for determining the first tally. In some exemplary embodiments, the mobile application prompts the user to verify the determination of the first tally generated using the computational techniques. In some exemplary embodiments, as described above with respect to FIG. 3, the method includes allowing the user to override the automatic or computational determination of the first tally if the user determines that the first tally is inaccurate.

[0032] In some exemplary embodiments, determining the first tally of the surgical instruments includes categorizing the surgical instruments. The determination of the first tally may include distinguishing between different surgical instruments, determining different types of the same surgical instrument, or both. For example, different types of surgical fabrics may be distinguished from one another (e.g., by material, function, size, brand, or any suitable combination thereof). As another example, two different types of surgical instruments may be distinguished from one another (e.g., distinguishing a scalpel from a pair of scissors). Any suitable method, including any one or more of the computational techniques described herein, may be used to determine the type of one or more surgical instruments, such as the type of one or more surgical fabrics. In some implementations, determining the first tally includes indicating separate categories of first tally in addition to a first tally of the total of all surgical instruments tallied, regardless of type. For example, the first tally may include a type-specific first tally for 18 inch by 18 inch surgical fabrics, and another type-specific first tally for 4 inch by 4 inch surgical fabrics.

[0033] Any of the methods described herein for determining the first tally may additionally or alternatively be used to determine the second tally. Similarly, any of the methods described herein for determining the first tally may additionally or alternatively be used to perform any other suitable analysis of the surgical instruments (e.g., tracking used instruments, tracking unused instruments, or both, as described below).

[0034] Example of determining the second aggregate Some exemplary embodiments of the method for tracking surgical instruments further include determining a second tally of the surgical instruments. As described above, the second tally of the surgical instruments can be compared to the first tally of the surgical instruments. The second tally can be determined at any suitable time during the procedure (e.g., a medical procedure such as a surgical procedure). For example, the second tally can be an "interim tally" determined at an intermediate point during the procedure, between the start and end of the procedure. Determining the interim tally can include updating an index counter of the surgical instruments during (e.g., throughout) the course of the procedure (e.g., as surgical instruments are used or designated for intent to be discarded). The interim tally can be compared to the first tally to indicate the number of "used" surgical instruments remaining at the site during the course of the procedure. The second tally can also be a final tally at the end of the procedure. It may be desirable to compare the final tally to the first tally to determine if all surgical instruments have been accounted for, limiting the risk of surgical instruments being retained within a patient undergoing the procedure. The end of the procedure can be indicated in any suitable manner. For example, the end of the procedure may be when the patient is ready to be removed from the operating room (e.g., after the surgical site has been closed), or before the patient's surgical site has been closed, but after the surgical procedure is completed.

[0035] Any of the methods described herein for determining the second tally may additionally or alternatively be used to determine the first tally. Similarly, any of the methods described herein for determining the second tally may additionally or alternatively be used to perform any other suitable analysis of the surgical instruments (e.g., tracking used instruments, tracking unused instruments, or both, as described below).

[0036] Example of using tally bags In some exemplary embodiments, tallying the surgical instruments includes the use of a tally bag to verify the location of the surgical fibers. The use of the tally bag may facilitate accurate and systematic tallying of the surgical fibers and reduce the risk of the surgical instruments being retained in the patient's body. One or more users may place surgical fibers designated for disposal into a pocket of the tally bag to facilitate efficient and accurate tallying. Additionally, disposal of surgical fibers or other instruments may, in some exemplary embodiments, include a multi-step instrument collection process. For example, in some operating room procedures, after the surgical fibers are used in the procedure, they may accumulate in one or more collection containers (e.g., a "kick bucket" in the form of an example of a metal bucket lined with a plastic liner located near the operating table or elsewhere in the operating room). During or after the procedure, one or more medical personnel may then remove the surgical fibers from the one or more collection containers and place the surgical instruments in the tally bag for tallying. In other exemplary embodiments, the surgical fibers may be placed directly into the tally bag without the use of a separate collection container. All surgical fabrics used in a procedure may be located using a tally bag. As noted above, the term "used" is not limited to surgical fabrics used to absorb blood or other liquids, but may refer to any surgical fabric introduced into an operating room and subsequently deemed disposable or otherwise designated to be discarded. According to some exemplary embodiments, manual tallying of surgical fabrics includes placing surgical instruments, such as surgical fabrics, into compartments of a tally bag according to a specific protocol, as described below. One or more methods of verifying the location of surgical instruments, such as those implementing one or more computational techniques, may be used to automate and reduce errors associated with manual tallying through the use of one or more tally bags.

[0037] In some situations, medical personnel may use standardized procedures or protocols in tallying surgical instruments using one or more tally bags. FIG. 6 illustrates one such protocol for use in an operating room. The instruments are placed in pockets 610 of tally bag 600 in the order indicated by the numbers, e.g., from bottom to top and right to left. Thus, if the procedure is performed correctly, the first filled pocket will be the pocket in the lower right corner (pocket 1) and the last filled pocket will be the pocket in the upper left corner (pocket 10). According to other protocols, tally bag 600 may be filled from top to bottom and left to right. However, other suitable protocols may be followed.

[0038] FIG. 4 illustrates a schematic of a tally bag 400 for use in a medical procedure. The tally bag 400 comprises a liner 402 and a number of pockets 410 or other compartments. In FIG. 4, the tally bag 400 includes ten pockets arranged in a rectangular array. As shown in FIG. 4, the ten pockets 410 are arranged in five rows, with each row including two of the pockets 410. Each row may include, for example, a larger pocket 412 formed between the liner 402 material and the front material, which may be divided into two smaller pockets 410 with a seal 414 forming a seam in the larger pocket 412. However, the tally bag 400 may include any suitable number of pockets 410, and the pockets 410 may be configured in any suitable manner. For example, some or all of the pockets 410 may be separate compartments, some or all of the pockets 410 may be made from a single larger pocket (e.g., the larger pocket 412) with each pocket 410 separated by a seal, or any suitable combination thereof. The tally bag 400 can be mounted to a stand, such as a pole 430 (e.g., an intravenous (IV) pole). The pole 430 can include a lateral mounting rod 431 that includes one or more (e.g., two) hooks 432. The tally bag 400 can include one or more (e.g., two) openings 404 that allow the tally bag 400 to be hung from the hooks 432. The tally bag 400 shown in FIG. 4 is rectangular and includes ten rectangular pockets 410, although the tally bag 400 and its pockets 410 can be of any suitable configuration.

[0039] The tally bag 400 may include (e.g., may be made of) any suitable material. For example, the lining 402 of the tally bag 400 may be made of a flexible material, such as a polymer, plastic, silicone, nylon, rayon, any other suitable material, or any suitable combination thereof. As another example, the lining 402 of the tally bag 400 may be composed of a more rigid material, such as a rigid plastic, polymer, metal, any other suitable material, or any suitable combination thereof. The lining 402 of the tally bag 400 may be transparent, translucent, opaque, tinted, or any suitable combination thereof. For example, the lining 402 may be opaque blue. An opaque tinted lining 402 (e.g., opaque blue) may make the tally bag 400 easier to individually identify among multiple tally bags (e.g., when multiple tally bags are layered in front of each other on the same pole 430 or when multiple poles (e.g., poles 430) are layered in front of each other) and may provide a contrasting background against which the surgical fabric stands out. The pocket 410 can be made of the same or different material as the backing 402. For example, the pocket 410 can be made of a flexible plastic, silicone, or polymeric material. A pocket 410 made of a flexible material may provide the advantage of allowing surgical fabric to be easily placed therein. The pocket 410 may also be transparent or opaque. In some exemplary embodiments, one or more of the pockets 410 are transparent to allow easy visualization and counting of surgical instruments.

[0040] While the tally bag 400 shown in FIG. 4 is rectangular and includes ten rectangular pockets 410, the tally bag 400 may be of any suitable configuration. For example, the lining 402 of the tally bag 400 may be rectangular, substantially square shaped, oval, triangular, or any suitable shape. The tally bag 400 need not have ten pockets 410 as shown in FIG. 4, but may include any suitable number of pockets 410. For example, the tally bag may include 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, or any suitable number of pockets 410. While FIG. 4 shows the tally bag 400 configured to hang from a pole 430, the tally bag 400 may be attached in any suitable manner. For example, the tally bag 400 may have one or more openings 404 or any other suitable attachment mechanism in any suitable location. For example, the tally bag may be configured to be attached to a hook on a wall, or to a hook or mounting rod configured to be positioned on a top or door or cabinet. As another example, the tally bag 400 may be attached to a wall with adhesive or other suitable fasteners. Additionally, the tally bag 400 does not have to be sized to fit the dimensions of a pole (e.g., an IV pole), and the tally bag 400, its pockets 410, or both, can be sized differently from other tally bags, their pockets, or both to accommodate different types of surgical instruments, such as surgical fabrics of different sizes.

[0041] In some exemplary embodiments, the pole 430 may also include a mount for mounting an imaging device. The one or more imaging devices may be used to generate one or more images for use in various components (e.g., operations) of the method for tracking surgical instruments, including determining the first tally, determining the second tally, or both. In particular, the one or more imaging devices mounted to the pole 430 to which the tally bag 400 is mounted may be used in determining the second tally of the surgical instruments in the tally bag 400. For example, the mount for the imaging device may make it easy for a user to take (e.g., capture) an image of the tally bag 400, such that the user does not have to hold the imaging device while taking the image. The one or more mounts may be fixedly mounted to the pole 430 or may be removably secured to the pole 430. Additionally, the mount may be adjustable along the pole 430 (e.g., configured to slide vertically up and down the pole 430), rotationally adjustable around the pole 430 (e.g., adjustable to the left or right side of the pole 430), or both. FIG. 5 illustrates generally an exemplary embodiment of a tally bag stand 500 including a mount 540 attached to a pole 530 configured to hold a mobile device 550 (e.g., a tablet or smartphone). In the exemplary embodiment shown, the mount 540 includes a grip 542 configured to securely and releasably hold the mobile device 550. The grip 542 can hold the mobile device 550 using any suitable mechanism, such as via, for example, a friction fit between the mount 540 and the mobile device 550, an adjustable clamp, a snap fit with mating parts on the mount 540 and the mobile device 550, one or more magnets, a removable adhesive layer attached to the mount 540 or the mobile device 550, or any suitable combination thereof. In the exemplary embodiment shown, the mount 540 is secured to the pole 530 by an attachment mechanism 544.The mounting mechanism 544 can include any suitable mounting mechanism, such as a clamp, a clasp, a hinge, a thread, one or more fasteners (e.g., mechanical fasteners, adhesives, both), or any suitable combination thereof. For example, the mounting mechanism 544 can include an adjustable clamp (e.g., adjustable jaws) or an expandable sleeve or other insert (e.g., inflatable foam, etc.) configured to attach to and adjust to the dimensions of the pole 530. In some exemplary embodiments, the mounting mechanism 544 includes one or more fasteners configured to couple the mount 540 to the pole 530 (e.g., a screw or bolt inserted through an opening in the mount 540 and the pole 530 and secured with a nut, other threaded backing, or other fastener backing). Although the mount 540 in FIG. 5 is depicted as extending horizontally from the pole 530, the mount 540 can extend at any suitable angle such that the camera of the mobile device 550 can be positioned to capture an image of the tally bag (e.g., tally bag 400) attached to the pole 530. Further, the mount 540 can be secured to any suitable portion of the tally bag stand 500. For example, the mount 540 can be secured to the pole 530, the mounting rod 531, or the hook 532 of the tally bag stand 500. Further, the mount 540 can have any suitable configuration. The mount 540 can be substantially straight, or the mount 540 can include a curved configuration. Further, the mount 540 can be length adjustable, angularly adjustable (e.g., adjustable in angle between the mount 540 and the pole 530), such as with a gooseneck configuration, or any suitable combination thereof. This can provide the advantage of allowing a user to adjust the position of the mount 540 according to the user's particular needs or the procedure at hand.

[0042] Examples of computational techniques for tracking surgical instruments In some exemplary embodiments, one or more computational techniques are used to identify, count, or both, surgical instruments that may have been placed in one or more tally bags (e.g., tally bag 400). It may be desirable to use one or more such computational techniques in combination with one or more tally bags to automate and streamline the process of surgical instrument location verification, and may improve the accuracy of the one or more counts and therefore improve compliance with protocols for using the one or more tally bags. For example, one or more images of tally bag 400 may be taken, and one or more computational techniques may be used to identify and count any surgical fibers within tally bag 400 (e.g., using computer vision, machine learning, or deep learning, as described elsewhere herein). For example, a user may take an image of tally bag 400 using an imaging system, such as an imaging system of mobile device 550 (e.g., a mobile computing device). FIG. 7 illustrates a schematic of a user 710 taking an image of tally bag 730 using a mobile device 720. As shown in FIG. 7, a user 710 can position an imaging system in the exemplary form of a mobile device 720 with a camera 722 such that the tally bag 730 is within the field of view of the camera 722, as shown by the dashed line in FIG. 7. One or more images generated by the camera 722 of the mobile device 720 can be analyzed using one or more computational techniques described herein. Although FIG. 7 shows the user 710 holding the mobile device 720 in front of the tally bag 730 to generate images, the imaging system (e.g., the mobile device 720) can be positioned relative to the tally bag 730 in any suitable manner, such as via a mount (e.g., mount 540 shown in FIG. 5). It can be directly on the tally bag 730, proximate to the tally bag 730, or angled toward a pocket of the tally bag 730 (e.g., pocket 410).

[0043] Tracking algorithm model example FIG. 8 illustrates a method according to some exemplary embodiments, showing example relationships and data flows between various trained models and other possible components of method 800 (e.g., a tracking algorithm) used to track surgical instruments placed in one or more tally bags (e.g., tally bag 400). Each model included in method 800 can use machine learning, deep learning, or both to achieve sub-goals. The algorithms of each model of method 800 can be combined (e.g., in a unique manner) to arrive at an accurate location of the surgical instruments in one or more tally bags and provide additional information to the user. In some exemplary embodiments, the method for tracking surgical instruments relies on two or more independent tallying approaches, as well as one or more error reduction techniques, to arrive at a robust method of tallying surgical instruments. Although method 800 (e.g., considered an overall tracking algorithm) described with reference to FIG. 8 is described with respect to tally bags, method 800 may be used to tally instruments (e.g., surgical instruments) configured or contained in any suitable manner. Furthermore, the steps of method 800 described below do not have to be performed in the order in which they are described or depicted in the flowchart shown in FIG. 8, but rather can be performed in any suitable order.

[0044] As shown in FIG. 8, the input of the method 800 is an image of one or more tally bags (810). The image of the one or more tally bags 810 may be acquired in any suitable manner using any suitable imaging device. In some exemplary embodiments, a user acquires an image of the tally bag 730 using a camera 722 on a mobile device 720 (e.g., a cell phone or tablet). However, any one or more of a variety of imaging systems, such as a digital camera, a camera on a computer, a video camera, or the like, may be used to acquire the input image. The image may then be received by a processor. The image may be preprocessed 820 for subsequent analysis. Preprocessing of the image may include, for example, brightness normalization, contrast correction, reducing the image size to dimensions suitable for processing by one or more deep learning or computer vision models, or any suitable combination thereof. Determining the tally (e.g., a first tally, a second tally, a further tally, or any suitable combination thereof) of surgical instruments within an imaged tally bag (e.g., tally bag 400 or 730) may include analyzing the image with one or more of a variety of computational techniques, such as a tally bag detection model 840 to detect the presence of a tally bag in the image, a tally bag shape model 830 to encode geometric features (e.g., dimensions and other characteristics of the tally bag and its one or more pockets within the tally bag), a surgical instrument detection model 850 to detect the presence of any instruments within the tally bag, an empty / full pocket classifier model 860 to determine the presence or absence of a surgical instrument (e.g., a general or specific type) within each pocket, or an instrument tally model 870 to synthesize information about the surgical instruments within the imaged tally bag, provide a tally of the surgical instruments within the imaged tally bag, or both. In some exemplary embodiments, after any pre-processing operations, method 800 utilizes two or more paths of instrument tally. For example, one path may be based on directly detecting the surgical instruments within the tally bag, and one path may be based on identifying pockets within the tally bag and classifying the pockets as empty or full.The method 800 may then combine the results of the multiple counting passes and use one or more error detection algorithms to arrive at a final location confirmation of the surgical instrument.

[0045] The tally bag detection model 840 may receive pre-processed or unprocessed images of one or more tally bags for analysis. The tally bag detection model 840 may identify the presence of at least one tally bag in the image based at least in part on one or more trained machine learning or deep learning models. Examples of suitable object detection models include faster region-based convolutional neural networks (R-CNN), single short detectors (SSD), and region-based fully convolutional networks (R-FCN), which can achieve high performance in object detection tasks. The tally bag detection model 840 may also utilize one or more of such trained models to identify the location of each tally bag in the image and identify or count 844 one or more edges, corners, or both, or any suitable combination thereof, of the tally bag. 9 and 10, tally bag detection model 840 may detect the presence of tally bag 910 and indicate the location of tally bag 910, one or more boundaries of tally bag 910, or both, in the displayed image (e.g., indicated by a corresponding bounding box). In some exemplary embodiments, tally bag detection model 840 may crop the tally bag image according to the identification of the tally bag (e.g., to substantially isolate the portion of the image that corresponds to the identified tally bag) and output a cropped tally bag image 842 for later use.

[0046] The tally bag shape model 830 may encode geometric information about one or more tally bags, such as the vertical and horizontal dimensions of a tally bag (e.g., tally bag 400 or 730), the location of its pockets (e.g., pocket 410), or both, to help identify particular geometric features of an identified tally bag. The tally bag dimensions 812 (e.g., with or without other geometric information) encoded within the tally bag shape model 830 may depend, for example, on the make (e.g., brand or manufacturer), model, or both of the tally bag, and may be pre-coded (e.g., manually entered as associated with a particular make or model of tally bag) or learned from training data. In some exemplary embodiments, the tally bag shape model 830 may accept one or more inputs, such as the location of a corner 814 or edge of a tally bag, with or without other suitable information that may help identify, analyze, or both, the shape of the tally bag. In some exemplary embodiments, the shape of the tally bag is identified based at least in part on knowledge of the manufacturer of the tally bag, which may be entered by a user (e.g., via a user interface of a mobile application) as user input, by scanning an identifier code (e.g., a barcode, a QR code, etc.) on the tally bag, its packaging, or by recognizing the manufacturer of the tally bag directly from an image of the tally bag (e.g., based on optical character recognition of a label, a distinctive color feature, a distinctive geometric feature, or any suitable combination thereof). For example, a user may enter the manufacturer of the tally bag, and the shape model may use the known information to help determine the geometric information of the tally bag. As another example, a user may manually identify the boundaries of the tally bag, its pockets, or both on a mobile device by applying reference markers or resizing one or more virtual contours for defining the tally bag, its pockets, or both. The tally bag shape model may automatically identify the location of the pockets of the tally using a homography or one or more other perspective transformations.Based on these locations, tally bag shape model 830 crops the image of each tally bag pocket for acceptance by empty / full pocket classifier model 860. Thus, tally bag shape model 830 can provide pocket images 832 (e.g., identification of portions of the image corresponding to tally bag pockets, cropped image segments, or any suitable combination thereof), which may also include information regarding the locations of the pockets, those pocket corners 834, or both. For example, as shown in the exemplary graphical user interface shown in FIG. 9, tally bag shape model 830 can detect the locations of tally bags, the shapes of the tally bag pockets, or both, and indicate the tally bag pockets (e.g., pockets 920A-920C) in the displayed image, with the pockets shown with corresponding bounding boxes.

[0047] As noted above, there are multiple exemplary embodiments (e.g., variations) of the method 800 for tracking (e.g., by counting) surgical instruments. In some exemplary embodiments, the surgical instrument detection model 850 uses a deep learning algorithm or other suitable artificial intelligence model to identify each surgical instrument (e.g., each surgical fiber) and the location of the surgical instrument within the cropped tally bag image 842 as generated by the tally bag detection model 840. The deep learning algorithm used in the surgical instrument detection model 850 may be trained on a dataset (e.g., a reference dataset or other training dataset) representing a diverse array of tally bag images, each labeled with the identity of each surgical instrument in the image. Thus, the deep learning algorithm of the surgical instrument detection model 850 may identify, locate, or both, the surgical instruments (e.g., surgical fibers) within each tally bag in the image. The surgical instrument detection model 850 may generate data regarding the boundaries of each surgical instrument within the tally bag. In the case of surgical fibers, this may be performed, for example, by identifying corners 852 of the fiber. Thus, the surgical instrument detection model 850 can provide the locations of all detected surgical instruments within one or more tally bags and enable a determination of a tally (e.g., a first tally, a second tally, a further tally, or any suitable combination thereof) of the surgical instruments within the one or more tally bags.

[0048] In a particular exemplary embodiment, the empty / full pocket classifier model 860 identifies whether each pocket is full or empty by analyzing images of one or more tally bags where the pockets have been identified by the tally bag shape model 830. The cropped images of each pocket generated by the tally bag shape model 830 are received by the empty / full pocket classifier model 860. The empty / full pocket classifier model 860 may use a deep learning algorithm trained on a dataset of tally bag images labeled with empty / full pocket identifiers to identify which pockets are empty and which pockets are full. Examples of suitable object classification networks include Inception, ResNet, MobileNet, MnasNet, and EfficientNet, which may achieve high performance in object classification tasks. The output 862 of the empty / full pocket classifier model 860 includes information indicating which pockets are empty and which pockets are full in one or more tally bags. Thus, the empty / full pocket classifier model 860 allows for the determination of the tally of surgical instruments in one or more tally bags (e.g., a first tally, a second tally, a further tally, or any suitable combination thereof). FIGS. 9 and 10 show example displays corresponding to the output 862 of the empty / full pocket classifier model 860. Based on a combination of information (e.g., from the surgical instrument detection model 850, the empty / full pocket classifier model 860, or both), a tally bag may be identified as a full tally bag (e.g., as shown in FIGS. 9 and 10), a partially full tally bag, or an empty tally bag. Additionally, as shown in FIGS. 9 and 10, each pocket in the image of the tally bag may be individually identified as containing a detected surgical instrument, such as a surgical sponge (e.g., pockets 920A and 920B), or as not containing a detected surgical instrument (e.g., pocket 920C). The instrument tally model 870 synthesizes the information generated by one or more of the aforementioned models to generate various information about each image analyzed.For example, the instrument tally model 870 may utilize prior information (e.g., determined tally bag shape, surgical instrument detection, or both) and perform post-processing to arrive at or provide information regarding one or more surgical instruments (such as surgical fabrics), one or more tally bags, or both, such as a final number of surgical instruments, location of tally bags, location of one or more surgical instruments, location of one or more empty pockets, and location of one or more full pockets. As shown in FIG. 8 , the instrument tally model 870 may determine a total instrument tally within each tally bag 872 in the image. For example, the instrument tally model 870 may utilize information from one or more of the aforementioned models, such as the surgical instrument detection model 850, the empty / full pocket classifier model 860, or both, to determine the number of surgical instruments per tally bag 872. Additionally, in some exemplary embodiments, the instrument tally model 870 may combine information from one or more of the other models described above in various ways to determine an instrument tally using a combination of multiple methods and compare the tally arrived at using each method (e.g., verifying the counts with each other) to determine a final surgical instrument tally.

[0049] For example, the instrument tally model 870 may compare the first estimate of surgical instruments generated by the surgical instrument detection model 850, as described above, with the second estimate of surgical instruments generated by the empty / full pocket classifier model 860, as described above, to determine a surgical instrument tally (e.g., a first tally of surgical instruments or a second tally of surgical instruments). In other words, the instrument tally model 870 may compare the positions of the detected surgical instruments in the tally bag obtained using two different types of deep learning models (e.g., by comparing the results of object detection by the surgical instrument detection model 850 with the results of object classification by the empty / full pocket classifier model 860). In some exemplary embodiments, the instrument tally model 870 receives information about the surgical instruments (e.g., surgical fibers) identified and detected by the surgical instrument detection model 850 to determine the first estimate of the surgical instrument tally. The instrument tally model 870 may use the information generated from the empty / full pocket classifier model 860 to generate the second estimate of the surgical instrument tally. If the surgical instrument tally generated using both methods matches, then the instrument tally model 870 can use the matching tally as output in providing a tally of instruments per tally bag 872. If the two tallys do not match, then the instrument tally model 870 can respond by using a unique and novel approach of combining the outputs of the surgical instrument detection model 850 and the empty / full pocket classifier model 860, which may include utilizing knowledge of the shape of the tally bag to modify the tally and improve the accuracy of tracking (e.g., counting), such as the approaches described below.

[0050] The instrument count model 870 may also perform one or more methods of validating one or more determinations of one or more instrument count estimates (e.g., if the counts based on the surgical instrument detection model 850 and the empty / full pocket classifier model 860 do not match). For example, the instrument count model 870 may take the total number of pockets identified or counted by the count bag shape model 830 and the total number of full or empty pockets identified or counted by the empty / full pocket classifier model 860 and determine how many surgical instruments (e.g., surgical fabrics) are in the pockets of the count bag (e.g., based on an assumption that there is only one surgical instrument in each full pocket). For example, if there are ten pockets identified by the count bag shape model 830 and one pocket is labeled as empty or nine pockets are labeled as full, then the instrument count model 870 may use this information to determine that the surgical instrument count is nine.

[0051] As shown in FIG. 8, additional useful information may be provided according to method 800 based on information obtained from one or more of the various models in addition to determining one or more tally of the surgical instruments (e.g., a first or second tally of surgical fibers in a tally bag). For example, the instrument tally model 870 may provide a tally bag location 874 in the image, which may be useful for automatically highlighting all tally bags for visualization. As another example, the instrument tally model 870 may provide the location of the identified or tally of the surgical instruments in the tally bag, such as an indication of which pockets of the tally bag contain the surgical instruments. This information may be used in one or more error correction methods, for example, as described below. This information may also be useful in determining compliance with an instrument tally protocol (e.g., a protocol for tallying surgical fibers). For example, if there is surgical fiber in the top pocket of the tally bag, but the bottom pocket of the tally bag is empty, this may indicate that the user did not follow an approved tally protocol that requires filling the pockets of the tally bag from bottom to top. As another example of useful information, the instrument count model 870 may provide the location of empty and full pockets within an image of the count bag, which may also be useful in determining compliance with counting protocols and may serve as validation of surgical instrument location or number of surgical instruments, highlighting the location of empty pockets within the count bag for easy evaluation by the end user, or both.

[0052] Example of combining models to reduce aggregation errors Method 800 (e.g., considered as an overall tracking algorithm for tracking surgical instruments) may employ a unique and novel approach to improve tracking (e.g., tally) accuracy by combining multiple (e.g., several) models (e.g., deep learning models), such as surgical instrument detection model 850 and empty / full pocket classifier model 860, and utilizing knowledge of tally bag shape (e.g., from tally bag shape model 830) to remove or mitigate various types of errors. For illustrative purposes, two example methods of combining multiple models are described below.

[0053] In one example of combining multiple models to improve overall tracking accuracy, different models can be combined to correct or address common errors in instrument counting models. For example, a first type of error occurs when there is a false detection of the presence of a surgical instrument (i.e., failure to detect the absence of a surgical instrument), leading to reduced accuracy and incorrect counting of surgical instruments. A second type of error occurs when there is a false detection of the absence of a surgical instrument (i.e., failure to detect the presence of a surgical instrument), leading to reduced recall and incorrect counting of surgical instruments.

[0054] In some exemplary embodiments of method 800, one or both of these types of errors are addressed by algorithmic matching (e.g., algorithmic error correction). For example, erroneous detection of the presence or absence of a surgical instrument may be addressed, for example, based at least in part on an assumption that the location of the detected surgical instrument should closely match the location of the detected pocket of the tally bag. Thus, one or more error correction models may identify and reject "identified" surgical instruments whose determined location does not match the determined location of the tally bag pocket. For example, the error correction model may reject an identified surgical instrument based on a Euclidean distance between the center (e.g., center of gravity) of the detected surgical instrument and the center of the closest tally bag pocket obtained from the shape of the tally bag. That is, if the Euclidean distance between the center of the detected surgical instrument and the center of the closest tally bag pocket exceeds a predetermined threshold, the error correction model may reject the identified surgical instrument as an error (e.g., as misidentified) and thus not include the identification in the tally of surgical instruments. Correspondence between the remaining (non-rejected) identified surgical instruments and the tally bag pockets may be established based on the shortest Euclidean distance. That is, the error correction model may match surgical instruments with particular pockets by determining the identified pocket that is closest to each instrument (e.g., to verify that each surgical instrument corresponds to a pocket).

[0055] Once a correspondence is established, a comparison can be performed between the instrument detection results and the empty / full pocket classification results. For example, if the surgical instrument detection model 850 fails to identify an actually present surgical instrument (an instance of the second type of error described above), this error can be corrected (e.g., by an error correction model) because the empty / full pocket classifier model 860 identifies the corresponding pocket as full. Similarly, if the surgical instrument detection model 850 erroneously identifies a surgical instrument in a pocket that was determined to be empty based on the empty / full classifier model 860, this instance of detection can be rejected (e.g., by an error correction model).

[0056] One advantage of this algorithmic matching approach is that it requires less training data to implement and improves performance in tracking surgical instruments compared to using a single model alone (e.g., using only the surgical instrument detection model 850 or only the empty / full pocket classifier model 860).

[0057] In other exemplary embodiments, multiple models can be combined to form a tracking ensemble network with the goal of improving overall tracking accuracy and reducing aggregate error. For example, in addition to using one or more training techniques as described in more detail below, an ensemble of surgical instrument detection networks can be trained with one or more suitable ensemble learning techniques (e.g., snapshot ensemble).

[0058] As an illustrative example, a penalty term can be added to the training of the surgical instrument detection model 850. Specifically, a penalty term can be added to modify the loss function of the surgical instrument detection model 850 by automatically penalizing the detection of a surgical instrument whose detected location is not close (e.g., within a threshold distance) to the corresponding (e.g., closest) pocket location. The penalty term can be related, for example, to the distance between a corner of the detected surgical instrument (e.g., surgical fabric) and a corner of the tally bag pocket (e.g., determined using the tally bag shape model 830). Adding this penalty term directly to the training of the surgical instrument detection model 850 incorporates prior knowledge about the shape of the tally bag (e.g., a sponge bag or other tally bag), thus effectively restricting the output of the surgical instrument detection model 850 to the vicinity of the location of the tally bag's pocket (e.g., a pocket grid). Furthermore, end-to-end training of the surgical instrument detection model 850 can be performed to automatically take into account geometric information of the tally bag that was available during the training phase.

[0059] Further, in this example, an ensemble of multiple instrument detection networks of different types can be trained by varying a training dataset (e.g., a dataset described further below), such as randomly or in a K-fold manner, varying one or more hyperparameters of the networks (e.g., hyperparameters such as depth, regularization, penalty weight, or a suitable combination thereof), and performing an ensemble learning technique, such as snapshot ensemble. Similarly, an ensemble of multiple empty / full pocket classification networks of different types can be trained by varying the network architecture, varying the hyperparameters of the networks (e.g., as described above), or any suitable combination thereof.

[0060] Finally, the results from all the different instrument detection networks and all the empty / full pocket classification networks may be used within a new stacked generalization network that may be referred to as a tracking ensemble network. The tracking ensemble network may be trained separately on a separate set of validation data (e.g., a "hold-out" validation set) to learn how to optimally combine predictions from each ensemble member. This separate training may help improve the overall accuracy of the tracking (e.g., tally), as well as the accuracy of each surgical instrument prediction, each empty / full pocket prediction, or any suitable combination thereof. For example, the tracking ensemble network may evaluate a 10-dimensional occupancy vector (e.g., for a 10-pocket tally bag, where "l" indicates a full pocket and "O" indicates an empty pocket) and accordingly calculate a loss function based on the dot product of the predicted occupancy vector (e.g., from each of the competing ensemble instrument detection networks and empty / full pocket classification networks) and the ground truth occupancy vector. Thus, once trained, the tracking ensemble network can weigh the predictions from each of the ensemble object detection and object classification networks to improve the overall accuracy (e.g., overall tracking algorithm) of method 800. Thus, the grid penalty and ensemble approach described above can utilize more training data to implement, which can further improve the performance of method 800.

[0061] Training data collection example Successful training of one or more of the above models may be aided by collecting a representative dataset of training examples and appropriately labeling the instruments (e.g., surgical instruments) in the dataset for the desired end goal. One or more of such datasets may, for example, be representative and include variants of tally bag images expected to be seen in an operating room or other setting in which the trained model will be used. Thus, the datasets used to train a model, such as one or more of the models of method 800 (e.g., a tracking algorithm) above, may include "good" and "bad" images, as described below.

[0062] In some exemplary embodiments of model training described herein, a good image may feature a tally bag from a complete front view (e.g., not a side view, top view, or angled view) that is correctly oriented vertically, with no part of the bag blocked by other objects. A good image can be focused on with sufficient resolution to read the text on the bag. Some imaging devices, such as mobile computing devices, may automatically take images with sufficient resolution by default. A tally bag in a good image has intact pocket dividers (e.g., the seams dividing the pockets are not torn), and each pocket may contain zero or one surgical instrument. One or more models of method 800 may be trained with a set of good images, including images that depict expected deformations during the course of various medical procedures. For example, a training dataset may include one or more deformations of the tally bag, the image background or setting, or both, as shown in Table 1 below. [Table 1]

[0063] In some exemplary embodiments of model training described herein, bad images may include images that are generally difficult to analyze (e.g., due to poor lighting, blurry images, or both), such as images in which features of the tally bag or surgical instruments are difficult to detect. Because bad images are likely to be submitted by users during practice, it may be useful to train one or more models on variants of bad images to create one or more robust tally models that are resistant to imperfect images of the tally bag. These bad images may also be used to help create "guardrail" algorithms that may warn users of incorrect usage of the tally bag, and may alert users when a bad image is taken and notify the user to provide a better image (e.g., a new image that is clearer). Examples of bad images include images where a portion of the bag is obstructed by a foreground object (e.g., a user's hand, a handrail, a tube, or any suitable combination thereof), images taken from an angle that makes it difficult to see a pocket divider or pocket seal (e.g., between two or more surgical instruments), images of a tally bag with a torn pocket divider or torn pocket seal (e.g., where such a tear has created one large pocket where there should have been two separate pockets), images with multiple surgical instruments in one pocket, images taken in incorrect lighting conditions or other conditions that may produce an image that is difficult for an algorithm to analyze and extract features from, and images having a combination of any one or more of the above aspects.

[0064] Other examples of validating aggregates In some exemplary embodiments, the methods described herein may include one or more additional verification stages to provide further checks (e.g., independent checks) on method 800. Providing one or more additional check methods may increase the accuracy of the location verification of the surgical instruments. Such additional methods may include prompting the user to manually verify the tally of the surgical instruments in the tally bag. Such additional methods may include requesting user verification of information generated by one or more of the various submodels of method 800. For example, one or more of such additional methods may include asking the user to verify that method 800 correctly identified the pocket, correctly identified the tally bag, correctly identified the full pocket, correctly identified the surgical fabric in the pocket, or any suitable combination thereof. For example, FIG. 11 illustrates an exemplary embodiment of a display on a mobile device, where the display prompts the user to verify that the number of surgical instruments (e.g., surgical sponges) identified and tallied by method 800 is correct. As indicated by the box surrounding the surgical instrument in FIG. 11, one or more of such additional methods may include displaying information to the user indicating what the method 800 identified as the surgical instrument.

[0065] In certain exemplary embodiments, the methods described herein may include using the weight information to verify the tally of the surgical instruments. For example, the pole to which the tally bag is attached may be coupled to a scale. The scale coupled to the pole may sense a change in weight each time a surgical fiber is placed in the tally bag and thus tally the surgical instruments based on the number of weight increments sensed by the scale. The scale may be coupled to the pole in any suitable manner. For example, the mounting pole, the tally bag, or both may be located on the scale (e.g., the mounting pole, the tally bag, or both may be located on top of the scale to measure the change in weight when an instrument is added to the tally bag). Alternatively, the scale may take the exemplary form of a strain gauge coupled to the mounting pole and coupled to the tally bag (e.g., one end of the strain gauge is suspended from the mounting pole and the tally bag is suspended from the other end of the strain gauge). In some exemplary embodiments, the weight information may be generated from one or more surgical instrument collection containers instead of or in addition to weight information generated by one or more other methods. For example, the collection container may include a weight measurement system. When a user removes a surgical instrument from the collection container, the weight measurement system senses a decrease in weight and determines that the fibers of the surgical instrument have been removed from the collection container. The processor can record the number of weight decreases to determine a tally of surgical instruments removed from the collection container, and thus a tally of surgical instruments previously present in the collection container. Thus, the use of one or more collection containers can provide an additional check on the tally of surgical instruments (e.g., surgical fibers).

[0066] In various exemplary embodiments, the methods described herein may include using time data from removal of the surgical instrument from the collection container and placement of the surgical instrument in the tally bag to provide additional verification of the surgical instrument tally. For example, the processor may generate a timestamp each time the weighing system senses removal of the surgical instrument from the collection container. Additionally, the processor may generate a timestamp each time the weighing system senses addition of the surgical instrument to the tally bag. The one or more processors may be configured to compare the timestamp of removal of the surgical instrument from the collection container (e.g., removal timestamp) and the timestamp of addition of the surgical instrument to the tally bag (e.g., add timestamp) to provide verification of the tally of the surgical instrument (e.g., additional verification). If the removal timestamp and the add timestamp are substantially consistent, it provides information indicating that the tally of the surgical fiber is correct.

[0067] Example of determining interim totals In some exemplary embodiments, determining the second tally includes determining an intermediate tally of the surgical instruments during the course of the medical procedure (e.g., a surgical procedure). In an alternative exemplary embodiment, determining the second tally includes determining a final tally of the surgical instruments after the medical procedure is completed. For example, a method discussed herein for tracking surgical instruments (e.g., method 800) may include comparing the second tally (e.g., as an intermediate or final tally) to the first tally to ascertain the whereabouts of the surgical instruments at any point during the course of the medical procedure. Various exemplary embodiments of such methods may include the use of any one or more of the above algorithms to determine the second tally, whether it is an intermediate or final tally.

[0068] Determining the intermediate tally of the surgical instruments may include updating an index counter of the surgical instruments. In some exemplary embodiments, updating the index counter includes analyzing an image of a tally bag that includes one or more surgical instruments (e.g., surgical fibers) in one or more pockets of the tally bag. As the tally bag is filled with surgical fibers by a user, one or more images of the tally bag may be generated. One or more processors may receive the one or more images of the tally bag and may process the one or more images using one or more computational techniques, such as one or more techniques described above, to identify and tally the surgical fibers in the one or more tally bags.

[0069] In some exemplary embodiments, instead of or in addition to analyzing an image of a tally bag containing a surgical instrument, the methods described herein include holding a surgical instrument within the field of view of a camera. For example, a user can hold a surgical instrument (e.g., a surgical fabric such as a surgical sponge) up to a camera, and the camera can generate an image depicting the surgical instrument. The image can be received and analyzed by one or more processors configured to detect the presence of the surgical instrument depicted in the image. The image of the surgical instrument can be used to track the surgical instrument. For example, an index counter can be updated each time a surgical instrument is held up to a camera and identified by one or more processors. In some exemplary embodiments, an index counter updated using an image analysis of a surgical instrument held up to a camera can be compared (e.g., as described above) to further index counters updated using an image analysis of surgical instruments in one or more tally bags to verify the index count of the surgical instrument. FIG. 13 illustrates an example of a graphical user interface in which images of individual surgical instruments (e.g., surgical fabric) are acquired and analyzed to update a tally of such surgical instruments. Specifically, a user 1210 holds a surgical fiber 1220 in front of a camera that captures an image (e.g., a depth image, a color image, or both) such that one or more processing components (e.g., a processor) may identify the surgical fiber 1220 in the image and include the imaged surgical fiber 1220 in the instrument tally 1230. Exemplary systems and methods for accomplishing the above are described in further detail in U.S. Pat. No. 8,897,523 and U.S. Patent Publication No. 2017 / 0186160, each of which is incorporated herein by this reference in its entirety. Additionally, as described below, one or more methods for estimating blood loss may be used in combination with one or more methods described herein for tracking surgical instruments, and thus may include a user holding one or more surgical instruments in front of a camera for imaging.

[0070] Determining the intermediate tally of surgical instruments may further include updating an index counter with the number of surgical fibers in one or more containers (e.g., one or more tally bags) as the containers are filled and imaged. This process may be repeated throughout the medical procedure to continually update the index counter and maintain a running tally of surgical instruments (e.g., surgical fibers) loaded into one or more containers at any one time. In some exemplary embodiments, it may be desirable to compare the intermediate tally (e.g., the current value of the index counter) to a first tally of surgical instruments. For example, a user may want to know how many surgical instruments are loaded and located in a set of tally bags, and how many surgical instruments are currently in use or not located.

[0071] Certain exemplary embodiments of the methods described herein include displaying or otherwise providing an intermediate tally to a user at any appropriate point during a medical procedure. Displaying the intermediate tally may provide the user with useful information regarding how many surgical fabrics have been located at a particular point in time. FIG. 14 illustrates an example of a graphical user interface 1300 that displays information regarding the intermediate tally. Box 1310 of the screen of FIG. 14 illustrates the total number of surgical sponges located (e.g., "17") compared to the first tally of sponges (e.g., "40"). Additionally, information related to the type of surgical sponges located (e.g., brand, size, function type, etc.) is displayed in boxes 1320, 1330, and 1340. The graphical user interface 1300 also provides functionality that allows the user to update the first tally by selecting the "Counted" icon 1350 and determine the final tally by selecting the final tally icon 1360.

[0072] Example of duplicate image detection Some exemplary embodiments of the methods described herein may further include determining whether a particular tally bag has been previously imaged and analyzed. Because a user may take images of multiple tally bags, multiple surgical instruments, or both, an individual tally bag, an individual surgical instrument, or both may be inadvertently imaged multiple times, leading to an inaccurate tally of surgical instruments. To combat these consequences of inadvertent duplicate images, one or more images may be analyzed to determine a measure of the likelihood that an image depicts the same tally bag as another image. An example method for determining whether a tally bag has already been located may include measuring or otherwise determining the likelihood that a first image and a second image each have image regions depicting at least a portion of the same fiber (e.g., within the same tally bag). Exemplary methods for measuring the likelihood that a first image and a second image each have image regions depicting at least a portion of the same fiber are described in detail in PCT Application No. PCT / US2017 / 068234, which is incorporated herein by reference in its entirety. An exemplary method for determining whether an image of a particular tally bag has already been received and analyzed may include receiving a first image in which a first image region depicts a tally bag, receiving a second image in which a second image region depicts a tally bag, and defining at least one classification feature based at least in part on a first aspect of the first image, a second aspect of the second image, or both. Such an exemplary method may further include measuring a likelihood that the first and second image regions depict at least a portion of the same tally bag, the likelihood measurement being based at least in part on the classification feature, such as by using a machine learning classification algorithm. Such an exemplary method may include comparing the measured likelihood of overlapping images to a predetermined threshold, the threshold being or including a cutoff for classifying the image pair as "non-overlapping" or "potentially overlapping." A similar process may be performed to measure the likelihood that the first image region and the second image region depict at least a portion of the same surgical fiber or other surgical instrument.

[0073] If the method 800 determines that the likelihood of duplicate imaging exceeds a predetermined threshold, a follow-up action may be prompted. For example, while generally each new image of a tally bag or surgical instrument obtained may correspond to an increase in the tally of the surgical instrument (e.g., a surgical instrument extracted from a patient or a sterile field of a surgical procedure for a second tally), the tally may be adjusted based at least in part on the measured likelihood that the tally bag, the surgical instrument, or both have been imaged multiple times. Detection of a potential duplicate image of a surgical tally bag or surgical instrument may, for example, trigger a prompt to a user to confirm whether the tally bag, the surgical instrument, or both have been imaged multiple times. As another example, detection of a potential duplicate image of a tally bag or surgical instrument may automatically refrain from increasing the tally of the surgical instrument. An exemplary method of determining whether a tally bag has already been imaged may include the use of an identification tag. For example, the tally bag, the surgical instrument, or both may include a scannable tag (e.g., an RFID tag or a barcode), and a scanner may be used to determine whether the tally bag has already been imaged. The scanner may be a separate scanning device or may be integrated into the mobile device. One or more exemplary methods including scanning tags or bar codes may be used in place of or in addition to one or more techniques including image analysis.

[0074] Example of determining final tally The second tally may include a final tally of surgical instruments in some exemplary embodiments. For example, exemplary embodiments of the methods described herein may include determining a final tally of surgical instruments and comparing the final tally to a first tally of surgical instruments. The final tally may be distinguished from an intermediate tally, for example, based on a final location check that references a tally of surgical instruments that is located at a point in the procedure where the surgical instruments are no longer being introduced or used. For example, the final tally may be performed at the end of the medical procedure. The end of the medical procedure may be indicated in any suitable manner. For example, the end of the medical procedure may be after closing the surgical site, after all surgical procedures have been performed on the patient except for closing the surgical site, when the patient is ready to be taken out of the operating room, or any other time point suitable for use as a final indication. In some exemplary embodiments, the determination of the final tally may be prompted by the user. For example, the mobile application or other computing program may allow the user to indicate when the final tally is to be indicated (e.g., selection of the final tally icon 1360 of FIG. 14). In some exemplary embodiments, the mobile application may prompt the user to verify that the tally of surgical instruments should be labeled as a final tally based on a specific instruction. For example, when the index counter reaches the value of a first tally, the mobile application may prompt the user to confirm whether the current value of the index counter should be indicated as a final tally. In other exemplary embodiments, the mobile application may request the user to indicate whether the index counter should be indicated as a final tally after a specific time has elapsed.

[0075] Example of providing notifications based on the first aggregate, the second aggregate, or both Various exemplary embodiments of methods described herein for tracking surgical instruments may include comparing a first tally of surgical instruments to a second tally of surgical instruments. For example, some exemplary embodiments of such methods for tracking surgical instruments include comparing an intermediate tally (e.g., between one or more intermediate tally) to the first tally, as described above. Certain exemplary embodiments of such methods include comparing the final tally to the first tally to determine whether the first and final tally match, and therefore, whether all surgical instruments introduced into the operating room have been accounted for. Comparing the final tally to the first tally allows medical staff to verify that all surgical textiles or other surgical instruments have been accounted for and are not being held by or within the patient. Some exemplary embodiments of such methods may include notifying a user whether the first tally matches the second tally (e.g., the final tally or the intermediate tally). For example, certain exemplary embodiments of such methods include displaying a summary report of surgical instrument aggregate measurements, in which a first aggregate, a second aggregate, and an indication of whether the first aggregate matches the second aggregate are displayed to one or more users.

[0076] FIG. 15 illustrates an example of a graphical user interface displaying a summary of surgical fiber data. The screen of FIG. 15 shows the number of fibers that have been "counted" (e.g., introduced into the operating room), the number of "off-scan" fibers (e.g., imaged into a tally bag), and the number of unused fibers (e.g., introduced into the operating room but not yet imaged into a tally bag). In some implementations, unused surgical fibers are placed into one or more tally bags in the same manner as used surgical fibers. In some other implementations, unused surgical fibers are located using a different method, such as being identified as unused in one or more tally bags and labeled as such on a display screen accordingly. In some exemplary embodiments, the methods described herein include notifying the user that the compared tally matches (e.g., by providing a notification to the user). Such a notification may include a short message indicating to the user that the tally matches, or the notification may be a symbol. For example, the notification may include displaying a check mark or equal sign or other suitable symbol next to the first and final tallies, displaying a color indicator, such as a screen that turns a particular color (e.g., green) when the tallies match, and playing an audio indication that the first and final tallies match, some form of flashing or vibration, or a suitable combination thereof, if the final tallies match. Additionally, as described in more detail below with respect to FIG. 16, a summary report may be provided to one or more users for personal reference, hospital records (e.g., for accountability purposes), or both, and the summary report may show the first and final tallies for each of one or more types of surgical instruments. For example, the summary may include or show a representation of a comparison between the first and second tallies, such as the "match" numbers shown in FIG. 15.

[0077] In certain exemplary embodiments, the methods described herein include providing a notification to the user that the compared tally does not match. In the situation where the final tally of the surgical instruments does not match the first tally of the surgical instruments, the notification may include a warning to the user. The warning may be in the form of an example display (e.g., on a mobile device) indicating to the user that there are one or more surgical instruments that have not been located. The display may include any one or more of the elements described above in connection with the notification that the compared tally matches, except in aspects inconsistent with indicating that the compared tally does not match. In some exemplary embodiments, notifying the user that the first and final tally of the surgical instruments do not match includes providing a predetermined protocol for proceeding in the event of a mismatched location. For example, the notification to the user may include displaying (e.g., on a screen of a mobile device) the steps of the predetermined protocol and providing both an accompanying audio cue on how to proceed if the first and final tally of the surgical instruments do not match. While medical personnel may be familiar with one or more protocols regarding how to proceed in the case of a mislocated surgical instrument, providing additional presence (e.g., additional instances of a protocol) that is not subject to the potential anxiety of a mislocated surgical instrument may be useful in helping users comply with the protocol. This may lead to fewer instances of retained surgical instruments in the patient's body, better detection of retained surgical instruments when they occur, more effective responses in the case of a retained surgical instrument, or any suitable combination thereof. If the first and final tallies do not match, some exemplary embodiments of the methods described herein may include automatically sending a notification to alert additional personnel, triggering a response protocol, or both. For example, a mismatched first and final tally may trigger a notification to the hospital's administrative staff, a radiologist (e.g., to obtain pre-approval to produce an x-ray for the patient), an x-ray technician, additional medical personnel to remedy the situation, or other appropriate personnel.

[0078] If the first tally and the final tally do not match, some exemplary embodiments of the methods discussed herein for verifying the location of the surgical instruments may include executing a predetermined protocol to determine if there are any retained surgical instruments (e.g., within the patient or held by the patient). Execution of the predetermined protocol may be performed with the aid of one or more computational tools, such as a device and display to guide the user through the protocol. The protocol may first include notifying all medical personnel in the operating room of any possibly retained surgical instruments. Next, the protocol may include recounting all surgical instruments in the operating room and checking the room for any surgical instruments that may have been left behind. Recounting the surgical instruments may include operating one or more computational tally systems using one or more manual tally methods, or any suitable combination thereof. Examples of manual tally methods include recounting the number of tally bags, recounting the number of surgical instruments in each tally bag, checking each pocket to determine if there are multiple surgical instruments in the pocket, or any suitable combination thereof. Checking the operating room for surgical instruments may include checking for trash (e.g., checking one or more trash cans), checking one or more collection containers, checking the patient's surroundings, checking one or more tally bags for instances of multiple instruments placed in the same pocket, or any suitable combination thereof. If the first and final tally do not match after recounting the surgical instruments and checking for misplaced surgical instruments, the protocol may include taking an x-ray or other internal image of the patient to detect the presence of surgical instruments held within the patient. Taking an x-ray image may include alerting a radiologist to the need to take an x-ray image, and may require approval from the radiologist to proceed. Taking an x-ray image may additionally or alternatively include notifying other appropriate medical staff. The protocol may also include notifying hospital administrators of the mismatched surgical instrument tally of other administrative personnel.

[0079] Additional method examples The computational systems and methods (e.g., algorithms) described herein have broad applicability for affecting various improvements in medical procedures. Some exemplary embodiments of such methods may include tracking user compliance, guiding users through aggregation protocols, tracking surgical instrument usage throughout a procedure, receiving patient and / or operating room information, triggering additional actions upon receiving specific instructions, or any suitable combination thereof, as described in detail below.

[0080] Certain exemplary embodiments of the methods described herein include tracking compliance with one or more protocols. Tracking user compliance may include tracking user compliance with a fiber tally protocol for using tally bags. As described above, the one or more predetermined procedures may govern one or more desired ways of placing instruments in the one or more tally bags. In accordance with the applicable one or more protocols, operating room personnel may be instructed to fill the pockets of each tally bag from bottom to top and right to left. In some circumstances, one or more algorithms developed in accordance with the one or more methods described above may be used to assess whether a user is complying with a bag filling protocol. Such an algorithm may receive one or more input images and determine compliance based on the one or more images. For example, a series of images taken over time may be analyzed to determine in which order the tally bags were filled. In other circumstances, video analysis may be used to identify and track the order in which the tally bags were filled. The algorithm may analyze a time series of images or videos to determine the order in which the user filled the pockets of the tally bag to assess the user's compliance with the applicable one or more protocols.

[0081] Some exemplary embodiments of the methods described herein include providing notifications upon detecting non-compliance with a tallying protocol. For example, such methods may include displaying an alert to a user as a reminder of the proper protocol, sending a notification to hospital administrative staff, other hospital personnel, or both, indicating misuse of the fiber tallying procedure, or any suitable combination thereof. As another example, such methods may include triggering an alert upon other medical personnel intervening in an incorrect filling procedure. For example, upon detecting a misplaced surgical instrument (e.g., a surgical instrument placed in a pocket of a tallying bag contrary to the prescribed procedure of the applicable protocol), such methods may notify a supervisor or coworker to intervene in the bag filling process.

[0082] Certain exemplary embodiments of the methods described herein include prompting a user to enter their name in conjunction with the use of a counting or tracking algorithm. The mobile application may prompt the user to enter their name or other unique identifier before or after operating the application to take an image of the counting bag for processing. For example, the mobile application may prompt the user to scan their ID badge to provide such identification information. Compliance data may be combined with the identification information to enhance user accountability. Combining compliance information with user identification information may provide an advantage to hospital administrators to implement training or disciplinary protocols for users who do not follow proper instrument counting procedures.

[0083] Various exemplary embodiments of the methods described herein include using a tally bag to guide a user through a tally protocol (e.g., a fiber tally protocol). One or more computational techniques can be used to indicate to the user which pocket of the tally bag to fill next or identify a pocket the user forgot to fill, for example, by presenting one or more visual cues, one or more audio cues, or any suitable combination thereof. For example, a display screen may provide a visual indication that directs one or more users to the next pocket to be filled (e.g., providing a box or circle around the pocket, displaying an arrow as a directional guide, or both). Thus, such a method can guide a user through a protocol for filling a tally bag.

[0084] Some exemplary embodiments of the methods described herein include providing one or more other types of tracking metrics for surgical instruments. Such methods can be used to gather information about a medical procedure at various points throughout the medical procedure. For example, such methods can include tracking used and unused surgical instruments. Tracking used and unused instruments can include a user manually separating or sorting the instruments based on whether they have been used. For example, a user can place used instruments in one container and unused instruments in another container. Various methods, such as one or more of the computational techniques described above, can be used to identify used and unused containers and tally used and unused instruments. Such methods can include taking images of a portion of the operating room with used and unused containers and can extract information from the images using one or more of the image analysis methods described herein. Identifying and tallying used and unused instruments during a medical procedure can provide one or more useful metrics about the medical procedure. A single snapshot at a particular point in time of a medical procedure (e.g., at the end of the medical procedure) can provide information about the number of instruments removed from packaging and the number of instruments actually used. A series of images taken throughout a medical procedure can be used to identify and count used and unused instruments, providing information about how instruments are used over time in that medical procedure.

[0085] Certain exemplary embodiments of the methods described herein include receiving information related to a medical procedure. Such methods may enable a user to input patient information (e.g., patient data), such as patient demographics or characteristics (e.g., height, weight, sex, age, etc.), the medical procedure to be performed, the expected length of the medical procedure, whether the medical procedure is scheduled or emergency, one or more pre-existing conditions, other relevant information, or any suitable combination thereof. Such methods may include receiving, displaying, or both, various information regarding one or more environmental parameters, such as which operating rooms are in use, which operating rooms are available, whether particular equipment is available in a particular operating room, which medical personnel are currently working, other suitable parameters, or any suitable combination thereof.

[0086] Some exemplary embodiments of the methods described herein include receiving information that may trigger an event or protocol (e.g., one or more requests or initiation of one or more preventative measures upon determining that a relatively high risk exists for one or more surgical instruments retained in the patient). Such methods may include enabling a user to request additional instruments or equipment for use in the medical procedure. Such methods may include triggering a protocol upon receiving information regarding one or more specific indications. One or more specific patient parameters combined with one or more of the specific occurrences during the procedure may indicate a high risk of a particular event occurring. For example, a patient having a BMI within a particular range in combination with a threshold number of surgical instruments expected to be used during the medical procedure (e.g., if a first tally exceeds a predetermined tally) may indicate a high risk of a surgical instrument being retained. Thus, such methods may include triggering one or more additional protocols using the patient data in combination with the surgical instrument tally once the threshold number of surgical instruments is tallied. Such protocols may include, for example, alerting an x-ray technician to prepare appropriate equipment to take an x-ray image of the patient, providing notification to a radiologist to obtain pre-approval to take an x-ray image of the patient, or both.

[0087] Certain exemplary embodiments of the methods described herein include displaying or otherwise providing information related to a medical procedure. Such methods may include providing, displaying, or both, a summary report. The summary report may be stored, for example, as part of the patient's electronic medical record, as evidence of compliance with one or more surgical instrument tracking protocols, or both. The summary report may include any suitable information. As described above, information related to one or more summaries of surgical instruments, one or more comparisons of surgical instrument summaries, or both, may be displayed in the summary report. Additionally or alternatively, one or more images of individual surgical instruments, individual tally bags, or both may be stored as a record (e.g., in the summary report), displayed, or both. However, additional information regarding the medical procedure may also be displayed, such as type of medical procedure, length of medical procedure, patient information, physician information, hospital information, or any suitable combination thereof. Metrics of blood loss may also be displayed. For example, if analysis of surgical fibers is used to monitor blood loss in a patient, an estimate of blood loss may be displayed in the summary report. The summary report may include one or more symbolic indications indicating the results of the medical procedure. For example, the display may be a particular color where blood loss is within a given range (e.g., if blood loss is within a predetermined range defined between predetermined thresholds, the display may be green and if blood loss is not within the predetermined range, the display may be red). One or more summary reports may be generated at any appropriate point during the medical procedure. The one or more summary reports may be displayed on a digital screen (e.g., a display screen of a mobile device such as a phone or tablet), stored in memory (e.g., as part of the patient's or hospital's electronic medical record), saved for later viewing, sent digitally to the patient or medical personnel, sent to a printer and printed, or any suitable combination thereof. FIG. 16 shows an example of a summary report that includes information regarding compliance with the aggregation protocol, as well as additional information regarding the medical procedure performed.16 includes information regarding the case number, the location of the case (e.g., which operating room was used), the start and end times of the medical procedure, the duration of the medical procedure, the number of surgical instruments tallied (e.g., a surgical fabric labeled "sponge"), and the name of the person who performed the final tally of the surgical instruments. According to various exemplary embodiments, the summary report may display any suitable information regarding the medical procedure, such as any of the information described herein.

[0088] Some exemplary embodiments of the methods described herein provide one or more ways to enter an "emergency mode," in which some functionality of the method or system may be altered. For example, the fiber counting algorithm (e.g., implemented via a mobile application) may allow a user to select emergency mode, which may indicate that the counting protocol is no longer being followed. This may trigger one or more of a variety of actions, such as ceasing collection of instrument counting data, refraining from tracking user compliance with the counting protocol, alerting hospital staff of the emergency, etc.

[0089] In certain exemplary embodiments of the methods described herein, one or more images (e.g., showing one or more surgical instruments, one or more tally bags, or any suitable combination thereof) can be analyzed to estimate a patient's condition (e.g., total blood loss, blood component loss, or both). For example, as shown in FIG. 12, a computing device 1100 can acquire one or more images 1102 of a surgical fabric using an imaging system 1122 with one or more cameras 1120. The one or more images 1102 can capture and depict a field of view 1108 that includes a surgical fabric 1106a, such as a piece of surgical gauze, surgical sponge, or surgical towel. The surgical fabric 1106a can be held by a user. As shown in FIG. 12, the captured images, including an image 1106b of the surgical fabric 1106a, can be displayed on a display 1118 of the computing device 1100, or the like. The one or more images 1102 can include a color image that can be analyzed to estimate blood content, blood component content, or both in the imaged surgical fabric. For example, as described in the above-incorporated U.S. Patent No. 8,897,523, color-related features (e.g., intensity) of the color image may be converted, either locally or by a remote computer system for such processing, into an estimate of the amount of blood or blood components (e.g., hemoglobin mass, whole blood volume, or both) within the surgical fabric 1106a.

[0090] Additionally, one or more images 1102 may include a depth image (e.g., including depth information based on infrared data, stereoscopic images, or both). Depth information from such a depth image may more easily distinguish objects in the foreground from other objects in the background. The depth image may be analyzed (e.g., as described in U.S. Patent Publication No. 2017 / 0186160, incorporated above) to confirm the presence of surgical fibers 1106a in the image, after which the computing device 1100 may index a local surgical fiber counter. The local surgical fiber counter may be used to verify a surgical instrument tally determined by any one or more of the above methods (e.g., using image analysis of a tally bag) or as an independent method of tallying surgical instruments.

[0091] Example of a system for tracking surgical instruments Also described herein are exemplary systems for tracking surgical instruments. Such systems for tracking surgical instruments may include one or more components configured to determine a first tally of the surgical instruments. For example, such systems may include one or more imaging systems, one or more weighing systems (e.g., scales), or any suitable combination thereof, used to determine the first tally of the surgical instruments. Such systems may include one or more components configured to determine a second tally of the surgical instruments. For example, such systems may include one or more tally bags with one or more pockets each configured to contain one or more surgical instruments. Such systems may include additional equipment used to contain and process the one or more surgical instruments. For example, such systems may include one or more collection containers for holding used surgical instruments, one or more stands for holding one or more tally bags, one or more mounts for holding one or more mobile computing devices used for imaging, image processing, or both. Such systems may include one or more imaging systems configured to generate one or more images of the one or more tally bags or other containers for the surgical instruments. The imaging system used to obtain the first tally may be the same as the imaging system used to obtain the second tally. Such a system may include one or more processing components for use in determining the first tally, the second tally, or both, and for comparing the first tally to the second tally. For example, such one or more processing components may be configured to analyze one or more images to identify surgical instruments appearing in the images, tally the surgical instruments in the images, or both. Such one or more processing components may utilize one or more computational techniques, such as computer vision, machine learning, depth vision, or any suitable combination thereof, to analyze the images to assist in tracking the surgical instruments.Such components (e.g., processing components, imaging components, or both) may be implemented via one or more mobile computing devices, each of which may include one or more cameras or other image sensors, one or more processors, one or more displays, or any suitable combination thereof.

[0092] According to some exemplary embodiments of the systems described herein, one or more of such computing devices may include, for example, mobile devices such as mobile phones, smartphones, tablet computers, laptop computers, or any suitable combination thereof, each of which may include a camera, a processor, a display, or any suitable combination thereof. However, in certain exemplary embodiments, some or all of the system components described herein may be separated as separate interconnected devices or other components. For example, the camera, the display, or both may be located substantially near the tally bag (e.g., in an operating room), while the processor may be remotely located (e.g., in an operating room separate from the camera, the display, or both, or outside the operating room) and communicate with the camera and the display via a wired or wireless communications network. One or more of such computing devices may be used to display information to one or more users. For example, the computing device may include one or more display screens capable of displaying information related to one or more surgical instrument tally, one or more patient parameters, one or more medical procedures, a hospital, an operating room, patient blood loss, user compliance, any suitable combination thereof, or any of the above information. One or more of such computing devices may be configured to provide audio information to one or more users.

[0093] One or more processors may be configured to perform some or all of any one or more of the methods described herein. For example, one or more of such processors may be configured to receive one or more images, perform one or more computational algorithms (e.g., computer vision, machine learning, deep learning, or any suitable combination thereof) to analyze such images, receive and analyze data (e.g., from user input or computer memory), and trigger one or more responses, or any suitable combination thereof. One or more of such processors may be configured to receive and analyze one or more images depicting one or more surgical instruments, one or more tally bags, weight information, timestamp data (e.g., paired with weight information), user input, or any suitable combination thereof, to determine first and second tally of surgical instruments. One or more of such processors may be configured, for example, to perform one or more machine learning algorithms to analyze one or more images and determine the number of surgical instruments shown in one or more images. One or more of such processors may be configured to update an intermediate tally of surgical instruments based on the number of surgical instruments appearing in one or more images. One or more of such processors may be configured to determine a final tally of the surgical instruments, where such determination of the final tally may include accepting manual input from a user. One or more of such processors may be configured to compare a first tally of the surgical instruments to a second tally of the surgical instruments (e.g., an intermediate tally or a final tally). One or more of such processors may be configured to generate a notification (e.g., an alert) based on a comparison of the first tally and the second tally. If the first tally and the second tally do not match, one or more of such processors may be configured to generate or display an alert or other notification.Additionally or alternatively, one or more of such processors may perform one or more additional tasks related to determining an tally of other instruments (e.g., used and unused instruments), tracking compliance information, accepting user input, triggering one or more responses to situations that arise during a medical procedure, sending information and alerts to appropriate personnel, or any suitable combination thereof.

[0094] In general, any one or more processors described herein may be configured to execute instructions stored in memory such that, when executing the instructions, the one or more processors perform one or more of the operations of one or more of the methods described herein. The instructions may be executed by a computer-executable component integrated with an application, applet, host, server, network, website, communication service, communication interface, hardware element, firmware element, software element, or any suitable combination thereof of a user computer, mobile device, wristband, smartphone, or any suitable combination thereof. The instructions may be stored in memory or other computer-readable medium such as RAM, ROM, flash memory, EEPROM, optical device (such as a CD or DVD), hard drive, floppy drive, or any suitable device.

[0095] One or more processors configured to perform some or all of any one or more of the methods described herein may be integrated into a handheld or mobile computing device or may be implemented as an external processor. In some exemplary embodiments, one or more of such processors may be incorporated into a computing device or system, such as a cloud-based computing system, a mainframe computing system, a grid computing system, or other suitable computing system. One or more of the systems described herein may include communication means for communicating data to one or more external processors. Additionally or alternatively, one or more of such processors may be incorporated into a remote server that receives images of the surgical instrument, reconstructs or analyzes the images as described above, and transmits information related to tracking of the surgical instrument.

[0096] As noted above, one or more exemplary embodiments of the systems described herein may include one or more imaging systems. For example, such a system may include a camera operable to generate one or more images of the surgical instruments (e.g., surgical instruments contained in one or more tally bags), such as a set of one or more still images or a portion of a video feed. The one or more images may include a color image, a depth image (e.g., infrared, stereoscopic, ultrasound, etc.), a hyperspectral image, another suitable type of image, or any suitable combination thereof. The camera may include at least one optical image sensor (e.g., CCD, CMOS, etc.), which captures a color optical digital image using the red, green, blue (RGB) color components of the pixels, or one or more other suitable optical components. For example, the camera may include a single image sensor paired with suitable corresponding optics, filters (e.g., a color filter array such as a Bayer pattern filter), or both. As another example, the camera may include multiple image sensors paired with suitable corresponding optics, such as at least one prism or diffractive surface for splitting white light into separate color channels (e.g., RGB), each detected by a respective image sensor. One or more imaging systems, one or more computational processing tools, or any suitable combination thereof may be configured on the same device, or the imaging system may be a separate component that communicates image data to one or more other components. One or more of such imaging systems may be part of a mobile device (e.g., a camera on a smartphone), which may or may not be configured to implement one or more of the processing components described above. Such imaging systems may be configured to transmit images to one or more processors for analysis, a database that stores the images (e.g., cloud storage), or both.

[0097] One or more exemplary embodiments of the system described herein may include a display operable to display or otherwise communicate information to a user (e.g., a doctor, a nurse, other medical personnel, or any suitable combination thereof), including patient information, one or more images of the surgical instruments, one or more quantified metrics related to the location of the surgical instruments, one or more notifications regarding a comparison of a first tally and a second tally (e.g., a final tally) of the surgical instruments, a predefined protocol for the user to follow if the first tally and the second tally of the surgical instruments do not match, one or more measurements of the possibility of duplicate imaging of the surgical instruments (e.g., surgical fibers), a value of an index counter of the surgical instruments, one or more warnings or prompts to the user indicating potential duplicate imaging of the surgical instruments, one or more summary reports, or any suitable combination thereof. The display may include a screen on a handheld or mobile device, a computer monitor, a television screen, a projector screen, or other suitable display. The display may be integrated into the same device as one or more components of the system. Additionally or alternatively, the display may include a standalone separate monitor, screen, or other suitable display.

[0098] According to some exemplary embodiments, the display may be configured to display a user interface (e.g., a graphical user interface) that allows a user to interact with the displayed information. For example, the user interface may allow a user to manipulate (e.g., zoom, crop, rotate, or any suitable combination thereof) one or more images, or to manually define an image region depicting at least a portion of a surgical instrument. As another example, the user interface may allow a user to select one or more display options (e.g., font, color, language, or any suitable combination thereof), select content (patient information, quantified metrics or other bodily fluid related information, alerts, etc.), or both. As yet another example, the user interface may allow a user to select one or more images to delete for depictions of overlapping surgical instruments, or to rotate, flip, or manipulate one or more images on the display. The display may be user interactive and may include a resistive or capacitive touch screen that responds to skin, a stylus, or other user touch. The display may be user interactive via a cursor controlled by a mouse, keyboard, or other input device. For example, the display may allow a user to manually enter information indicating the first tally, the second tally, patient information, or any suitable combination thereof.

[0099] One or more exemplary embodiments of the systems described herein may include a speaker or other suitable audio system that communicates information related to the location of the surgical instruments (e.g., first tally or second tally), the value of an index counter for the surgical instruments, one or more notifications regarding a comparison of the first and second tally of the surgical instruments (e.g., final tally), a predetermined protocol for the user to follow if the first and second tally of the surgical instruments do not match, one or more measurements of the possibility of duplicate imaging of the surgical instruments, one or more warnings or prompts to the user indicating the possibility of duplicate imaging of the surgical instruments, one or more summary reports, or any suitable combination thereof. The display, the audio system, or both may provide one or more notifications or warnings if the first tally does not match the second tally (e.g., final tally). This may be useful to prompt a particular action in response, such as prompting the user to begin a specified protocol. As another example, the display, audio system, or both may provide one or more warnings or prompts to the user to indicate the possibility that two or more images depict the same surgical instrument, which may prompt the user to verify whether duplicate imaging has occurred and help reevaluate the location of counted or uncounted surgical instruments, or any suitable combination thereof.

[0100] One or more exemplary embodiments of the systems described herein may include instruments and equipment for use in surgical instrument counting methods. For example, such systems may include one or more collection containers, one or more count bags for use in tracking and counting surgical instruments, one or more stands configured to hold the count bags, one or more mounts configured to hold one or more imaging systems (such as mobile devices), one or more weight-sensing components coupled to one or more components of the system, or any suitable combination thereof.

[0101] One or more exemplary embodiments of the system described herein may include one or more collection containers for use in collecting surgical instruments, such as surgical fibers. The collection containers may be used by medical personnel to place used instruments prior to tallying (e.g., after the fibers have been used). The collection container (e.g., a "kick bucket") may be of any suitable size, shape, or material. The collection container may have a circular cross section. The collection container may be or include a small bucket-like container and may include a liner. The collection container may be made of metal, plastic, or any suitable material. The liner of the collection container may be made of any suitable material. For example, the liner may be a clear plastic liner. A clear liner may provide the advantage of easy visualization of both dirty and clean surgical instruments. The collection container may be of any suitable size and may accommodate any suitable number of surgical instruments. Some exemplary embodiments of such a system may include one or more collection containers located at various locations throughout the operating room. For example, one or more collection containers may be located near the operating table to allow a doctor or nurse to easily dispose of the surgical instruments after use.

[0102] One or more exemplary embodiments of the system described herein may include one or more tally bags for use in containing and tallying the surgical instruments. For example, as described above, the tally bag may include a lining and one or more pockets configured to contain one or more surgical instruments. The tally bag may be of any suitable size, and such a system may include one or more tally bags of various sizes. For example, different tally bags may be sized to contain different types of surgical instruments (e.g., different sized surgical fibers). Various types of surgical fibers come in various sizes, and the tally bag may be appropriately sized to contain any suitable number and any suitable type of surgical fibers. For example, the tally bag may be sized to contain between 1 and 30 surgical fibers. Similarly, the tally bag may have any suitable number of pockets to contain any suitable number of surgical fibers. Although the fiber tally protocol generally involves placing one surgical fiber per pocket, the pockets of the tally bag may be configured to contain any suitable number of surgical fibers. Additionally, the pockets of the tally bag may be of a size and shape sufficient to contain the various types of surgical fibers. For example, the pockets of the tally bag may be configured to accommodate various sizes of surgical fabrics, such as surgical fabrics ranging in size from 4 inches by 4 inches to 18 inches by 18 inches. The pockets of the tally bag may also be configured to accommodate various types of surgical fabrics, such as laparotomy fabrics (e.g., surgical fabrics for laparotomy).

[0103] One or more exemplary embodiments of the system described herein for tracking surgical instruments may include one or more stands, such as a pole with one or more hooks, mounts, or any suitable combination thereof. The stand may include one or more mounts for holding one or more imaging devices (e.g., mobile devices). Such a system may include one or more tally bags, each configured to contain a surgical instrument, such as a surgical fabric, and aid one or more users in implementing one or more tally protocols for the surgical instruments. As described above (e.g., with reference to FIG. 4 ), the tally bag may include a lining and one or more pockets. The lining may include one or more openings configured to interact with one or more hooks of the pole to suspend the tally bag from the one or more hooks. The components of the tally bag (e.g., the lining and pockets) may be made of any suitable material and may have any suitable coloring. For example, the lining of the tally bag may have an opaque blue, opaque white, or other suitable color, and the pockets may be made of a transparent material. It may be beneficial for the lining of one or more of the tally bags to be opaque and colored so that the tally bags are easily identifiable, even when stacked (e.g., stacked on top of each other), so that they contrast in color with both the soiled and clean surgical instruments placed therein (e.g., soiled and clean surgical fabrics). Additionally, a transparent pocket may be beneficial to allow visualization of the surgical instruments in the pocket, visualization of one or more markers on the surgical instruments (e.g., tags on the surgical fabrics), or both. The tally bags may be of any suitable shape and size, such as a shape and size that is useful for hanging the tally bag from a pole. The exemplary embodiment shown in FIG. 4 uses a rectangular tally bag with a rectangular pocket, but the tally bag and its pocket may be of any suitable shape and size, as described above.

[0104] One or more exemplary embodiments of the system described herein for tracking surgical instruments may include one or more weight sensing components. Such weight sensing components may be used to sense the presence of surgical instruments, weigh them, tally them, or any suitable combination thereof. As described above, the one or more weight sensing components may include one or more scales used to determine a first tally of the surgical instruments. For example, the one or more weight sensing components may be configured to receive the first tally of the surgical instruments and communicate the weight information to a processor to determine the first tally. The one or more weight sensing components may also be coupled to or incorporated into one or more collection containers (e.g., tally bags or "kick buckets") to determine a second tally of the surgical instruments. The one or more collection containers may be coupled to or include scales or other suitable weight sensing components to sense a decrease in weight when a user removes a surgical instrument from the one or more collection containers. The one or more tally bags may be coupled to one or more weight sensing components in the exemplary form of one or more strain gauges or scales coupled to the pole to sense a change in weight when surgical instruments are added to the tally bag to determine a second tally of the surgical instruments.

[0105] The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the present subject matter. However, it will be apparent to one skilled in the art that specific details are not required to practice the present subject matter. Thus, the foregoing descriptions of certain exemplary embodiments have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the present subject matter to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The exemplary embodiments were selected and described in order to explain the principles of the present subject matter and some of its practical applications, thereby enabling those skilled in the art to utilize various exemplary embodiments with various modifications as suited to the present subject matter and the particular use contemplated. It is intended that the claims and their equivalents define the scope of the invention.

[0106] The following recited description describes various examples of the methods, machine-readable media, and systems (eg, machines, devices, or other apparatuses) described herein.

[0107] A first example provides a method for tracking a surgical instrument, the method comprising: On one or more processors Receiving a first tally of surgical instruments; Receiving one or more images, each image being of a field of view including one or more surgical instruments; determining a second count of the surgical instrument based at least in part on the one or more received images; and and providing a notification based on a comparison of the first tally of surgical instruments to the second tally of surgical instruments.

[0108] A second example provides a method as in the first example, where determining a second tally of surgical instruments includes identifying one or more surgical instruments in the one or more received images.

[0109] A third example provides a method according to the second example, in which at least one of the received images is a depth image.

[0110] A fourth example provides a method according to the third example, in which the depth image is an infrared image.

[0111] A fifth example provides a method according to any of the second to fourth examples, wherein at least one of the received images is a color image.

[0112] A sixth example provides a method according to any of the second to fifth examples, wherein the surgical instrument is a surgical fabric.

[0113] A seventh example provides a method of any of the second to sixth examples, wherein identifying the one or more surgical instruments includes identifying and characterizing a vessel including one or more compartments in the one or more images.

[0114] An eighth example provides a method according to the seventh example, wherein the container comprises a liner and a plurality of pockets arranged in a rectangular grid on the liner.

[0115] A ninth example provides a method according to the eighth example, wherein the plurality of pockets includes a transparent material.

[0116] A tenth example provides the method of the eighth example, wherein the backing comprises an opaque material.

[0117] An eleventh example provides the method of any of the seventh to tenth examples, wherein characterizing the vessel includes determining one or more parameters of the vessel.

[0118] A twelfth example provides a method as in the eleventh example, wherein determining the one or more parameters of the identified vessel includes quantifying one or more compartments of the vessel.

[0119] A thirteenth example provides a method as in the eleventh or twelfth example, wherein determining one or more parameters of the identified container includes determining a geometric arrangement of one or more compartments of the container.

[0120] A fourteenth example provides a method according to any of the eleventh to thirteenth examples, wherein determining the one or more parameters of the identified container includes determining the presence or absence of a surgical instrument in each of one or more compartments of the container.

[0121] A fifteenth example provides a method according to any of the first to fourteenth examples, wherein determining the second tally of the surgical instruments is further based, at least in part, on detecting a change in weight of a container configured to hold one or more surgical instruments.

[0122] A sixteenth example provides a method of any of the first to fifteenth examples, wherein receiving the first tally of surgical instruments includes identifying one or more surgical instruments within the image.

[0123] A seventeenth example provides a method according to any of the first to sixteenth examples, wherein receiving the first tally of the surgical instruments is based, at least in part, on detecting weights of the one or more surgical instruments.

[0124] An eighteenth example provides the method of any of the first to seventeenth examples, wherein the first tally of surgical instruments is provided via user input.

[0125] A nineteenth example provides the method of any of the first to eighteenth examples, wherein determining the second tally of the surgical instrument includes updating a current index value of the surgical instrument.

[0126] A twentieth example provides the method of the nineteenth example, further comprising displaying a current index value of the surgical instrument on a display.

[0127] A twenty-first example provides a method according to any of the first to twentieth examples, further comprising prompting a user to verify accuracy of at least one of the first tally and the second tally.

[0128] A twenty-second example provides a method according to any of the first to twenty-first examples, further comprising displaying at least one of the first tally of surgical instruments, the second tally of surgical instruments, or a notification on the display.

[0129] A twenty-third example provides the method of any of the first to twenty-second examples, further including providing a summary report of the tracked surgical instruments.

[0130] A twenty-fourth example provides a method of any of the first to twenty-third examples, wherein providing a notification based on a comparison between the first tally and the second tally includes generating an alert in response to the first tally not matching the second tally.

[0131] A twenty-fifth example provides a method according to the twenty-fourth example, where the alert follows a predetermined protocol.

[0132] A twenty-sixth example provides a method according to any of the first to twenty-fifth examples, wherein at least a portion of the first tally is received prior to the medical procedure.

[0133] A twenty-seventh example provides the method of any of the first to twenty-sixth examples, wherein the second tally is determined during or after the medical procedure.

[0134] A twenty-eighth example provides a method as in any of the first to twenty-seventh examples, wherein determining the second aggregation includes analyzing the one or more received images using one or more machine learning models.

[0135] A twenty-ninth example provides a method as in the twenty-eighth example, wherein the one or more machine learning models incorporate deep learning techniques.

[0136] A thirtieth example provides a system for tracking a surgical instrument, the system comprising: The one or more processors are configured to perform operations including: The operation is Receiving a first tally of surgical instruments; Receiving one or more images, each image being of a field of view including one or more surgical instruments; determining a second count of the surgical instruments based at least in part on the one or more received images; and providing a notification based on a comparison of the first tally of surgical instruments to the second tally of surgical instruments.

[0137] A thirty-first example provides the system described in the thirtieth example, wherein determining the second tally of surgical instruments includes identifying one or more surgical instruments within the one or more received images.

[0138] A thirty-second example provides the system of the thirty-first example, wherein at least one of the received images is a depth image.

[0139] A thirty-third example provides the system of the thirty-second example, wherein the depth image is an infrared image.

[0140] A thirty-fourth example provides a system according to any of the thirty-first to thirty-third examples, wherein at least one of the received images is a color image.

[0141] A thirty-fifth example provides a system described in any of the thirty-first to thirty-fourth examples, wherein identifying the one or more surgical instruments includes identifying and characterizing a container including one or more compartments in the one or more images.

[0142] A thirty-sixth example provides the system of the thirty-fifth example, wherein characterizing the vessel includes determining one or more parameters of the vessel.

[0143] A thirty-seventh example provides the system of the thirty-sixth example, wherein determining the one or more parameters of the identified container includes quantifying one or more compartments of the container.

[0144] A thirty-eighth example provides a system as described in the thirty-sixth example or the thirty-seventh example, wherein determining one or more parameters of the identified container includes determining a geometric arrangement of one or more compartments of the container.

[0145] A thirty-ninth example provides a system described in any of the thirty-sixth to thirty-eighth examples, wherein determining the one or more parameters of the identified container includes determining the presence or absence of a surgical instrument in each of the one or more compartments of the container.

[0146] A fortieth example provides a system according to any of the thirty-fifth to thirty-ninth examples, wherein the container is a flexible container including a lining and a plurality of pockets arranged in a rectangular grid.

[0147] A forty-first example provides the system of the fortieth example, wherein the plurality of pockets comprises a transparent material.

[0148] A forty-second example provides the system of the fortieth example, wherein the backing comprises an opaque material.

[0149] A forty-third example provides a system according to any of the thirty to forty-second examples, further comprising an optical sensor configured to generate one or more images received by the processor.

[0150] A forty-fourth example provides a system according to any one of the thirtieth to forty-third examples, further comprising a display screen.

[0151] A forty-fifth example provides the system of the forty-fourth example, wherein the display screen is configured to display a summary report of the tracked surgical instruments.

[0152] A 46th example provides a system as described in the 44th example or the 45th example, wherein the display screen is configured to display a notification based on a comparison between the first tally and the second tally.

[0153] A forty-seventh example provides the system of the forty-sixth example, wherein the notification includes an alert according to a predefined protocol if the first tally does not match the second tally.

[0154] A forty-eighth example provides the system of any of the thirty to forty-seventh examples, wherein the surgical instrument is a surgical fabric.

[0155] A forty-ninth example provides a system according to any of the thirty to forty-eighth examples, wherein at least one of the one or more processors is within a mobile computing device that is removably mounted to a stand.

[0156] A fiftieth example provides a method for tracking a surgical instrument, the method comprising: On one or more processors, receiving an image of a field of view, the field of view including one or more surgical instruments; Determining a presence of one or more surgical instruments within the image; and providing an indication of the determined presence of the one or more surgical instruments in the image.

[0157] A fifty-first example provides a method as in the fifty-first example, further comprising quantifying one or more surgical instruments in the image based on the at least one trained machine learning model.

[0158] A 52nd example provides a method as in the 51st example, wherein the machine learning model incorporates deep learning techniques.

[0159] A 53rd example provides a method according to any of the 50th to 52nd examples, wherein determining the presence of the one or more surgical instruments includes detecting a presence of a container in the image, the container including a plurality of compartments, each compartment configured to receive at least one respective surgical instrument.

[0160] A fifty-fourth example provides the method of the fifty-third example, further comprising classifying each compartment based on the presence or absence of a surgical instrument.

[0161] A 55th example provides a method as recited in any of the 50th to 54th examples, wherein providing an indication of the determined presence includes providing an indication of the determined presence of at least one surgical instrument in the image.

[0162] A fifty-sixth example provides the method of any of the fifty-fifth to fifty-fifth examples, wherein the one or more surgical instruments include a surgical fabric.

[0163] A fifty-seventh example provides the method of any of the fifty-sixth examples, wherein the one or more surgical instruments include a surgical instrument.

[0164] A fifty-eighth example provides a carrier medium carrying machine-readable instructions for controlling a machine to perform the operations (eg, method operations) performed in any one of the preceding examples.

[0165] A fifty-ninth example provides a machine-readable medium (e.g., a non-transitory machine-readable storage medium) including instructions that, when executed by one or more processors of the machine, cause the machine to perform operations (e.g., method operations) specified in any one of the preceding examples.

Claims

1. determining, by one or more processors, a first tally of surgical instruments, the first tally quantifying the set of surgical instruments; acquiring, by the one or more processors, an image depicting at least a portion of the set of surgical instruments; determining, by the one or more processors, a second tally of the set of surgical instruments based on a depiction of at least the portion of the set of surgical instruments in the image; providing, by the one or more processors, a notification based on a comparison of the first tally of the surgical instruments to the second tally of the surgical instruments; determining the second count of surgical instruments includes identifying at least one surgical instrument depicted in the image by characterizing a container including at least one compartment, the container including the at least one compartment depicted in the image; The one or more processors are configured to characterize the container by identifying whether each compartment of the at least one compartment is full or empty based on the image using a pocket classifier model, and to determine the second tally of the surgical instruments based on an output from the pocket classifier model. method.

2. The method of claim 1 , wherein the characterization of the container comprises quantifying the at least one compartment of the container based on the image.

3. The method of claim 1 , wherein the characterization of the container comprises determining a geometry of the at least one compartment of the container based on the image.

4. The method of claim 1 , wherein the characterization of the container comprises determining, based on the image, the presence or absence of a corresponding surgical instrument in each of the at least one compartment of the container.

5. 10. The method of claim 1, wherein the determination of the second tally of surgical instruments is based on a change in a detected weight of a container configured to receive at least one surgical instrument of the set of surgical instruments.

6. The method of claim 1 , wherein the determining the first collection of surgical instruments comprises identifying the set of surgical instruments from a further image.

7. The method of claim 1 , wherein the determination of the first count of surgical instruments is based on a weight of at least one surgical instrument of the set of surgical instruments.

8. 2. The method of claim 1, wherein the providing of the notification based on a comparison of the first tally to the second tally comprises providing a warning in response to the first tally not matching the second tally.

9. A machine-readable medium containing instructions, The instructions, when executed by one or more processors of a machine, cause the machine to: determining a first tally of surgical instruments, the first tally quantifying the set of surgical instruments; acquiring an image depicting at least a portion of the set of surgical instruments; determining a second tally of the set of surgical instruments based on a representation of at least the portion of the set of surgical instruments in the image; providing a notification based on a comparison of the first tally of the surgical instruments to the second tally of the surgical instruments; determining the second count of surgical instruments includes identifying at least one surgical instrument depicted in the image by characterizing a container including at least one compartment, the container including the at least one compartment depicted in the image; The one or more processors are configured to characterize the container by identifying whether each compartment of the at least one compartment is full or empty based on the image using a pocket classifier model, and to determine the second tally of the surgical instruments based on an output from the pocket classifier model. Machine-readable medium.

10. 1. A system comprising: one or more processors; a memory for storing instructions; The instructions, when executed by at least one of the one or more processors, provide the system with: determining a first tally of surgical instruments, the first tally quantifying the set of surgical instruments; acquiring an image depicting at least a portion of the set of surgical instruments; determining a second tally of the set of surgical instruments based on a representation of at least the portion of the set of surgical instruments in the image; providing a notification based on a comparison of the first tally of the surgical instruments to the second tally of the surgical instruments; determining the second count of surgical instruments includes identifying at least one surgical instrument depicted in the image by characterizing a container including at least one compartment, the container including the at least one compartment depicted in the image; The one or more processors are configured to characterize the container by identifying whether each compartment of the at least one compartment is full or empty based on the image using a pocket classifier model, and to determine the second tally of the surgical instruments based on an output from the pocket classifier model. system.

11. The system of claim 10 , wherein the characterization of the container comprises quantifying the at least one compartment of the container based on the image.

12. The system of claim 10 , wherein the characterization of the container comprises determining a geometry of the at least one compartment of the container based on the image.

13. The system of claim 10 , wherein the characterization of the container includes determining a presence or absence of a corresponding surgical instrument in each of the at least one compartment of the container based on the image.

Images