System and method for tracking surgical articles

Abstract

The present disclosure relates to systems and methods for tracking surgical articles. Methods and systems for tracking surgical articles are described herein. A method may be performed by one or more processors and may include receiving a first count of surgical articles, receiving one or more images, where each image is a field of view including one or more surgical articles, determining a second count of surgical articles based at least in part on the one or more received images, and displaying the second count of surgical articles based on the determined second count. And providing a notification based on a comparison between the first count of surgical articles and the second count of surgical articles.

Description

[0001] This application is a divisional application of the invention application with an application date of May 29, 2020, an international application number of PCT / US2020 / 035189, an entry date of the Chinese national phase of January 28, 2022, an application number of 202080054694.0, and an invention name of “Systems and methods for tracking surgical items”.

[0002] Priority claim

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 856,701, filed on June 3, 2019, entitled “SYSTEMS AND METHODS FOR TRACKING SURGICAL ITEMS,” which is incorporated herein by reference in its entirety. Technical Field

[0004] The subject matter disclosed herein relates generally to the field of healthcare technology, and more particularly to new and useful systems and methods for tracking surgical items. Such systems include special purpose machines that facilitate tracking surgical items, including software configuration computerized variations of such special purpose machines and improvements to such variations, and improvements to the technology of such special purpose machines that improve upon other special purpose machines that facilitate tracking surgical items. Background Art

[0005] Tracking surgical items during medical procedures can be important for reducing the risk of items being inadvertently left in a patient's body. Current methods for ensuring that surgical items are not left in a patient's body revolve around manual tracking and counting of surgical items. However, manual counting is often incorrect due to fatigue of medical staff, complex operating procedures that make it difficult to track surgical items, poor counting systems, and items sticking together or otherwise obscured. In addition, compliance with established protocols may be imperfect, particularly during long or complex procedures. Therefore, surgical items left behind remain a significant problem in medical procedures. Items left behind in a patient's body during a surgical procedure can pose significant health risks to the patient and increase the total time and cost of medical care. Therefore, new and improved methods and systems for tracking surgical items during medical procedures may be desirable. Summary of the invention BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a schematic block diagram depicting a method of tracking surgical items according to some example embodiments.

[0007] Figure 2 is a schematic block diagram depicting a method of tracking surgical items according to some example embodiments.

[0008] Figure 3 is a diagram depicting a graphical user interface showing information related to a first count of surgical items, according to some example embodiments.

[0009] Figure 4 is a schematic diagram depicting a container configured to receive surgical items for counting, according to some example embodiments.

[0010] Figure 5 is a schematic diagram depicting a mounting structure configured to hold an imaging device according to some example embodiments.

[0011] Figure 6 is a schematic diagram depicting a protocol for counting surgical items, according to some example embodiments.

[0012] Figure 7 is a schematic diagram depicting a user operating a mobile device to obtain an image of a container configured to receive a surgical item, according to some example embodiments.

[0013] Figure 8 is a schematic block diagram depicting components of a method of considering surgical items, according to some example embodiments.

[0014] Fig. 9 is a diagram depicting a display of information generated by a method for tracking surgical items, according to some example embodiments.

[0015] Fig.10 is a diagram depicting a display of information generated by a method for tracking surgical items, according to some example embodiments.

[0016] Fig.11 is a diagram depicting a graphical user interface prompting a user to verify surgical fabric inventory, according to some exemplary embodiments.

[0017] Fig.12 is a schematic diagram depicting a method of analyzing surgical fabric, according to some example embodiments.

[0018] Fig.13 is a diagram depicting a graphical user interface showing information related to tracking surgical items, according to some example embodiments.

[0019] Fig.14 is a diagram depicting a graphical user interface showing information related to tracking surgical items at intermediate time points during a surgical procedure, according to some example embodiments.

[0020] Fig.15is a diagram depicting a graphical user interface including a summary report of an inventory of surgical items, according to some example embodiments.

[0021] Fig.16 is a diagram depicting a summary report according to some example embodiments. DETAILED DESCRIPTION

[0022] Example methods (e.g., algorithms) facilitate tracking surgical items, and example systems (e.g., special machines configured by special software) are configured to facilitate tracking surgical items. Examples represent possible variations only. Unless otherwise explicitly stated, structures (e.g., structural components, such as modules) are optional and can be combined or subdivided, and operations (e.g., in processes, algorithms, or other functions) can be sequentially varied or combined or subdivided. In the following description, for the purpose of explanation, numerous specific details are set forth to provide a thorough understanding of various example embodiments. However, it will be apparent to those skilled in the art that the present subject matter can be practiced without these specific details.

[0023] Any one or more of the example methods and systems discussed herein can be used to track (e.g., identify, consider, monitor, etc.) one or more surgical items. For example, such methods and systems can be used to compare the counts of surgical items at different time points to determine whether all surgical items are considered. Such methods and systems can provide accurate identification and counting of surgical items using computerized technologies such as computer vision, machine learning, deep learning, or any suitable combination thereof. For example, the use of complex computerized technologies can improve the accuracy of surgical item counting, improve compliance with counting protocols, simplify the inventory process of surgical items, reduce the total cost and time associated with the risk of surgical items left behind, or any suitable combination thereof. Accurate inventory of surgical items introduced into medical procedures can help avoid inadvertently leaving surgical items in the patient's body after the medical procedures. Tracking surgical items can also provide useful information about the use of surgical items during the procedure, compliance with established protocols, or both, and can help users trigger various responses when specific indications occur.

[0024] The methods and systems described herein can be used in a variety of settings, including hospital or clinic settings (e.g., surgical or clinical settings), military settings (e.g., battlefields), or other suitable medical treatment settings. Although the methods and systems described herein are primarily described with reference to tracking surgical items such as surgical fabrics (e.g., surgical sponges or surgical gauze pads), it should be understood that in other variations, such methods and systems can additionally or alternatively be used to track other items (e.g., surgical instruments or equipment).

[0025] Although the example methods are described herein primarily with respect to tracking surgical fabrics, such methods may be applied to any item used in a medical procedure. For example, such methods may include identifying, considering, or otherwise tracking other surgical items, such as scalpels, scissors, sutures, needles, gauze, bandages, or any other suitable items. Considering various types of surgical items may also involve the use of a multi-compartment container for counting surgical items (such as surgical fabrics). Such a multi-compartment container may be referred to as a "counting bag." However, in some example embodiments, the method for tracking surgical items may not use a counting bag. Computerized technology may be used to count items without using a counting bag, and may be used to count grouped or organized items in any suitable manner.

[0026] Overview of the Example

[0027] Described herein are, inter alia, example methods for tracking surgical items. These methods may be performed by one or more processors. In some example embodiments, a method for tracking surgical items includes receiving a first count of surgical items, receiving one or more images, wherein each image depicts a field of view including one or more surgical items, and determining a second count of surgical items based at least in part on the one or more received images. Some example embodiments of the methods discussed herein may also include providing a notification based on the first count of surgical items, the second count of surgical items, or both. For example, a notification may be provided based on a comparison between the first count of surgical items and the second count of surgical items. In some example embodiments, the first count may generally be determined prior to a medical procedure, and the second count may generally be determined during or after the medical procedure.

[0028] The receiving of the first count of surgical items may include identifying one or more surgical items depicted in the image. For example, the identification of the one or more surgical items may include using computer vision, machine learning, other computerized techniques, or any suitable combination thereof. In some example embodiments, the receiving of the first count of surgical items may be based at least in part on detecting the weight of the one or more surgical items. In certain example embodiments, the first count of surgical items may be determined based at least in part on user input. The determination of the second count of surgical items may include identifying one or more surgical items depicted in one or more received images. One or more of the received images may, for example, be a depth image (e.g., an infrared depth image), a color image, or other suitable type of image. In some example embodiments, the identification of the one or more surgical items depicted in the one or more received images includes identifying and characterizing a container that includes at least one compartment and is depicted in the one or more images. The characterization of the container may include determining one or more parameters of the container. The determination of the one or more parameters of the container may, for example, include quantifying one or more compartments in the container, determining the geometric arrangement of one or more compartments in the container, determining the presence or absence of surgical items in at least some (e.g., each) of the one or more compartments, or any suitable combination thereof. In various example embodiments, determination of the second count of surgical items may additionally or alternatively be based at least in part on detecting a change in weight of a container configured to hold one or more surgical items.

[0029] The determination of the second count of the surgical item may include updating the current index value of the surgical item. Some example embodiments of the methods discussed herein include displaying the current index value of the surgical item on a display (e.g., on a graphical user interface of a mobile computing device). Certain example embodiments of such methods include prompting a user to verify the accuracy of the first count of the surgical item, the second count of the surgical item, or the two counts of the surgical item. Various example embodiments of these methods include displaying the first count of the surgical item, the second count of the surgical item, a notification, or any suitable combination thereof on a display (e.g., a display screen). For example, some example embodiments of the methods discussed herein include providing a summary report of the surgical items being tracked. In certain example embodiments, the method includes providing a notification based on a comparison between the first count and the second count, and may include generating a reminder in response to the first count not matching the second count. The reminder may be based on a predetermined protocol. For example, the reminder may display a predetermined protocol, guide the user through the protocol, or both.

[0030] The present invention also particularly describes an example system for identifying, tracking, and accounting for surgical items. The system for tracking surgical items may include one or more processors. For example, the one or more processors may be configured to receive a first count of surgical items, receive one or more images, wherein each image depicts a field of view including one or more surgical items, and determine a second count of surgical items based at least in part on the one or more received images. The one or more processors may also provide a notification based on the first count of surgical items, the second count of surgical items, or both. For example, a notification may be provided based on a comparison of the first count of surgical items with the second count of surgical items. The one or more processors may also be configured to identify one or more surgical items depicted in one or more received images. The one or more surgical items may be surgical fabrics (e.g., surgical sponges or a piece of surgical gauze). At least one of the received images may be a depth image. In some example embodiments, the depth image is an infrared image. At least one of the received images may be a color image. The one or more processors may also be configured to identify and characterize a container that includes at least one compartment and is depicted in one or more images. In some example embodiments, the one or more processors are configured to determine one or more parameters of the container. Example parameters include the geometric arrangement of one or more compartments in the container, the presence or absence of surgical items in each of the one or more compartments, or both. The container can be a flexible container including a backing and a plurality of pockets arranged in a grid (e.g., a rectangular grid). The pockets of the container can be transparent. The backing of the container can be composed of an opaque material, a colored material, or any suitable combination thereof.

[0031] Certain example embodiments of the systems discussed herein also include an optical sensor configured to generate an image received by the one or more processors. In some example embodiments, the system includes a display screen configured to display a summary report of the tracked surgical items. The display screen may also be configured to display a notification, for example, based on a comparison between the first count and the second count. According to some example embodiments, if the first count does not match the second count, the notification includes a reminder according to a predetermined protocol. According to some example embodiments, at least one of the one or more processors is included in a mobile computing device. The mobile computing device may be removably mounted to a bracket or other suitable mounting device.

[0032] Some example embodiments of the methods discussed herein include determining the presence of one or more surgical items, and such methods may be performed by one or more processors. For example, tracking of surgical items may include receiving an image of a field of view, wherein the field of view includes one or more surgical items, determining the presence of one or more surgical items in the image, and providing an indication of the presence of the determined one or more surgical items in the image. Some example methods also include quantifying one or more surgical items in the image based on at least one machine learning model. In some example embodiments, the machine learning model may be combined with one or more deep learning techniques. Other computerized techniques may be used additionally or alternatively to determine the presence of one or more surgical items in the image. The example method may include detecting items in each of one or more compartments of a container. The determination of the presence of one or more surgical items may include detecting the presence of a container in an image, wherein the container includes a plurality of compartments, and each compartment is configured to receive at least one corresponding surgical item. The one or more surgical items may include one or more surgical fabrics, one or more surgical instruments, or any suitable combination thereof. According to some example embodiments, the methods discussed herein include classifying each compartment of the container based on the presence or absence of the surgical item. Providing an indication of the determined presence of a surgical item may include providing an indication of the determined presence of at least one surgical item depicted in the image.

[0033] Example Method for Tracking Surgical Items

[0034] Described herein are example methods for tracking (e.g., identifying, detecting, accounting for, detecting an object over time, etc.) surgical items, such as during a medical procedure. Such methods for tracking surgical items may use various computerized components, such as processors, optical detection systems, computer vision, machine learning, deep learning algorithms, or any suitable combination thereof, to automate and simplify the process of identifying, tracking, and accounting for surgical items. Figure 1 Schematic depictions of some example embodiments of a method 100 for tracking surgical items in a medical procedure are provided. The method 100 may include determining a first count of one or more surgical items S110, determining a second count of one or more surgical items S120, and providing a notification S130 based on at least one of the first count of the one or more surgical items and the second count of the one or more surgical items. For example, the notification may be provided based on a comparison between the first count and the second count (e.g., confirming a check between the first count and the second count, tracking progress toward incrementally increasing counts until the first count and the second count are checked, alerting a user when the first and second counts fail to check (e.g., fail to match), or any suitable combination thereof).

[0035] Figure 2 An additional example embodiment of a method 200 for tracking surgical items is shown in . The method 200 may include, at one or more processors, receiving a first count of surgical items S220, receiving one or more images S230, wherein each image is a field of view including one or more surgical items, and determining a second count of surgical items based at least in part on the one or more received images S240. The method 200 may include providing a notification based on a comparison between the first count of surgical items and the second count of surgical items S250.

[0036] In some example embodiments, a first count of surgical items may be associated with a first time, and a second count of surgical items may be associated with a second time after the first time. For example, the first count of surgical items may be an "initial count" and correspond to the number of surgical items introduced into an operating room, removed from packaging, or otherwise prepared for use during a medical procedure. The second count of surgical items may correspond to the number of surgical items used during the medical procedure. As described herein, the term "used" refers to items that are intended to be discarded, and the used items may be soiled or unsoiled (e.g., with blood, other fluids, other contents, or any suitable combination thereof). The second count may be measured at an intermediate time point or at a "final" time point at the end of a procedure (e.g., after the procedure is completed but before surgical closure) as a "final count."

[0037] Maintaining (e.g., updating) and comparing counts of surgical items at various points in time in a medical procedure can facilitate accurate inventory of surgical items. This can provide the benefit of preventing surgical items from being left behind in a patient's body. In some example embodiments, the method of comparing counts may include checking an initial count of surgical items with a final count of surgical items (e.g., checking whether the initial count matches the final count). Such a check may provide the benefit of determining whether all surgical items are considered before surgical closure. According to certain example embodiments, the methods discussed herein may also include providing a notification to a user based on whether the first count matches the second count. If the first count does not match the second count, such methods may include notifying or otherwise reminding the user to take one or more predetermined actions (e.g., searching for surgical items that were not considered).

[0038] As described above, any one or more of the methods described herein for tracking (e.g., identifying and / or detecting) surgical items may be implemented using one or more computerized techniques. In some example embodiments, such methods for tracking surgical items involve the use of computer vision, machine learning, deep learning algorithms, or any suitable combination thereof. For example, computer vision may include one or more computerized techniques for identifying objects in frames of images, image series, or videos. As another example, machine learning or deep learning techniques may involve training and using computerized mathematical models to perform specific tasks by identifying complex patterns associated with surgical items in images (e.g., in still images, in image series, or in frames of videos). In some machine learning techniques, computer algorithms "learn" to perform complex tasks by training on different data sets including a large number of labeled images (e.g., labeled with features associated with surgical items). Labeled data sets can be used to train machine learning models to recognize certain patterns, which enables the model to perform programmed tasks with a high level of accuracy. In some embodiments, even when the model is in use, it can continue to learn by updating the algorithm based on real-world data received over time and continuously updating parameters to improve object tracking, detection performance, or both. For example, machine learning can be used to generate a model that identifies one or more items, counts one or more items, or both. Deep learning is a type of machine learning, for example, it can be based on an artificial neural network that uses multiple layers or analysis to extract relevant features from input data to detect, identify, or otherwise track surgical items. As used herein, the term "computing technology" refers to any one or more of the methods described herein and should not be limited to describing one technology. In addition, in some embodiments, aspects of multiple computing technologies can be combined. For example, one or more computer vision techniques can be implemented via machine learning, deep learning, or both. For example, computer vision for identifying surgical items can utilize machine learning algorithms, deep learning algorithms, or both to perform recognition of surgical items.

[0039] According to various example embodiments, a mobile computing device such as a mobile phone or a tablet computer is used to perform one or more various aspects of the method for tracking surgical items discussed herein. For example, a software application executable by one or more processors on a mobile device can be used to perform one or more operations, including determining a first count, determining a second count, comparing the first count with the second count, and notifying a user based on the comparison. One or more processors on the mobile device can be configured to analyze an image. One or more processors of the mobile device can be configured to perform a machine learning algorithm, a computer vision technique, a deep learning algorithm, or any suitable combination thereof in conjunction with any one or more of the methods described herein. In some example embodiments, the mobile application is configured to perform all steps in the method for tracking surgical items and display information to one or more users based on one or more of the operations (e.g., steps) in the method performed. Implementing one or more of such methods for tracking surgical items on a mobile device can provide the benefits of automating and simplifying the inventory process for surgical items, as well as providing additional virtual presence to guide users through appropriate inventory protocols to facilitate accurate inventory of surgical items. In addition, implementing computerized technology on a mobile device can help improve the method for tracking surgical items, which is highly accessible and intuitive to use. In other example embodiments, one or more external processors may be used to perform some or all of the operations of the method. For example, a mobile device may be used to capture one or more images, receive information, and display the information, and an external processor may be used to perform analysis of the one or more images using one or more machine learning algorithms.

[0040] Example of determining the first count

[0041] Some example embodiments of the method for tracking surgical items include receiving a first count of surgical items. In some embodiments, the first count can establish a corresponding number of one or more surgical items that have been introduced or otherwise prepared for intended use in a medical procedure (e.g., introduced into an operating room, removed from a package, or both). The first count can be established once at any time during the procedure. In some embodiments, the first count is or includes the number of surgical items introduced before or at the beginning of a surgical procedure. In some embodiments, the first count is then updated throughout the procedure (e.g., as the surgical items are introduced into an operating room, removed from a package, or both). For example, a package of surgical items can be introduced into an operating room and removed from a package at the beginning of a procedure (e.g., before any surgical steps are performed on a patient). Subsequently, at a certain point during the medical procedure, such as when performing an operation, a second package of surgical items can be retrieved, introduced into an operating room, and its surgical items can be removed from a package. Therefore, the first count can be updated (e.g., incremented) with the number of surgical items in the second package. Any number of surgical items may be introduced at any point during the procedure, and the first count may be updated each time a new set of surgical items is introduced.

[0042] Establishing a first count of items can provide a benefit that facilitates the inventory of all surgical items at the end of the procedure. For example, the methods discussed herein can include comparing the count of items present at the end of the procedure with the first count to determine whether all surgical items are inventoried (e.g., as opposed to, for example, being left in the patient's body). Various suitable methods can be used to establish a first count of surgical items, such as those described below. In some embodiments, establishing a first count of items can be performed manually. For example, the first count of surgical items can be received as a manual input or other user input. As another example, the first count of surgical items can be established additionally or alternatively by one or more processors (e.g., by one or more computing techniques, as described below). For example, compared to manual methods, computing techniques for establishing a first count can provide the following benefits: establishing a first count with increased accuracy and reduced user errors, and simplifying the process of determining a first count.

[0043] In some example embodiments, as described above, a first count of items may be received as user input. For example, establishing a first count of surgical items may involve a manual counting technique. A user may manually count a plurality of surgical items to determine a first count and provide the first count via a user interface. For example, a user may receive a package containing one or more surgical items, remove the items from the package, and count surgical fabrics as he or she separates the fabrics from one another (such as when arranging them on a surface such as a table). In some embodiments, the manual count is entered into a computerized system by a user (e.g., manually entered, narrated, etc. on a keypad or touch screen). For example, a user may enter the manual count into a computing system via a mobile application executable on a mobile computing device. In some embodiments, the mobile application may prompt the user to enter the manual count.

[0044] In some example embodiments, the determination of the first count of surgical items may additionally or alternatively involve the use of computer vision, machine learning, deep learning, other computing techniques, or any suitable combination thereof. For example, establishing the first count may include analyzing an image to identify one or more surgical items, counting one or more surgical items, or both. Any suitable imaging device may be used to obtain images of one or more surgical items to be counted in the first count. For example, an imaging system such as on a mobile computing device (e.g., a mobile phone, a tablet, etc.) may be used to capture images of one or more surgical items. The image may be a still image or a frame from a video feed. In some embodiments, multiple images (e.g., multiple frames from a video feed) are processed to determine the first count of surgical items. While in some embodiments the image is or includes a color image, in other embodiments the image is or includes a depth image (e.g., infrared, stereo, ultrasound, etc.), a hyperspectral image, or other suitable types of images. In addition, multiple images of one or more surgical items may be obtained simultaneously, nearly simultaneously, or sequentially, and may be of different types (e.g., a color image combined with a depth image). In general, one or more processors may be configured to receive one or more images, analyze one or more images, or both. In addition, the one or more processors may implement computerized techniques (e.g., using computer vision, machine learning, deep learning, or any suitable combination thereof) to identify one or more surgical items, count one or more surgical items, or both, where the one or more surgical items are depicted in an image received by the one or more processors. Such processors may be implemented in a computing device such as a mobile computing device. In some embodiments, a mobile application executable on a computing device may be used in conjunction with any of the variants described below to obtain, receive, and process data related to a first count of surgical items.

[0045] In some example embodiments, determining a first count of surgical items using image analysis includes the following operations. A user obtains a package containing one or more surgical items (such as surgical fabrics) and brings the package into an operating room. The user can remove one or more surgical items from their packages, separate the surgical items, and arrange them on a surface such as a table. The user can then use an imaging system (such as a camera on a mobile computing device) to obtain one or more images of the surgical items on the table. The processor of the mobile computing device then analyzes the image to identify the surgical items separated in the image, count or otherwise quantify the number of surgical items in the image, or both. For example, a computing technique such as computer vision, machine learning, deep learning technology, or any suitable combination thereof can be used to determine the first count of surgical items, including any one or more of those techniques described in further detail below with respect to determining a second count of surgical items. The processor can store information about the detection and counting of surgical items in a memory. In some embodiments, the determination of the first count includes updating the first count by adding the number of surgical items counted by the processor in the image to the number of surgical items previously counted (e.g., surgical items depicted in an image taken at a previous time point). Additionally or alternatively, determination of the first count may include counting a number of surgical items in images captured at a point in time (eg, where no surgical items were previously counted).

[0046] In some example embodiments, one or more computerized techniques include detecting a characteristic arm movement of a user to determine a first count of surgical items. Characteristic arm movements can be characterized in any suitable manner. For example, detecting characteristic arm movements can include identifying a user's "handling" movement when a user separates surgical items, because it is often the case in surgical procedures that medical personnel "distribute" surgical fabrics onto a surface to count them when they remove surgical fabrics from packaging. An imaging system with a camera can be positioned so that the system views the user's arm movements (e.g., a front view, a side view, a top view, or any suitable combination thereof), and a processor can receive a still frame image or video from the imaging system. An imaging system and computerized analysis (e.g., performed by a processor) can be used to detect a sweeping (e.g., distributing) arm movement when a user removes surgical fabrics from packaging to place the surgical fabrics on a surface. Therefore, each arm movement detected by the processor can correspond to a surgical item being counted and can contribute to the first count of surgical items.

[0047] In certain example embodiments, one or more computerized techniques utilize weight information to determine a first count. For example, surgical items may be sequentially placed on the scale, and incremental weight changes may be detected by one or more processors to determine a first count of surgical items. That is, based on incremental weight changes that occur when additional surgical items are placed on the scale, the processor may be able to detect the presence of additional surgical items. Each time the processor detects an incremental weight change, the processor may increase the first count.

[0048] Additionally or alternatively, according to some example embodiments, a plurality of surgical items may be placed on the scale, and information about the weight of each individual surgical item may be used to determine a first count of the surgical items. For example, a user may be prompted by a user interface to enter the type of surgical item to be weighed. For example, the type of surgical item may be entered by searching a database of existing surgical item types, navigating a hierarchical menu of surgical item types, or entering a surgical item type. The surgical item type may be classified or otherwise distinguished, for example, based on function (e.g., type of fabric, surgical instrument, etc.) or item-specific features (such as brand, size, material, etc.). Additionally or alternatively, the surgical item type may be determined based on other identifications (e.g., scanning a barcode or radio frequency identification (RFID) tag associated with the surgical item type). Based on the received surgical item type, a processor in communication with the scale may receive information about the weight of a single surgical item. Thus, information about the weight of the surgical item may be manually entered by a user, stored in a lookup table in a memory, or obtained by scanning an electronic tag such as a barcode or RFID tag on the surgical item. Based on the information regarding the weight of the individual surgical items, the processor may determine the number of surgical items on the scale by dividing the total weight of the plurality of surgical items by the known weight of the individual surgical items.

[0049] As described above, the mobile application may be used in conjunction with one or more weight-based methods of determining the first count. Figure 3 An example embodiment of a graphical user interface 300 depicting information related to determining a first count via use of a scale is depicted. Figure 3 The graphical user interface 300 displays the type of surgical items being counted 310, and the current determination of the first count 320. The graphical user interface 300 instructs the user to place a new surgical item on the scale 330. In addition, the graphical user interface 300 provides selectable icons 340a and 340b to allow the user to manually override (340a) the first count and provide a manual count, and complete (340b) the first count (e.g., indicating that all surgical items of the selected type have at least currently been counted in the first count).

[0050] In some example embodiments, additionally or alternatively, one or more computing techniques for determining the first count involve the use of scannable tags (e.g., RFIDs, bar codes, or any suitable combination thereof). For example, surgical items may include individual RFID tags that may be scanned when the surgical items are introduced into an operating room, when the surgical items are removed from packaging, or at some other point in time prior to use. In other cases, a package of surgical items may include a scannable tag that is configured such that a single scan of the scannable tag can count multiple surgical items. For example, a package of surgical items may include a scannable tag that communicates the number of items in the package to a processor to be included in the first count. Thus, a method of determining the first count may include scanning a label on a surgical item or a package of the surgical item to determine the first count.

[0051] The determination of the first count may include one or more of the methods described above. In some example embodiments, multiple methods of determining the first count are used to verify the accuracy of the first count. The methods of determining the first count may be combined in any suitable manner. For example, a weight-based method (such as detecting incremental weight changes in a scale or utilizing known weight information about a single surgical item) may be combined with an image-based method (such as using one or more computerized techniques to identify and count surgical items in an image). As another example, an image-based method may be combined with one or more computing techniques involving identifying characteristic movements of a user. Any one or more of the methods described herein for determining the first count may be verified using one or more manual counting techniques. In some example embodiments, the mobile application prompts the user to verify the determination of the first count generated using the computing techniques. In some example embodiments, as described above with respect to Figure 3 As described, the method includes allowing a user to override the automatic or computerized determination of the first count if the user determines that the first count is inaccurate.

[0052] In some example embodiments, the determination of the first count of surgical items includes classifying the surgical items. The determination of the first count may include distinguishing different surgical items, determining different types of the same surgical item, or both. For example, different types of surgical fabrics can be distinguished from each other (e.g., by material, function, size, brand, or any suitable combination thereof). As another example, two different types of surgical items can be distinguished from each other (e.g., distinguishing scalpels from scissors). Any suitable method, including any one or more computerized techniques described herein, can be used to determine the type of one or more surgical items (such as the type of one or more surgical fabrics). In some embodiments, in addition to the first total count of all surgical items counted regardless of type, the determination of the first count also includes a separate category indicating the first count. For example, the first count may include a type-specific first count of 18-inch x 18-inch surgical fabrics, and another type-specific first count of 4-inch x 4-inch surgical fabrics.

[0053] Any method described herein for determining the first count may additionally or alternatively be used to determine the second count. Similarly, any method described herein for determining the first count may additionally or alternatively be used to perform any other suitable analysis of surgical items (e.g., tracking used items, tracking unused items, or both, as described below).

[0054] Example of determining the second count

[0055] Some example embodiments of the method for tracking surgical items also include determining a second count of surgical items. As described above, the second count of surgical items can be compared to the first count of surgical items. The second count can be determined at any suitable time point in a procedure (e.g., a medical procedure, such as a surgical procedure). For example, the second count can be an "intermediate count" determined at an intermediate time point between the start time point and the end time point of the procedure during the procedure. The determination of the intermediate count can include updating an index counter of the surgical item during the procedure process (e.g., throughout the process) (e.g., when the surgical item is used or otherwise designated as intended to be discarded). The intermediate count can be compared to the first count to help indicate how many remaining "used" surgical items are left on site during the procedure process. The second count can also be the final count at the end of the procedure. It can be desirable to compare the final count with the first count to determine whether all surgical items have been counted and limit the risk of surgical items being left in the patient undergoing surgery. The end of the procedure can be represented in any suitable manner. For example, the end of the procedure may be when the patient is ready to be released from the operating room (eg, after the surgical site has been closed), or before the patient's surgical site has been closed but after the surgical procedure is complete.

[0056] Any method described herein for determining the second count may additionally or alternatively be used to determine the first count. Similarly, any method described herein for determining the second count may additionally or alternatively be used to perform any other suitable analysis of surgical items (e.g., tracking used items, tracking unused items, or both, as described below).

[0057] Example of using counting bags

[0058] In some exemplary embodiments, counting surgical items involves using a counting bag to count surgical fabrics. The use of the counting bag can facilitate accurate and organized counting of surgical fabrics and reduce the risk of surgical items being left behind in the patient. One or more users can place surgical fabrics designated for disposal into the pockets of the counting bag to facilitate efficient and accurate counting. In addition, in some exemplary embodiments, discarding surgical fabrics or other items can include a multi-step item collection process. For example, in some operating room protocols, after surgical fabrics are used in the protocol, they can be piled up in one or more collection containers (e.g., a "kick bucket," which is an example form of a metal bucket lined with a plastic liner and placed near the operating table or another location in the operating room). During or after the protocol, one or more medical staff can then remove surgical fabrics from one or more collection containers and place surgical items in the counting bag for counting. In other exemplary embodiments, surgical fabrics can be placed directly into the counting bag without using a separate collection container. All surgical fabrics used in the protocol can be counted using the counting bag. As noted above, the term "used" is not limited to surgical fabrics used to absorb blood or other fluids, but can refer to any surgical fabric that is introduced into an operating room and subsequently deemed disposable or otherwise designated for disposal. According to some exemplary embodiments, manual counting of surgical fabrics involves placing surgical items, such as surgical fabrics, into compartments of a counting bag according to a particular protocol as described below. One or more methods of counting surgical items, such as those implementing one or more computerized techniques, can be used to automate and reduce errors associated with manual counting using one or more counting bags.

[0059] In some cases, medical personnel may employ a standardized procedure or protocol when counting surgical items using one or more counting bags. Figure 6 One such protocol used in an operating room is depicted. The items are placed in pockets 610 of a counting bag 600 in an order indicated by numbers, for example, from bottom to top and from right to left. Thus, when the procedure is followed correctly, the first pocket to be filled is the bottom right pocket (Pocket 1), and the last pocket to be filled is the top left pocket (Pocket 10). According to other protocols, the counting bag 600 may be filled from top to bottom and from left to right. However, any other suitable protocol may be followed.

[0060] Figure 4 Schematically depicted is a counting bag 400 for use in medical procedures. The counting bag 400 includes a backing 402 and a plurality of pockets 410 or other compartments. Figure 4 In the example, the counting bag 400 includes ten pockets arranged in a rectangular array. Figure 4As shown in , ten pockets 410 are arranged in five rows, each row including two pockets 410. For example, each row may include a larger pocket 412 formed between the backing 402 material and the front material, and the larger pocket 412 may be divided into two smaller pockets 410, and the seal 414 forms a seam in the larger pocket 412. However, the counting bag 400 may include any suitable number of pockets 410, and the pockets 410 may be configured in any suitable manner. For example, some or all of the pockets 410 may be separate compartments, some or all of the pockets 410 may be composed of a single larger pocket (e.g., larger pocket 412), each pocket 410 is separated by a seal, or any suitable combination thereof. The counting bag 400 may be mounted to a support, such as a rod 430 (e.g., an intravenous (IV) rod). The rod 430 may include a transverse mounting rod 431, which includes one or more (e.g., two) hooks 432. The counting bag 400 may include one or more (eg, two) openings 404 that allow the counting bag 400 to be hung from the hook 432. Figure 4 The counting bag 400 depicted in FIG. 4 is rectangular and includes ten rectangular pockets 410 , but the counting bag 400 and its pockets 410 may have any suitable configuration.

[0061] The counting bag 400 can include (e.g., be made of) any suitable material. For example, the backing 402 of the counting bag 400 can be made of a flexible material, such as a polymer, plastic, silicone, nylon, rayon, any other suitable material, or any suitable combination thereof. As another example, the backing 402 of the counting bag 400 can be made of a more rigid material, such as a rigid plastic, a polymer, a metal, any other suitable material, or any suitable combination thereof. The backing 402 of the counting bag 400 can be transparent, translucent, opaque, colored, or any suitable combination thereof. For example, the backing 402 can be opaque blue. The opaque and colored backing 402 (e.g., opaque blue) can make the counting bag 400 easier to be individually identified among multiple counting bags (e.g., when multiple counting bags are stacked in front of each other on the same rod 430, or when multiple rods (e.g., rod 430) are stacked in front of each other), and can provide a contrasting background against which the surgical fabric stands out. Pocket 410 can be made of the same or different material as backing 402. For example, pocket 410 can be made of flexible plastic, silicone, or polymer material. Pocket 410 made of flexible material can provide the benefit of allowing surgical fabrics to be easily placed therein. Pocket 410 can also be transparent or opaque. In some example embodiments, one or more pockets 410 are transparent to allow easy visualization and counting of surgical items.

[0062] Although Figure 4 The counting bag 400 depicted in FIG. 4 is rectangular and includes ten rectangular pockets 410, but the counting bag 400 can have any suitable configuration. For example, the backing 402 of the counting bag 400 can be rectangular, generally square, oval, triangular, or any suitable shape. The counting bag 400 need not be as Figure 4 As depicted in FIG. 4 , there are ten pockets 410 and may include any suitable number of pockets 410. For example, the counting bag may include 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, or any suitable number of pockets 410. Although Figure 4 A counting bag 400 configured to be suspended from a rod 430 is depicted, but the counting bag 400 can be mounted in any suitable manner. For example, the counting bag 400 can have one or more openings 404 or any other suitable attachment mechanism in any suitable location. For example, the counting bag can be configured to be mounted to a hook on a wall, or mounted on a hook or mounting rod that is configured to be placed on the top of a door or cabinet. As another example, the counting bag 400 can be fixed to the wall with an adhesive or other suitable fastener. In addition, the size of the counting bag 400 does not need to fit the dimensions of a rod (e.g., an IV rod), and the size of the counting bag 400, its pocket 410, or both can be sized differently from other counting bags, their pockets, or both to accommodate different types of surgical items, such as surgical fabrics of different sizes.

[0063] In some example embodiments, the rod 430 may also include a mount for mounting an imaging device. One or more imaging devices may be used to generate one or more images for various components (e.g., operations) of the method for tracking surgical items, including the determination of the first count, the determination of the second count, or both. In particular, one or more imaging devices mounted to the rod 430 to which the counting bag 400 is mounted may be used to determine the second count of surgical items in the counting bag 400. For example, a mount for an imaging device may make it easier for a user to take (e.g., capture) an image of the counting bag 400, so that the user does not have to hold the imaging device when taking the image. One or more mounts may be fixedly attached to the rod 430 or may be removably fixed to the rod 430. In addition, the mount may be adjustable along the rod 430 (e.g., configured to slide vertically up and down along the rod 430), rotatably adjustable around the rod 430 (e.g., adjustable to the left or right side of the rod 430), or both. Figure 5An example embodiment of a counting bag holder 500 is schematically depicted, including a mount 540 attached to a rod 530, the rod 530 being configured to hold a mobile device 550 (e.g., a tablet or smart phone). In the example embodiment shown, the mount 540 includes a handle 542 configured to securely and releasably hold the mobile device 550. The handle 542 can hold the mobile device 550 using any suitable mechanism, for example, via a friction fit between the mount 540 and the mobile device 550, an adjustable clamp, a snap fit 550 with mating portions on the mount 540 and the mobile device, one or more magnets, a removable adhesive layer attached to the mount 540 or the mobile device 550, or any suitable combination thereof. In the example embodiment shown, the mount 540 is secured to the rod 530 by an attachment mechanism 544. The attachment mechanism 544 can include any suitable attachment mechanism, such as a clamp, a clasp, a hinge, a thread, one or more fasteners (e.g., mechanical fasteners, adhesives, both), or any suitable combination thereof. For example, the attachment mechanism 544 can include an adjustable clamp (e.g., an adjustable jaw) or an expandable sleeve or other insert (e.g., expandable, foam, etc.) that is configured to attach to and adjust to the size of the rod 530. In some example embodiments, the attachment mechanism 544 includes one or more fasteners that are configured to couple the mount 540 to the rod 530 (e.g., a screw or bolt is inserted through the mount 540 and through an opening in the rod 530 and secured with a nut, other threaded backing, or other fastener backing). Although Figure 5 The mounting member 540 in the embodiment is depicted as extending horizontally from the rod 530, but the mounting member 540 can extend at any suitable angle so that the camera of the mobile device 550 can be positioned to capture an image of the counting bag (e.g., the counting bag 400) mounted on the rod 530. In addition, the mounting member 540 can be fixed to any suitable portion of the counting bag holder 500. For example, the mounting member 540 can be fixed to the rod 530, the mounting rod 531, or the hook 532 of the counting bag holder 500. In addition, the mounting member 540 can have any suitable configuration. The mounting member 540 can be substantially straight, or the mounting member 540 can include a curved configuration. In addition, the mounting member 540 can be adjustable in length, adjustable in angle (e.g., the angle between the mounting member 540 and the rod 530 is adjustable) (such as having a gooseneck structure), or any suitable combination thereof. This can provide the benefit of allowing the user to adjust the positioning of the mounting member 540 according to the specific needs of the user or the procedure at hand.

[0064] Example Computerized Technology for Tracking Surgical Items

[0065] In some example embodiments, one or more computerized techniques are used to identify, count, or both surgical items that may have been placed into one or more counting bags (e.g., counting bag 400). Using one or more such computerized techniques in conjunction with one or more counting bags may be desirable to automate and simplify the process of surgical item inventory, and may improve the accuracy of one or more counts to comply with a protocol using one or more counting bags. For example, one or more images of counting bag 400 may be taken, and one or more computerized techniques may be used to identify and count any surgical fabrics in counting bag 400 (e.g., using computer vision, machine learning, or deep learning, as described elsewhere herein). For example, a user may use an imaging system such as an imaging system of mobile device 550 (e.g., a mobile computing device) to take an image of counting bag 400. Figure 7 Schematically depicting a user 710 using a mobile device 720 to take an image of a counting bag 730. Figure 7 As shown in FIG. 1 , a user 710 may position an imaging system in the example form of a mobile device 720 having a camera 722 such that a counting bag 730 is in the field of view of the camera 722, as shown in FIG. Figure 7 One or more images generated by the camera 722 of the mobile device 720 may be analyzed using one or more computing techniques described herein. Figure 7 A user 710 is depicted holding a mobile device 720 in front of a counting bag 730 to generate an image, but the imaging system (e.g., mobile device 720) may be positioned relative to the counting bag 730 in any suitable manner, such as via a mount proximate to the counting bag 730 (e.g., Figure 5 ), or directly on the counting bag 730 that is angled toward a pocket of the counting bag 730 (eg, pocket 410 ).

[0066] Example Model of Tracking Algorithm

[0067] Figure 8Methods according to some example embodiments are schematically depicted, and example relationships and data flows between various trained models and other possible components of a method 800 (e.g., tracking algorithms) for tracking surgical items placed in one or more counting bags (e.g., counting bag 400) are shown. Each model included in method 800 can use machine learning, deep learning, or both to achieve its sub-goals. The algorithms in each model of method 800 can be combined (e.g., in a unique manner) to achieve an accurate inventory of surgical items in one or more counting bags, as well as to provide additional information to a user. In some example embodiments, the method of tracking surgical items relies on two or more independent counting methods and one or more error reduction techniques to achieve a robust method of counting surgical items. Although reference is made to the present invention, the present invention further provides a method for tracking surgical items. Figure 8 The method 800 described (e.g., considered as an overall tracking algorithm) is described in terms of counting bags, but the method 800 can be used to count items (e.g., surgical items) that are constructed or contained in any suitable manner. In addition, the steps of the method 800 listed below do not need to be performed in the order in which they are described. Figure 8 The processes described or depicted in the flowcharts shown in the drawings may be performed in any suitable order.

[0068] like Figure 8As depicted in , the input to method 800 is an image (810) of one or more counting bags. The images of one or more counting bags 810 may be obtained in any suitable manner using any suitable imaging device. In some example embodiments, a user obtains an image of the counting bags 730 using a camera 722 on a mobile device 720 (e.g., a mobile phone or tablet). However, the input image may be obtained using any one or more of a variety of imaging systems, such as a digital camera, a camera on a computer, a video camera, etc. The image may then be received by a processor. The image may be pre-processed 820 for subsequent analysis. Pre-processing of the image may involve, for example, brightness normalization, contrast correction, reducing the image size to a dimension suitable for processing by one or more deep learning or computer vision models, or any suitable combination thereof. Determining the count (e.g., first count, second count, further count, or any suitable combination thereof) of surgical items in an imaged counting bag (e.g., counting bag 400 or 730) can include analyzing the image with one or more of a variety of computerized techniques, such as a counting bag detection model 840 for detecting the presence of a counting bag in the image, a counting bag geometry model 830 that encodes geometric features (e.g., the size and other features of the counting bag and one or more pockets within the counting bag), a surgical item detection model 850 for detecting the presence of any item in the counting bag, an empty / full pocket classifier model 860 (e.g., general or specific type) for determining the presence or absence of a surgical item in each pocket, or an item counting model 870 for synthesizing information about surgical items in the imaged counting bag, providing a count of surgical items in the imaged counting bag, or both. In some example embodiments, after any pre-processing operations, method 800 utilizes two or more approaches to item counting. For example, one approach can be based on directly detecting surgical items in the counting bag, while one approach can be based on identifying pockets in the counting bag and classifying the pockets as empty or full. The method 800 may then combine the results of the multiple counting passes and may employ one or more error detection algorithms to arrive at a final inventory of surgical items.

[0069] The counting bag detection model 840 can receive one or more pre-processed images of counting bags, or alternatively, unprocessed images, for analysis. The counting bag detection model 840 can identify the presence of at least one counting bag in the image based at least in part on one or more trained machine learning or deep learning models. Examples of suitable object detection models include faster region-based convolutional neural networks (R-CNN), single short detectors (SSD), and region-based fully convolutional networks (R-FCN), which can achieve high performance on object detection tasks. The counting bag detection model 840 can also utilize one or more of such trained models to identify the location of each counting bag in the image, identify or count 844 one or more edges, corners, or both of the counting bags, or any suitable combination thereof. For example, as Fig. 9 and 10 As shown in the example graphical user interface shown in , the counting bag detection model 840 can detect the presence of the counting bag 910 and indicate the location of the counting bag 910, one or more boundaries of the counting bag 910, or both in the displayed image (e.g., indicated by corresponding bounding boxes). In some example embodiments, the counting bag detection model 840 can crop the counting bag image based on the identification of the counting bag (e.g., substantially isolate the portion of the image corresponding to the identified counting bag) and output the cropped counting bag image 842 for later use.

[0070] The counting bag geometry model 830 may encode geometric information about one or more counting bags, such as the vertical and horizontal dimensions of the counting bag (e.g., counting bag 400 or 730), the location of its pockets (e.g., pocket 410), or both, to help identify specific geometric features of the identified counting bag. The counting bag dimensions 812 encoded in the counting bag geometry model 830 (e.g., with or without other geometric information) may depend, for example, on the make (e.g., brand or manufacturer), model, or both of the counting bag, and may be pre-encoded (e.g., manually entered, such as associated with a specific make or model of the counting bag) or learned from training data. In some example embodiments, the counting bag geometry model 830 may receive one or more inputs, such as the location of the counting bag corners 814 or edges, with or without any other suitable information that may help identify the geometry of the counting bag, analyze the geometry of the counting bag, or both. In some example embodiments, the counting bag geometry is identified based at least in part on knowledge of the making of the counting bag, which can be input by the user as a user entry (e.g., through a user interface on a mobile application), identified via scanning an identifier code (e.g., a barcode, a QR code, etc.) on the counting bag, its packaging, or by identifying the making of the counting bag directly from an image of the counting bag (e.g., based on optical character recognition of a label, characteristic color features, characteristic geometric features, or any suitable combination thereof). For example, the user can input the making of the counting bag, and the geometric model can use the known information to help determine the geometric information of the counting bag. As another example, the user can manually identify the boundaries of the counting bag, its pockets, or both on a mobile device by applying reference markers or adjusting the size of one or more virtual contours to define the counting bag, its pockets, or both. The counting bag geometric model can use homography or one or more other perspective transformations to automatically identify the location of the counting bag. Based on these locations, the counting bag geometric model 830 crops the image of each counting bag pocket for receipt by the empty / full bag classifier model 860. Thus, the counted bag geometry model 830 may provide a pocket image 832 (e.g., corresponding to an identification of a portion of an image of a counted bag pocket, a cropped image segment, or any suitable combination thereof) that may also include information about the location of the pocket, its pocket corners 834, or both. Fig. 9 As shown in the example graphical user interface shown in , the counting bag geometry model 830 can detect the location of the counting bags, the shape of the pockets of the counting bags, or both, and indicate the pockets of the counting bags (e.g., pockets 920A-920C) in the displayed image, with the pockets indicated by corresponding bounding boxes.

[0071] As described above, there are multiple example embodiments (e.g., variants) of the method 800 for tracking (e.g., by counting) surgical items. In some example embodiments, the surgical item detection model 850 uses a deep learning algorithm or other suitable artificial intelligence model to identify each surgical item (e.g., each surgical fabric) and the location of the surgical item in the cropped counted bag image 842, as generated by the counted bag detection model 840. The deep learning algorithm employed in the surgical item detection model 850 can be trained on a data set (e.g., a reference data set or other training data set) representing different arrays of counted bag images, each array being labeled with the identity of each surgical item in the image. The deep learning algorithm of the surgical item detection model 850 can therefore identify, locate, or both surgical items (e.g., surgical fabric) in each counted bag in the image. The surgical item detection model 850 can generate data about the boundaries of each surgical item in the counted bag. For example, in the case of surgical fabric, this can be performed by identifying the fabric corner 852. Thus, the surgical item detection model 850 can provide the locations of all detected surgical items within one or more counting bags and enable determination of a count of surgical items in one or more counting bags (e.g., a first count, a second count, a further count, or any suitable combination thereof).

[0072] In certain example embodiments, the empty / full pocket classifier model 860 identifies whether each pocket is full or empty by analyzing an image of one or more counting bags, where the pockets have been identified by the counting bag geometry model 830. As generated by the counting bag geometry model 830, a cropped image of each pocket is received by the empty / full pocket classifier model 860. The empty / full pocket classifier model 860 can be trained on a dataset of counting bag images labeled with empty / full pocket identifiers using a deep learning algorithm to identify which pockets are empty and which pockets are full. Examples of suitable object classification networks include: Inception, ResNet, MobileNet, MnasNet, and EfficientNet, which can achieve high performance on object classification tasks. The output 862 of the empty / full pocket classifier model 860 includes information indicating which pockets are empty and which pockets are full in the one or more counting bags. Therefore, the empty / full pocket classifier model 860 enables the determination of the count (e.g., a first count, a second count, a further count, or any suitable combination thereof) of surgical items in one or more counting bags. Fig. 9 and 10An example display corresponding to the output 862 of the empty / full bag classifier model 860 is depicted. Based on a combination of information (e.g., from the surgical object detection model 850, the empty / full bag classifier model 860, or both), the counted bag can be identified as a full counted bag (e.g., as Fig. 9 and 10 ), partially full count bags or empty count bags. Fig. 9 and 10 As shown in , each pocket in the image of the counted bags can be individually identified as including a detected surgical item (such as a surgical sponge (e.g., pockets 920A and 920B)) or as not including a detected surgical item (e.g., pocket 920C). The item counting model 870 synthesizes information generated by one or more of the previously described models to generate various information about each analyzed image. For example, the item counting model 870 can utilize prior information (e.g., determined counted bag geometry, surgical item detection, or both) and perform post-processing to arrive at or provide information about one or more surgical items (e.g., surgical fabrics), one or more counted bags, or both, such as a final surgical item count, counted bag locations, one or more surgical item locations, one or more locations of empty pockets, and one or more locations of full pockets. As shown in FIG. Figure 8 , the item counting model 870 can determine the total item count in each count bag 872 in the image. For example, the item counting model 870 can utilize information from one or more of the previously discussed models (such as the surgical item detection model 850, the empty / full pocket classifier model 860, or both) to determine the number of surgical items in each count bag 872. Furthermore, in some example embodiments, the item counting model 870 combines information from one or more of the other models discussed above in various ways to determine the item count using a combination of multiple methods, and compares the counts derived using each method (e.g., cross-checks the counts) to determine a final surgical item count.

[0073] For example, the item counting model 870 may compare a first estimate of surgical items generated by the surgical item detection model 850 as described above and a second estimate of surgical items generated by the empty / full pocket classifier model 860 as described above to determine a surgical item count (e.g., a first count of surgical items or a second count of surgical items). In other words, the item counting model 870 may compare the locations of surgical items detected within a counted bag obtained using two different types of deep learning models (e.g., by comparing the results of object detection performed by the surgical item detection model 850 with the results of object classification performed by the empty / full pocket classifier model 860). In some example embodiments, the item counting model 870 receives information about surgical items (e.g., surgical fabrics) identified and detected by the surgical item detection model 850 to determine a first estimate of the surgical item count. The item counting model 870 may use the information generated from the empty / full pocket classifier model 860 to generate a second estimate of the surgical item count. If the surgical item counts generated using the two methods match, then the item counting model 870 may use the matched counts as output to provide a count of items for each counted bag 872. If the two counts do not match, the item counting model 870 can respond by employing a unique and novel approach that combines the outputs of the surgical item detection model 850 and the empty / full pocket classifier model 860, which may include leveraging knowledge of the counting bag geometry to correct the counts and improve tracking (e.g., counting) accuracy, such as the approach described below.

[0074] The item counting model 870 may also perform one or more methods of verifying one or more determinations of one or more item count estimates (e.g., if the counts based on the surgical item detection model 850 and the empty / full pocket classifier model 860 do not match). For example, the item counting model 870 may use the total number of pockets identified or counted by the counting bag geometry model 830 and the total number of full or empty pockets identified or counted by the empty / full pocket classifier model 860 to determine how many surgical items (e.g., surgical fabrics) are located in the pockets of the counting bag (e.g., based on the assumption that there is only one surgical item in each full pocket). For example, if the counting bag geometry model 830 identifies ten pockets and one pocket is marked as empty or nine pockets are marked as full, the item counting model 870 may use this information to determine that the surgical item count is nine.

[0075] like Figure 8As shown in , in addition to determining one or more counts of surgical items (e.g., a first or second count of surgical fabrics in a counting bag), additional useful information can also be provided according to method 800 based on information obtained from one or more of the various models. For example, the item counting model 870 can provide a counting bag location 874 within the image, which can be useful for automatically highlighting all counting bags for visualization. As another example, the item counting model 870 can provide the location of surgical items identified or counted within the counting bag, such as an indication of which pockets of the counting bag contain surgical items. This information can be used, for example, in one or more error correction methods, as described below. This information can also be useful for determining whether an item counting protocol (e.g., a protocol for counting surgical fabrics) is met. For example, if there is surgical fabric in the top pocket of the counting bag, but the bottom pocket of the counting bag is empty, this can indicate that the user is not following the approved counting protocol that requires the pockets of the counting bag to be filled from bottom to top. As another example of useful information, the item counting model 870 can provide the location of empty and full pockets in the counting bag image. This is also useful for determining compliance with counting protocols and can be used as verification of surgical item location or quantity of surgical items, highlighting the location of empty pockets in a counting bag for evaluation by the end user, or both.

[0076] Example of reducing counting errors by combining models

[0077] Method 800 (e.g., considered as an overall tracking algorithm for tracking surgical items) can employ a unique and novel approach of combining multiple (e.g., several) models (e.g., deep learning models) such as surgical item detection model 850 and empty / full pocket classifier model 860, as well as leveraging knowledge of counting bag geometry (e.g., from counting bag geometry model 830) to improve tracking (e.g., counting) accuracy by removing or otherwise mitigating different types of errors. For illustration purposes, two example approaches for combining multiple models are described below.

[0078] In one example of combining multiple models to improve overall tracking accuracy, different models can be combined to correct or otherwise address common errors in item counting models. For example, a first type of error occurs when there is a false detection of the presence of a surgical item (i.e., a failure to detect the absence of a surgical item), which results in reduced precision and incorrect counting of surgical items. A second type of error occurs when there is a false detection of the absence of a surgical item (i.e., a failure to detect the presence of a surgical item), which results in reduced recall and incorrect counting of surgical items.

[0079] In some example embodiments of method 800, one or both of these types of errors are addressed by algorithmic checking (e.g., algorithmic error correction). For example, erroneous detections of the presence or absence of surgical items can be addressed, for example, based at least in part on the assumption that the location of the detected surgical item should be well consistent with the location of the detected pocket of the counting bag. Thus, one or more error correction models can identify and reject "identified" surgical items whose determined locations are inconsistent with the determined locations of the counting bag pockets. For example, the error correction model can reject the identified surgical items based on the Euclidean distance between the center (e.g., centroid) of the detected surgical item and the center of the nearest counting bag pocket obtained from the geometry of the counting bag. That is, if the Euclidean distance between the center of the detected surgical item and the center of the nearest counting bag pocket is above a predetermined threshold, then the error correction model can reject the identified surgical item as an error (e.g., incorrectly identified) and accordingly not include this identification in the surgical item count. The correspondence between the remaining (non-rejected) identified surgical items and the counting bag pockets can be established based on the shortest Euclidean distance. That is, the error correction model can match surgical items to specific pockets by determining the identified pocket that is closest to each item (e.g., to verify that each surgical item corresponds to a pocket).

[0080] Once the correspondence is established, the item detection results can be compared to the empty / full pocket classification results. For example, if the surgical item detection model 850 fails to identify a surgical item that is actually present (an instance of the second type of error described above), then the error can be corrected (e.g., by the error correction model) because the empty / full pocket classifier model 860 will identify the corresponding pocket as full. Similarly, if the surgical item detection model 850 incorrectly identifies a surgical item in a pocket that is determined to be empty based on the empty / full classifier model 860, then this instance of detection can be rejected (e.g., by the error correction model).

[0081] One advantage of this algorithm verification approach is that it requires less training data to implement and improves the performance of surgical object tracking compared to using any single model alone (e.g., using only the surgical object detection model 850 or only the empty / full pocket classifier model 860).

[0082] In other example embodiments, multiple models may be combined to form a tracking ensemble network with the goal of improving overall tracking accuracy and reducing counting errors. For example, in addition to using one or more training techniques (such as those described in more detail below), one or more suitable ensemble learning techniques (e.g., snapshot ensembles) may be used to train an ensemble of surgical object detection networks.

[0083] As an illustrative example, a penalty term can be added to the training of the surgical object detection model 850. Specifically, the loss function of the surgical object detection model 850 can be modified by adding a penalty term to automatically penalize the detection of surgical objects whose detected locations are not close (e.g., within a threshold distance) to their corresponding (e.g., nearest) pocket locations. The penalty term can, for example, be related to the distance between the corners of the detected surgical objects (e.g., surgical fabric) and the corners of the counting bag pockets (e.g., as determined using the counting bag geometry model 830). Adding this penalty term directly to the training of the surgical object detection model 850 incorporates prior knowledge of the geometry of the counting bags (e.g., sponge bags or other counting bags), thereby effectively constraining the output of the surgical object detection model 850 to be close to the locations of the pockets of the counting bags (e.g., the grid of pockets). In addition, end-to-end training of the surgical object detection model 850 can be performed to automatically take into account the geometric information of the counting bags that is available during the training phase.

[0084] Furthermore, in this example, an ensemble of multiple item detection networks of different types can be trained by changing the training dataset (e.g., a dataset as further described below) (such as randomly or in a K-fold manner), changing one or more hyperparameters of the network (e.g., hyperparameters such as depth, regularizer, penalty weight, or any suitable combination thereof), and performing an ensemble learning technique (such as snapshot ensemble). Similarly, an ensemble of multiple empty / full pocket classification networks of different types can be trained by changing the network architecture, changing the hyperparameters of the network (e.g., as described above), or any suitable combination thereof.

[0085] Finally, the results from all different object detection networks and all empty / full pocket classification networks can be used in a new stacked generalization network, which can be referred to as a tracking ensemble network. The tracking ensemble network can be trained separately on a set of different validation data (e.g., a "reserved" validation set) to learn how to best combine predictions from each ensemble member. This separate training can help improve the overall accuracy of tracking (e.g., counting), as well as the accuracy of each surgical object prediction, each empty / full pocket prediction, or any suitable combination thereof. For example, the tracking ensemble network can evaluate a 10-dimensional occupancy vector (e.g., for a counting bag of 10 pockets, where "l" represents a full pocket and "O" represents an empty pocket), and accordingly calculate a loss function based on the dot product of the predicted occupancy vector (e.g., from each of the competing ensemble object detection networks and empty / full pocket classification networks) and the ground truth occupancy vector. Thus, when trained, the tracking ensemble network can weight the predictions from each of the ensemble object detection and object classification networks to improve the overall accuracy of method 800 (e.g., the overall tracking algorithm). Therefore, although the above-described grid penalty and ensemble method can be implemented with more training data, it can further improve the performance of method 800.

[0086] Example of collecting training data

[0087] Successful training of one or more of the above models can be aided by collecting a representative dataset of training examples and appropriately labeling the items in the dataset (e.g., surgical items) for the desired end goal. For example, one or more of such datasets can be representative and include variations of images of counting bags that are expected to be found in an operating room or other environment in which the trained model will be used. Thus, a dataset used to train a model (such as one or more of the models in the above method 800 (e.g., tracking algorithm)) can include "good" images as well as "bad" images, as described below.

[0088] In some example embodiments of model training discussed herein, a good image can feature a counting bag from a complete front view (e.g., not a side view, top view, or oblique view) of the counting bag in a properly vertical orientation, with no portion of the bag obstructed by other objects. The good image can be focused with sufficient resolution to read text on the bag. Some imaging devices (such as mobile computing devices) can automatically capture images with sufficient resolution by default. The counting bag in the good image can have intact pocket dividers (e.g., the seams separating the pockets are not torn), and each pocket can contain either zero (0) or one (1) surgical item. One or more models of method 800 can be trained on a collection of good images, including images depicting changes expected during various medical procedures. For example, a training data set can include one or more changes in the counting bag, in the background or setting of the image, or both, as listed in Table 1 below.

[0089]

[0090]

[0091] Table 1

[0092] In some example embodiments of model training discussed herein, bad images may include images that are generally difficult to analyze, such as images in which features of the counting bag or surgical items are difficult to detect (e.g., due to poor lighting, blurry images, or both). Users may send bad images during practice, so it may be beneficial to train one or more models on variations of bad images to create one or more robust counting models that are resilient to imperfect images of the counting bag. These bad images may also be used to help create a "guardrail" algorithm that can warn users of incorrect ways to use the counting bag and can alert users when bad images are taken to notify the user to provide a better image (e.g., a new, clearer image). Examples of bad images include images in which a portion of the bag is obscured by a foreground object (e.g., by a user's hand, a track, a tube, or any suitable combination thereof), images taken from an angle that makes it difficult to see a pocket divider or a pocket seal (e.g., between two or more surgical items), images of a counter bag with a torn pocket divider or a torn pocket seal (e.g., such a tear creates a single large pocket where there should be two individual pockets), images with multiple surgical items in a single pocket, images taken under incorrect lighting conditions or any other conditions that may produce an image that is difficult for the algorithm to analyze and extract features, and images having any one or more of the foregoing aspects in combination.

[0093] Examples of other verifications for counting

[0094] In some example embodiments, the methods discussed herein may include one or more additional verification steps to provide further checks (e.g., independent checks) of method 800. Providing one or more additional checking methods may improve the accuracy of the surgical item inventory. Such additional methods may include prompting a user to manually verify the count of surgical items in a counting bag. Such additional methods may include requesting a user to verify information generated by one or more of the various sub-models in method 800. For example, one or more of such additional methods may include requesting a user to verify that method 800 has correctly identified a pocket, has correctly identified a counting bag, has correctly identified a full pocket, has correctly identified surgical fabric in a pocket, or any suitable combination thereof. For example, Fig.11 An example embodiment of a display on a mobile device is depicted, wherein the display prompts a user to verify that the number of surgical items (e.g., surgical sponges) identified and counted by method 800 is correct. Fig.11 As shown by the box surrounding the surgical item in , one or more of such additional methods may include displaying information to the user indicating what the method 800 has identified as a surgical item.

[0095] In certain example embodiments, the methods discussed herein may include using weight information to verify the count of surgical items. For example, a rod mounted with a counting bag may be coupled to a scale. Each time a surgical fabric is placed in the counting bag, the scale coupled to the rod can sense the change in weight and count the surgical items accordingly based on the number of weight increases sensed by the scale. The scale may be coupled to the rod in any suitable manner. For example, the mounting rod, the counting bag, or both may be placed on the scale (e.g., the mounting rod, the counting bag, or both may be located on top of the scale to measure the change in weight when the item is added to the counting bag). Alternatively, the scale may take the example form of a strain gauge coupled to the mounting rod and coupled to the counting bag (e.g., one end of the strain gauge is suspended on the mounting rod and the counting bag is suspended at the other end of the strain gauge). In some example embodiments, instead of or in addition to the weight information generated via one or more other methods, weight information may be generated from one or more surgical item collection containers. For example, the collection container may include a weight measurement system. When a user removes a surgical item from the collection container, the weight measurement system senses the decrease in weight and determines that the surgical item fabric has been removed from the collection container. The processor can record the number of weight reductions to determine the count of surgical items removed from the collection container and, therefore, the count of surgical items previously present in the collection container. Thus, the use of one or more collection containers can provide an additional check on the count of surgical items (e.g., surgical fabrics).

[0096] In various example embodiments, the methods discussed herein may include using time data from the removal of surgical items from a collection container, and placing the surgical items in a counting bag may provide additional verification of the surgical item count. For example, a processor may generate a timestamp each time the weight measurement system senses that a surgical item is removed from a collection container. In addition, a processor may generate a timestamp each time the weight measurement system senses that a surgical item is added to a counting bag. One or more processors may be configured to compare a timestamp of the removal of a surgical item from a collection container (e.g., a removal timestamp) and a timestamp of the addition of a surgical item to a counting bag (e.g., an addition timestamp) to provide verification of the surgical item count (e.g., additional verification). If the removal timestamp and the addition timestamp are substantially consistent, information indicating that the surgical fabric count is correct is provided.

[0097] Example of determining the middle count

[0098] In some example embodiments, determination of the second count includes determining an intermediate count of surgical items during the course of a medical procedure (e.g., a surgical procedure). In alternative example embodiments, determination of the second count includes determining a final count of surgical items after the medical procedure is completed. For example, the methods discussed herein for tracking surgical items (e.g., method 800) may include comparing the second count (e.g., as an intermediate count or as a final count) to the first count to inventory the surgical items at any point during the course of the medical procedure. Various example embodiments of such methods may include using any one or more of the above-described algorithms to determine the second count, whether it is an intermediate count or a final count.

[0099] Determination of the intermediate count of surgical items may include updating an index counter of the surgical items. In some example embodiments, updating of the index counter includes analyzing an image of a counting bag that includes one or more surgical items (e.g., surgical fabrics) in one or more pockets of the counting bag. One or more images of the counting bag may be generated as the counting bag is filled with surgical fabrics by a user. One or more processors may receive the one or more images of the counting bag and may process the one or more images using one or more computerized techniques, such as one or more of the techniques described above, to identify and count the surgical fabrics in the one or more counting bags.

[0100] In some example embodiments, instead of analyzing an image of a counting bag including surgical items or as a supplement thereof, the methods discussed herein include lifting the surgical items into the field of view of the camera. For example, a user can lift a surgical item (e.g., a surgical fabric, such as a surgical sponge) in front of the camera, and the camera can generate an image depicting the surgical item. The image can be received and analyzed by one or more processors configured to detect the presence of the surgical item depicted in the image. The image of the surgical item can be used to track the surgical item. For example, each time a surgical item is lifted in front of the camera and identified by one or more processors, an index counter can be updated. In some example embodiments, an index counter updated using an image analysis of the surgical item lifted in front of the camera can be compared with another index counter (e.g., as described above) updated using an image analysis of the surgical item in one or more counting bags to verify the index count of the surgical item. Fig.13 An example of a graphical user interface is depicted in which images of individual surgical items (e.g., surgical fabrics) are obtained and analyzed to update a count of such surgical items. Specifically, a user 1210 holds up a surgical fabric 1220 in front of a camera that captures an image (e.g., a depth image, a color image, or both) so that one or more processing components (e.g., a processor) can identify the surgical fabric 1220 in the image and include the imaged surgical fabric 1220 in the item count 1230. Example systems and methods for achieving the above purposes are described in more detail in U.S. Patent No. 8,897,523 and U.S. Patent Publication No. 2017 / 0186160, each of which is incorporated herein by reference in its entirety. In addition, as described below, one or more methods of estimating blood loss can be used in conjunction with one or more methods for tracking surgical items discussed herein, and accordingly can include a user holding one or more surgical items in front of a camera for imaging.

[0101] Determination of an intermediate count of surgical items may also include updating an index counter with the amount of surgical fabric in one or more containers (e.g., in one or more counting bags) as the containers are filled and imaged. This process may be repeated throughout the medical procedure to continuously update the index counter and maintain a running count of surgical items (e.g., surgical fabric) that have been loaded into the one or more containers at any given point in time. In some example embodiments, it may be desirable to compare the intermediate count (e.g., the current value of the index counter) to the first count of surgical items. For example, a user may want to know how many surgical items have been loaded into the collection of counting bags and have been counted, and how many surgical items are currently in use or not counted.

[0102] Certain example embodiments of the methods discussed herein include displaying or otherwise providing intermediate counts to a user at any suitable point during a medical procedure.Displaying intermediate counts can provide the user with useful information about how much surgical fabric has been counted at a particular point in time. Fig.14 An example of a graphical user interface 1300 displaying information regarding intermediate counts is depicted. Fig.14 The box 1310 of the middle screen shows the total number of surgical sponges counted (e.g., "17") compared to the first count of sponges (e.g., "40"). In addition, information related to the type of surgical sponges counted (e.g., brand, size, functional type, etc.) is displayed in boxes 1320, 1330, and 1340. The graphical user interface 1300 also provides a feature that allows the user to update the first count by selecting the "Count Up" icon 1350 and determine the final count by selecting the final count icon 1360.

[0103] Example of detecting duplicate images

[0104] Some example embodiments of the methods discussed herein may also include determining whether a particular counting bag has been imaged and analyzed before. Because a user can take images of multiple counting bags, multiple surgical items, or both, there is the possibility of accidentally imaging an individual counting bag, an individual surgical item, or both multiple times, which can result in inaccurate counting of surgical items. In order to counteract these consequences of accidentally repeating imaging, one or more of the images may be analyzed to determine a measurement of the possibility that the image depicts the same counting bag as another image. An example of a method for determining whether a counting bag has been counted may include measuring or otherwise determining the possibility that a first image and a second image have image areas that each depict at least a portion of the same fabric (e.g., in the same counting bag). An example method for measuring the possibility that the first and second images have image areas that each depict at least a portion of the same fabric is described in detail in PCT Application No. PCT / US2017 / 068234, which is incorporated herein by reference in its entirety. An example method for determining whether an image of a particular counting bag has been received and analyzed may include receiving a first image in which a first image area depicts a counting bag, receiving a second image in which a second image area depicts a counting bag, and defining at least one classification feature based at least in part on a first aspect of the first image, a second aspect of the second image, or both. Such example methods may also include measuring a likelihood that the first and second image regions depict at least a portion of the same counting bag, wherein the measure of likelihood is based at least in part on a classification feature, such as by using a machine learning classification algorithm. Such example methods may include comparing the measured likelihood of duplicate imaging to a predetermined threshold, wherein the threshold is or includes a cutoff value for classifying an image pair as "not a duplicate" or "potentially a duplicate." A similar process may be performed to measure a likelihood that the first image region and the second image region depict at least a portion of the same surgical fabric or another surgical item.

[0105] If method 800 determines that the possibility of repeated imaging is higher than a predetermined threshold, then subsequent actions can be prompted. For example, although generally speaking each new image of the counting bag or surgical item obtained can correspond to an increase in the count of surgical items (e.g., surgical items that have been extracted from the patient or from the sterile area of ​​the surgical procedure for the second count), the count can be adjusted based at least in part on the possibility that the counting bag, surgical items, or both have been imaged more than once. The detection of potential repeated images of surgical counting bags or surgical items can, for example, trigger a prompt to the user to confirm whether the counting bag, surgical items, or both have been imaged more than once. As another example, the detection of potential repeated images of the counting bag or surgical items can automatically prevent the counting of the surgical items from being incremented. An example method for determining whether the counting bag has been imaged can include the use of an identification tag. For example, the counting bag, surgical items, or both can include a scannable tag (e.g., an RFID tag or a barcode), and a scanner can be used to determine whether the counting bag has been imaged. The scanner can be a separate scanning device or can be incorporated into a mobile device. As an alternative or in addition to one or more techniques involving image analysis, one or more example methods involving scanning labels or bar codes may be employed.

[0106] Example of determining the terminal count

[0107] In some example embodiments, the second count may include a final count of surgical items. For example, example embodiments of the methods discussed herein may include determining a final count of surgical items and comparing the final count to a first count of surgical items. The final count may be distinguished from intermediate counts, for example, based on the fact that a final count refers to a count of surgical items counted at a point in time when surgical items are no longer introduced or used in a procedure. For example, the final count may be taken at the end of a medical procedure. The end of a medical procedure may be indicated in any suitable manner. For example, the end of a medical procedure may be after the surgical site has been closed, after all surgical procedures other than closure of the surgical site have been performed on the patient, once the patient is ready to be released from the operating room, or any other point in time suitable for use as a final indication. In some example embodiments, the determination of the final count may be prompted by a user. For example, a mobile application or other computerized program may allow a user to indicate when to indicate the final count (e.g., selecting Fig.14In some example embodiments, the mobile application may prompt the user to verify that the count of the surgical item should be marked as a final count based on an indication. For example, when the index counter reaches the value of the first count, the mobile application may prompt the user to confirm whether the current value of the index counter should be indicated as a final count. In other example embodiments, the mobile application may require the user to indicate whether the index counter should be indicated as a final count after a certain amount of time has passed.

[0108] Examples of providing notification based on the first count, the second count, or both

[0109] Various example embodiments of the method for tracking surgical items discussed herein may include comparing a first count of surgical items with a second count of surgical items. For example, some example embodiments of such methods for tracking surgical items include comparing an intermediate count (e.g., among one or more intermediate counts) with a first count, as described above. Certain example embodiments of such methods include comparing a final count with a first count to determine whether the first and final counts match, and whether all surgical items introduced into the operating room have been counted accordingly. Comparing the final count with the first count can assist medical staff in confirming that all surgical fabrics or other surgical items have been counted, rather than being retained or left in the patient's body by the patient. Some example embodiments of such methods may include notifying a user whether the first count matches the second count (e.g., the final count or the intermediate count). For example, certain example embodiments of such methods include displaying a summary report of a surgical item count measurement, wherein an indication of the first count, the second count, and whether the first count matches the second count is displayed to one or more users.

[0110] Fig.15 An example of a graphical user interface displaying a summary of surgical fabric data is depicted. Fig.15The screen in shows the number of "counted" (e.g., introduced into the operating room) fabrics, the number of "scanned" (e.g., imaged in a counting bag) fabrics, and the number of unused (e.g., introduced into the operating room but not yet imaged in a counting bag) fabrics. In some embodiments, unused surgical fabrics are placed in one or more counting bags in the same manner as used surgical fabrics. In some other embodiments, unused surgical fabrics are counted using a different method, such as being identified as unused in one or more counting bags and marked as such on a display screen accordingly. In some example embodiments, the methods discussed herein include notifying a user (e.g., by providing a notification to the user) that the compared counts match. Such a notification may include a brief message indicating to the user that the counts match, or the notification may be symbolic. For example, the notification may include displaying a check mark or equal sign or any other suitable symbol next to the first and final counts, displaying a color indication such as the screen turning a particular color (e.g., green) when the counts match, playing an audio indication that the first count matches the final count, some form of flashing or vibration if the final counts match, or any suitable combination thereof. In addition, as described below with respect to Fig.16 As further described, a summary report may be provided to one or more users for personal reference, hospital records (e.g., accountability purposes), or both, and the summary report may indicate a first count and a final count for each of one or more types of surgical items. For example, the summary may include or indicate a representation of a comparison between the first count and the second count, such as Fig.15 The “check” number shown in .

[0111] In certain example embodiments, the methods discussed herein include providing a notification to a user that the compared counts do not match. In the event that the final count of surgical items does not match the first count of surgical items, the notification may include a reminder to the user. The reminder may be in the form of an example display (e.g., on a mobile device) indicating to the user that one or more surgical items have not been counted. In addition to the aspect of notifying that the compared counts do not match, the display may include any one or more of the elements described above with reference to the notification that the compared counts do match. In some example embodiments, notifying the user that the first count of surgical items does not match the final count includes providing a predetermined protocol for proceeding in the event of an unmatched count. For example, the user's notification may include displaying (e.g., on the screen of a mobile device) the steps of a predetermined protocol, providing accompanying audio prompts for both, about what to do when the first count of surgical items does not match the final count. Although medical personnel may be familiar with one or more protocols on how to proceed in the event that surgical items are not counted, providing an additional presence (e.g., an additional instance of the protocol) that is not affected by the potential discomfort of uncounted surgical items can be beneficial for helping users comply with the protocol. This can result in fewer instances of surgical items being left behind in a patient, better detection of surgical items being left behind when they do occur, more effective responses in the event that a surgical item is left behind, or any suitable combination thereof. In the event that the first count and the final count do not reconcile, some example embodiments of the methods discussed herein can include automatically sending a notification to alert additional personnel, triggering a response protocol, or both. For example, an unreconciled first count and final count can trigger a notification to hospital administrators, to radiologists (e.g., to obtain pre-approval to generate an x-ray of the patient), to x-ray technicians, to additional medical personnel to help correct the situation, or to any other suitable personnel.

[0112] If the first count does not match the final count, some example embodiments of the method for counting surgical items discussed herein may include executing a predetermined protocol to determine whether there are surgical items left behind (e.g., left behind in the patient or retained by the patient). The execution of the predetermined protocol may be completed with the assistance of one or more computerized tools, such as a device and a display that guides the user through the protocol. The protocol may first include notifying all medical personnel in the operating room of surgical items that may be left behind. The protocol may then include recounting all surgical items in the operating room and checking the room for surgical items that may be misplaced. The recounting of surgical items may include operating one or more computerized counting systems, using one or more manual counting methods, or any suitable combination thereof. Examples of manual counting methods include recounting the number of counting bags, recounting the number of surgical items in each counting bag, checking each pocket to determine whether there is more than one surgical item in the pocket, or any suitable combination thereof. Checking surgical items in the operating room may include checking trash (e.g., checking one or more trash cans), checking one or more collection containers, checking the area around the patient, checking one or more counting bags for multiple items placed in the same pocket, or any suitable combination thereof. If the first count still does not match the final count after recounting the surgical items and checking for misplaced surgical items, the protocol may include taking an x-ray image or another internal image of the patient to detect the presence of surgical items left inside the patient. The taking of the x-ray image may include alerting the radiologist that an x-ray image needs to be taken and may require approval from the radiologist to proceed. The taking of the x-ray image may additionally or alternatively include notifying other appropriate medical personnel. The protocol may also include notifying other administrative staff of the hospital administration of the unreconciled surgical item count.

[0113] Example of attach method

[0114] The computerized systems and methods (e.g., algorithms) described herein have broad applicability to effect various improvements in medical procedures. Some example embodiments of such methods may include tracking user compliance, guiding a user through a counting protocol, tracking surgical item usage throughout a procedure, receiving patient and / or operating room information, triggering additional actions upon receipt of certain indications, or any suitable combination thereof, as described in detail below.

[0115] Certain example embodiments of the methods discussed herein include tracking compliance with one or more protocols. Tracking of user compliance may include tracking user compliance with a fabric counting protocol using a counting bag. As described above, one or more predetermined procedures may govern one or more desired methods of placing items in one or more counting bags. According to one or more applicable protocols, operating room personnel may be instructed to fill the pockets of each counting bag from bottom to top and from right to left. In some cases, one or more algorithms developed according to one or more of the above methods may be used to assess whether a user has complied with the bag filling protocol. Such an algorithm may receive one or more input images and determine compliance based on the one or more images. For example, a series of images taken over time may be analyzed to determine in what order the counting bags are filled. In other cases, video analysis may be used to identify and track the order in which the counting bags are filled. The algorithm may analyze a temporal sequence of images or videos to determine the order in which the user fills the pockets of the counting bag and assess the user's compliance with the applicable one or more protocols.

[0116] Some example embodiments of the methods discussed herein include providing a notification upon detecting a failure to comply with a counting protocol. For example, such methods may include displaying an alert to a user as a reminder of the correct protocol, sending a notification to hospital administrators, other hospital personnel, or both, indicating an incorrect application of a fabric counting procedure, or any suitable combination thereof. As another example, such methods may include triggering an alert to other medical personnel to intervene in an incorrect filling procedure. For example, upon detecting a misplaced surgical item (e.g., a surgical item placed in a pocket of a counting bag in contradiction to a predetermined procedure of an applicable protocol), such methods may notify a supervisor or a companion to intervene in the bag filling process.

[0117] Certain example embodiments of the methods discussed herein include prompting the user to enter their name in conjunction with the use of a counting or tracking algorithm. The mobile application may prompt the user to enter their name or other unique identifier before or after operating the application to take an image of the counted bag for processing. For example, the mobile application may prompt the user to scan his or her ID badge to provide such identification information. Compliance data may be paired with identification information to enhance user accountability. Pairing compliance information with user identification information may provide the benefit of allowing hospital administration to implement training or disciplinary protocols for users who do not follow proper item counting procedures.

[0118] Various example embodiments of the methods discussed herein include using a counting bag to guide a user through a counting protocol (e.g., a fabric counting protocol). One or more computerized techniques can be used to indicate to a user which pocket of the counting bag to fill next or to identify a pocket that the user forgot to fill, for example, by presenting one or more visual cues, one or more audio cues, or any suitable combination thereof. For example, a display screen can provide a visual indication that guides one or more users to the next pocket to fill (e.g., by providing a box or circle around the pocket, displaying an arrow as a directional guide, or both). Thus, such methods can guide a user through a protocol for filling a counting bag.

[0119] Some example embodiments of the methods discussed herein include providing one or more other types of tracking metrics for surgical items. Such methods can be used to collect information about the medical procedure at various points in time throughout the medical procedure. For example, such methods can include tracking used and unused surgical items. Tracking used and unused items can include a user manually separating or sorting items based on whether the items have been used. For example, a user can place used items in one container and unused items in a separate container. Various methods (such as one or more of the above-mentioned computerized techniques) can be used to identify used and unused containers and count used and unused items. Such methods can include taking an image of a portion of an operating room with used and unused containers, and one or more of the image analysis methods described herein can be used to extract information from the image. Identifying and counting used and unused items during a medical procedure can provide one or more useful metrics about the medical procedure. A single snapshot at a specific point in time in a medical procedure (e.g., at the end of a medical procedure) can provide information about how many items were taken out of their packaging and how many items were actually used. A series of images taken throughout a medical procedure can be used to identify and count used versus unused items, and can provide information about how items are used over time in that medical procedure.

[0120] Certain example embodiments of the methods discussed herein include receiving information related to a medical procedure. Such methods may allow a user to input patient information (e.g., patient data), such as patient demographics or characteristics (e.g., height, weight, gender, age, etc.), a medical procedure to be performed, the expected length of the medical procedure, whether the medical procedure is scheduled or urgent, one or more pre-existing conditions, other relevant information, or any suitable combination thereof. Such methods may include receiving, displaying, or both various information about one or more environmental parameters, such as which operating rooms are in use and which are available, whether certain equipment is available in a particular operating room, which medical staff is currently working, other suitable parameters, or any suitable combination thereof.

[0121] Some example embodiments of the methods discussed herein include receiving information that can trigger an event or protocol (e.g., one or more requests or initiations of one or more precautions in response to determining that there is a relatively high risk that one or more surgical items are left behind in the patient). Such methods may include allowing a user to request additional items or equipment for use in a medical procedure. Such methods may include triggering a protocol after receiving information about one or more of certain indications. There may be a situation where one or more specific patient parameters combined with one or more of the occurrences of a specific event during the procedure indicate a high risk of a specific event occurring. For example, a patient with a BMI within a certain range combined with a threshold number of surgical items expected to be used during a medical procedure (e.g., when a first count exceeds a predetermined number) may indicate an increased risk of surgical items being left behind. Therefore, such methods may include using patient data in combination with surgical item counts to trigger one or more additional protocols when a threshold number of surgical items are counted. For example, such a protocol may involve reminding an X-ray technician to prepare appropriate equipment to take an X-ray image of the patient, providing a notification to a radiologist to obtain prior approval to take an X-ray image of the patient, or both.

[0122] Certain example embodiments of the methods discussed herein include displaying or otherwise providing information related to a medical procedure. Such methods may include providing, displaying, or both summary reports. The summary report may be stored, for example, as part of a patient's electronic medical record, as evidence of compliance with one or more surgical item tracking protocols, or both. Any suitable information may be included in the summary report. As described above, information related to one or more counts of surgical items, one or more comparisons of the counts of surgical items, or both may be displayed in the summary report. Additionally or alternatively, one or more images of individual surgical items, individual counting bags, or both may be stored, displayed, or both as a record (e.g., in the summary report). However, additional information about the medical procedure may also be displayed, such as the type of medical procedure, the length of the medical procedure, patient information, doctor information, hospital information, or any suitable combination thereof. A blood loss measurement may also be displayed. For example, in the case of using analysis of surgical fabrics to monitor patient blood loss, an estimate of blood loss may be displayed in the summary report. The summary report may include a display of one or more symbolic indications of the results of the medical procedure. For example, when the amount of blood loss is within a given range, the display may be a particular color (e.g., when the amount of blood loss is within a predetermined range defined between predetermined thresholds, the display may be green, and when the amount of blood loss is not within the predetermined range, the display may be red). One or more summary reports may be generated at any suitable point during the medical procedure. The one or more summary reports may be displayed on a digital screen (e.g., a display screen of a mobile device such as a phone or tablet), stored in a memory (e.g., as part of an electronic medical record for a patient or for a hospital) and saved for later review, sent digitally to a patient or medical staff, sent to a printer and printed, or any suitable combination thereof. Fig.16 An example of a summary report is depicted that includes information regarding compliance with the counting protocol, as well as additional information regarding the medical procedure performed. Fig.16 The summary report in includes information regarding the case number, the location of the case (e.g., which operating room was used), the start and end times of the medical procedure, the duration of the medical procedure, the number of surgical items counted (e.g., surgical fabrics, represented as "sponges"), and the name of the person who performed the final count of the surgical items. According to various example embodiments, the summary report may display any suitable information about the medical procedure, such as any of the information described herein.

[0123] Some example embodiments of the methods discussed herein provide one or more ways to enter an "emergency mode" in which some functionality of the method or system may be altered. For example, a fabric counting algorithm (e.g., implemented via a mobile application) may allow a user to select an emergency mode, which may indicate that a counting protocol is no longer being followed. This may trigger one or more of a variety of actions, such as stopping the collection of item count data, refraining from tracking user compliance with a counting protocol, or alerting hospital staff to an emergency situation.

[0124] In certain example embodiments of the methods discussed herein, one or more images (e.g., depicting one or more surgical items, one or more counting bags, or any suitable combination thereof) may be analyzed to estimate patient status (e.g., total blood loss, blood component loss, or both). Fig.12 As depicted in , computing device 1100 may use imaging system 1122 having one or more cameras 1120 to acquire one or more images 1102 of surgical fabric. One or more images 1102 may capture and depict a field of view 1108 including surgical fabric 1106a (such as a piece of surgical gauze, surgical sponge, or surgical towel). Surgical fabric 1106a may be held by a user. Fig.12 As shown in , captured images including image 1106b of surgical fabric 1106a can be displayed on a display 1118, such as on computing device 1100. One or more images 1102 can include a color image that can be analyzed to estimate the amount of blood, blood component content, or both in the imaged surgical fabric. For example, as described in U.S. Pat. No. 8,897,523, incorporated above, color-related features (e.g., intensity) in the color image can be transformed into an estimate of the amount of blood or blood component (e.g., a mass of hemoglobin, a volume of whole blood, or both) in surgical fabric 1106a, either locally or via a remote computer system to perform such processing.

[0125] Moreover, one or more images 1102 may include a depth image (e.g., containing depth information based on infrared data, stereo images, or both). The depth information from such a depth image may make it easier to distinguish objects located in the foreground from other objects in the background. The depth image may be analyzed (e.g., as described in U.S. Patent Publication 2017 / 0186160, incorporated above) to confirm the presence of surgical fabric 1106a in the image, after which the computing device 1100 may index a local surgical fabric counter. The local surgical fabric counter may be used to verify a count of surgical items determined by any one or more of the above methods (e.g., using image analysis of a counting bag) or as an independent method of counting surgical items.

[0126] Example system for tracking surgical items

[0127] Also described herein is an example system for tracking surgical items. Such a system for tracking surgical items may include one or more components configured to determine a first count of surgical items. For example, such a system may include one or more imaging systems, one or more weighing systems (e.g., a scale), or any suitable combination thereof, for determining a first count of surgical items. Such a system may include one or more components configured to determine a second count of surgical items. For example, such a system may include one or more counting bags, each having one or more pockets, configured to hold one or more surgical items. Such a system may include additional equipment for holding and processing one or more surgical items. For example, such a system may include one or more collection containers for holding used surgical items, one or more brackets for holding one or more counting bags, and one or more brackets for holding one or more mobile computing devices for imaging, image processing, or both. Such a system may include one or more imaging systems configured to generate one or more images of one or more counting bags or other containers for surgical items. The imaging system for obtaining the first count may be the same as the imaging system for obtaining the second count. Such a system may include one or more processing components for determining the first count, the second count, or both, and comparing the first count with the second count. For example, such one or more processing components may be configured to analyze one or more images to identify surgical items appearing in the images, count surgical items in the images, or both. Such one or more processing components may utilize one or more computing techniques (such as computer vision, machine learning, deep vision, or any suitable combination thereof) to analyze images to help track surgical items. Such components (e.g., processing components, imaging components, or both) may be implemented via one or more mobile computing devices, each of which may include one or more cameras or other image sensors, one or more processors, one or more displays, or any suitable combination thereof.

[0128] According to some example embodiments of the system discussed herein, one or more of such computing devices may include, for example, a mobile device, such as a mobile phone, a smart phone, a tablet computer, a laptop computer, or any suitable combination thereof, each of which may include a camera, a processor, a display, or any suitable combination thereof. However, in certain example embodiments, some or all of the system components discussed herein may be separated as discrete interconnected devices or other components. For example, the camera, the display, or both may be substantially located near the counting bag (e.g., in an operating room), while the processor may be located at a remote location (e.g., in an operating room separated from the camera, the display, or both, or outside the operating room), communicating with the camera and the display via a wired or wireless communication network. One or more of such computing devices may be used to display information to one or more users. For example, a computing device may include one or more display screens that may display information related to: one or more counts of surgical items, one or more patient parameters, one or more medical procedures, a hospital, an operating room, patient blood loss, user compliance, any suitable combination thereof, or any of the above information. One or more of such computing devices may be configured to provide audio information to one or more users.

[0129] One or more processors may be configured to perform some or all of any one or more of the methods described herein. For example, one or more of such processors may be configured to receive one or more images, execute one or more computing algorithms (e.g., computer vision, machine learning, deep learning, or any suitable combination thereof) to analyze such images, receive and analyze data (e.g., from user input or from computer memory), trigger one or more responses, or any suitable combination thereof. One or more of such processors may be configured to receive and analyze one or more images depicting one or more surgical items, one or more images depicting one or more counting bags, weight information, timestamp data (e.g., paired with weight information), user input, or any suitable combination thereof to determine the first and second counts of surgical items. One or more of such processors may be configured to, for example, execute one or more machine learning algorithms to analyze one or more images to determine the number of surgical items shown in one or more images. One or more of such processors may be configured to update the intermediate count of surgical items based on the number of surgical items appearing in one or more images. One or more of such processors may be configured to determine the final count of surgical items, and such determination of the final count may include receiving manual input from a user. One or more of such processors may be configured to compare a first count of surgical items with a second count of surgical items (e.g., an intermediate count or a final count). One or more of such processors may be configured to generate a notification (e.g., a reminder) based on the comparison of the first count with the second count. If the first count does not match the second count, one or more of such processors may be configured to generate or display an alert or other notification. Additionally or alternatively, one or more of such processors may perform one or more additional tasks related to determining counts of other items (e.g., used items versus unused items), tracking compliance information, receiving user input, triggering one or more responses to situations that occur during a medical procedure, sending information and alerts to appropriate personnel, or any appropriate combination thereof.

[0130] In general, any one or more of the processors discussed herein may be configured to execute instructions stored in a memory so that when it executes instructions, the one or more processors perform one or more of the methods described herein. Instructions may be performed by a computer executable component integrated with a user computer, mobile device, wristband, smart phone, or any suitable combination thereof, a small application, a host, a server, a network, a website, a communication service, a communication interface, a hardware element, a firmware element, a software element, or any suitable combination thereof. Instructions may be stored in a memory or other computer-readable medium, such as RAM, ROM, flash memory, EEPROM, an optical device (e.g., CD or DVD), a hard drive, a floppy disk drive, or any suitable device.

[0131] One or more processors configured to perform some or all of any one or more of the methods discussed herein may be integrated into a handheld or mobile computing device or may be implemented as an external processor. In some example embodiments, one or more of such processors may be incorporated into a computer device or system (such as a cloud-based computer system, a large computer system, a grid computer system, or other suitable computer system). One or more of the systems discussed herein may include a communication component that transmits data to one or more external processors. Additionally or alternatively, one or more of such processors may be incorporated into a remote server that receives images of surgical items, reconstructs or analyzes images as described above, and transmits information related to tracking of surgical items.

[0132] As described above, one or more example embodiments of the systems discussed herein may include one or more imaging systems. For example, such a system may include a camera that generates one or more images (such as a collection of one or more still images or a portion of a video feed) of a surgical item (e.g., a surgical item contained in one or more counting bags). The one or more images may include a color image, a depth image (e.g., infrared, stereo, ultrasound, etc.), a hyperspectral image, another suitable type of image, or any suitable combination thereof. The camera may include at least one optical image sensor (e.g., CCD, CMOS, etc.) that captures a color optical digital image having red, green, and blue (RGB) color components of pixels, or one or more other suitable optical components. For example, the camera may include a single image sensor paired with a suitable corresponding optical device, a filter (e.g., a color filter array such as a Bayer pattern filter), or both. As another example, the camera may include multiple image sensors paired with a suitable corresponding optical device (such as at least one prism or a diffractive surface) to separate white light into separate color channels (e.g., RGB), each color channel being detected by a corresponding image sensor. One or more imaging systems, one or more computing processing tools, or any suitable combination thereof may be configured on the same device, or the imaging system may be a separate component that transmits image data to one or more other components. One or more of such imaging systems may be part of a mobile device (e.g., a camera on a smartphone), which may or may not be configured to implement one or more of the above-described processing components. Such an imaging system may be configured to transmit images to one or more processors for analysis, a database storing images (e.g., cloud storage), or both.

[0133] One or more example embodiments of the systems discussed herein may include a display for displaying or otherwise communicating information to a user (e.g., a doctor, nurse, other medical personnel, or any suitable combination thereof), including patient information, one or more images of surgical items, one or more quantitative metrics associated with the inventory of surgical items, one or more notifications of a comparison between a first count and a second count (e.g., a final count) of surgical items, a predetermined protocol for a user to follow in the event that the first count and the second count of the surgical items do not match, one or more measurements of the likelihood of repeat imaging of surgical items (e.g., surgical fabrics), a value of an index counter for surgical items, one or more warnings or prompts indicating potential repeat imaging of surgical items to the user, one or more summary reports, or any suitable combination thereof. The display may include a screen on a handheld or mobile device, a computer monitor, a television screen, a projector screen, or other suitable display. The display may be integrated in the same device as one or more components of the system. Additionally or alternatively, the display may include an independent separate monitor, screen, or other suitable display.

[0134] According to some example embodiments, the display may be configured to display a user interface (e.g., a graphical user interface) that enables a user to interact with the displayed information. For example, the user interface may enable a user to manipulate (e.g., scale, crop, rotate, or any suitable combination thereof) one or more of the images or manually define an image area that depicts at least a portion of a surgical item. As another example, the user interface may enable a user to select one or more display options (e.g., fonts, colors, languages, or any suitable combination thereof), select content (e.g., patient information, quantified measurements or other fluid-related information, reminders, etc.), or both. As yet another example, the user interface may enable a user to select one or more images for deletion due to repeated depictions of surgical items, or to rotate, flip, or otherwise manipulate one or more images on the display. The display may be user interactive and include a resistive or capacitive touch screen that responds to skin, stylus, or other user contact. The display may interact with the user via a cursor controlled by a mouse, keyboard, or other input device. For example, the display may allow a user to manually input information indicating a first count, a second count, patient information, or any suitable combination thereof.

[0135] One or more example embodiments of the systems discussed herein may include a speaker or other suitable audio system that conveys information related to an inventory of surgical items (e.g., a first count or a second count), the value of an index counter for a surgical item, one or more notifications of a comparison between a first count and a second count (e.g., a final count) of a surgical item, a predetermined protocol for a user to follow in the event that the first count and the second count of a surgical item do not match, one or more measurements of the likelihood of repeat imaging of a surgical item, one or more warnings or prompts indicating potential repeat imaging of a surgical item to a user, one or more summary reports, or any suitable combination thereof. When the first count and the second count (e.g., the final count) do not match, the display, the audio system, or both may provide one or more notifications or reminders, which may be useful for prompting certain actions in response, such as prompting the user to begin a specified protocol. As another example, the display, the audio system, or both may provide one or more reminders or prompts to a user indicating the likelihood that two or more images depict the same surgical item, which may be useful for prompting the user to confirm whether repeat imaging has occurred, reassess the location of counted or uncounted surgical items, or any suitable combination thereof.

[0136] One or more example embodiments of the systems discussed herein may include items and devices for use in surgical item counting methods. For example, such a system may include one or more collection containers, one or more counting bags for tracking and counting surgical items, one or more racks configured to hold counting bags, one or more racks configured to hold one or more imaging systems (e.g., mobile devices), one or more weight detection components coupled to one or more components of the system, or any suitable combination thereof.

[0137] One or more example embodiments of the systems discussed herein may include one or more collection containers for collecting surgical items (such as surgical fabrics). Before counting (e.g., after the fabrics are used), medical staff may use the collection container to place the used items. The collection container (e.g., a "kick bucket") may be of any suitable size, shape, or material. The collection container may have a circular cross-section. The collection container may be or include a small barrel-shaped container and may include a liner. The collection container may be made of metal, plastic, or any suitable material. The liner of the collection container may be of any suitable material. For example, the liner may be a transparent plastic liner. The transparent liner may provide the benefit of facilitating easy visualization of dirty and clean surgical items. The collection container may be of any suitable size and may hold any suitable number of surgical items. Some example embodiments of such systems may include one or more collection containers placed in different locations throughout the operating room. For example, one or more collection containers may be placed near the operating table to allow a doctor or nurse to easily discard surgical items after use.

[0138] One or more example embodiments of the systems discussed herein may include one or more counting bags for holding and counting surgical items. For example, as described above, the counting bag may include a backing and one or more pockets configured to hold one or more surgical items. The counting bag may be of any suitable size, and such a system may include one or more counting bags of different sizes. For example, the sizes of different counting bags may be designed to hold different types of surgical items (e.g., surgical fabrics of different sizes). Different types of surgical fabrics have different sizes, and the sizes of the counting bags may be appropriately designed to hold any suitable number and any suitable type of surgical fabrics. For example, the size of the counting bag may be designed to hold 1 to 30 surgical fabrics. Similarly, the counting bag may have any suitable number of pockets to hold any suitable number of surgical fabrics. Although the fabric counting protocol generally involves placing one surgical fabric in each pocket, the pockets of the counting bag may be configured to hold any suitable number of surgical fabrics. In addition, the pockets of the counting bag may have sufficient size and shape to hold various types of surgical fabrics. For example, the pockets of the counting bag may be configured to hold surgical fabrics of various sizes, such as surgical fabrics ranging in size from 4 inches × 4 inches to 18 inches × 18 inches. The pockets of the counter bag may also be configured to accommodate various types of surgical fabrics, such as laparotomy fabrics (eg, surgical fabrics used in laparotomy procedures).

[0139] One or more example embodiments of systems for tracking surgical items discussed herein may include one or more supports, such as a rod having one or more hooks, mounts, or any suitable combination thereof. The supports may include one or more supports for holding one or more imaging devices (e.g., mobile devices). Such systems may include one or more counting bags, each of which is configured to hold surgical items (such as surgical fabric) and facilitate one or more users in implementing one or more counting protocols for surgical items. As described above (e.g., with reference to Figure 4 ), the counting bag may include a backing and one or more pockets. The backing may include one or more openings configured to interact with one or more hooks of the pole to hang the counting bag on the one or more hooks. The counting bag components (e.g., backing and pockets) may be made of any suitable material and may have any suitable color. For example, the backing of the counting bag may have an opaque blue, opaque white, or other suitable color, and the pocket may be made of a transparent material. It may be beneficial to make the backing of one or more counting bags opaque and colored so that the counting bags are easy to identify even when stacked (e.g., one on top of another) so that the color contrasts with the bags of dirty and clean surgical supplies (e.g., dirty and clean surgical linens) placed therein. In addition, the transparent pocket may be beneficial in allowing visualization of surgical items within the pocket, visualization of one or more markings on the surgical items (e.g., labels on surgical linens), or both. The counting bag may have any suitable shape and size, such as a shape and size that is beneficial for hanging the counting bag on a pole. Although Figure 4 The example embodiment shown in uses a rectangular counting bag with rectangular pockets, but the counting bag and its pockets may have any suitable shape and size, as described above.

[0140] One or more example embodiments of the system for tracking surgical items discussed herein may include one or more weight detection components. Such weight detection components may be used to sense the presence of surgical items, weigh them, count them, or any suitable combination thereof. As described above, one or more weight detection components may include one or more scales for determining a first count of surgical items. For example, one or more weight detection components may be configured to receive a first count of surgical items and transmit weight information to a processor to determine the first count. One or more weight detection components may also be coupled to one or more collection containers (e.g., counting bags or "kick buckets") or combined therewith to determine a second count of surgical items. One or more collection containers may be coupled to, or may include, a scale or other suitable weight detection component to sense a reduction in weight when a user removes surgical items from one or more collection containers. One or more counting bags may be coupled to one or more weight detection components in the example form of one or more strain gauges or scales, which are coupled to a rod to sense a weight change when surgical items are added to a counting bag to determine a second count of surgical items.

[0141] For the purpose of explanation, the above description uses specific nomenclature to provide a thorough understanding of the present subject matter. However, it will be apparent to those skilled in the art that specific details are not required in order to practice the present subject matter. Therefore, for the purpose of illustration and description, the foregoing description of specific example embodiments is presented. They are not intended to be exhaustive or to limit the present subject matter to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. Example embodiments are selected and described to explain the principles of the present subject matter and some practical applications thereof, so that other persons skilled in the art can utilize the present subject matter and various example embodiments with various modifications suitable for the intended specific use. The claims and their equivalents are intended to define the scope of the present invention.

[0142] The following enumerated description describes various examples of methods, machine-readable media, and systems (eg, machines, devices, or other apparatuses) discussed herein.

[0143] A first example provides a method of tracking surgical items, the method comprising:

[0144] At one or more processors:

[0145] receiving first count of surgical items;

[0146] receiving one or more images, wherein each image has a field of view including one or more surgical objects;

[0147] determining a second count of surgical items based at least in part on the one or more received images; and

[0148] A notification is provided based on a comparison between the first count of surgical items and the second count of surgical items.

[0149] A second example provides a method according to the first example, wherein the determining of the second count of surgical items includes identifying one or more surgical items in the one or more received images.

[0150] A third example provides a method according to the second example, wherein at least one of the received images is a depth image.

[0151] A fourth example provides the method according to the third example, wherein the depth image is an infrared image.

[0152] A fifth example provides a method according to any one of the second to fourth examples, wherein at least one of the received images is a color image.

[0153] A sixth example provides the method according to any one of the second to fifth examples, wherein the surgical item is a surgical fabric.

[0154] A seventh example provides a method according to any of the second to sixth examples, wherein the identification of the one or more surgical items includes identifying and characterizing a container including one or more compartments in the one or more images.

[0155] An eighth example provides the method according to the seventh example, wherein the container includes a backing and a plurality of pockets arranged in a rectangular grid on the backing.

[0156] A ninth example provides the method of the eighth example, wherein the plurality of pockets comprises a transparent material.

[0157] A tenth example provides the method of the eighth example, wherein the backing comprises an opaque material.

[0158] An eleventh example provides the method of any one of the seventh to tenth examples, wherein the characterization of the container includes determining one or more parameters of the container.

[0159] A twelfth example provides the method according to the eleventh example, wherein the determining of the one or more parameters of the identified container includes quantifying the one or more compartments of the container.

[0160] A thirteenth example provides a method according to the eleventh example or the twelfth example, wherein the determining of the one or more parameters of the identified container includes determining a geometric arrangement of the one or more compartments of the container.

[0161] A fourteenth example provides a method according to any one of the eleventh to thirteenth examples, wherein the determination of the one or more parameters of the identified container includes determining the presence or absence of a surgical item in each of the one or more compartments of the container.

[0162] A fifteenth example provides the method of any of the first to fourteenth examples, wherein the determination of the second count of surgical items is further based at least in part on detecting a change in weight of a container configured to hold one or more surgical items.

[0163] A sixteenth example provides the method of any of the first to fifteenth examples, wherein the receiving of the first count of surgical items includes identifying one or more surgical items in the image.

[0164] A seventeenth example provides the method of any of the first to sixteenth examples, wherein receiving the first count of surgical items is based at least in part on detecting a weight of the one or more surgical items.

[0165] An eighteenth example provides the method of any of the first to seventeenth examples, wherein the first count of surgical items is provided by user input.

[0166] A nineteenth example provides the method of any of the first to eighteenth examples, wherein the determining of the second count of surgical items includes updating a current index value of the surgical items.

[0167] A twentieth example provides the method according to the nineteenth example, further comprising displaying a current index value of the surgical item on a display.

[0168] A twenty-first example provides the method according to any one of the first to twentieth examples, further comprising prompting a user to verify the accuracy of at least one of the first count and the second count.

[0169] A twenty-second example provides a method according to any one of the first to twenty-first examples, further comprising displaying at least one of a first count of surgical items, a second count of surgical items, or a notification on a display.

[0170] A twenty-third example provides the method of any of the first to twenty-second examples, further comprising providing a summary report of the tracked surgical items.

[0171] A twenty-fourth example provides the method of any one of the first to twenty-third examples, wherein providing a notification based on a comparison between the first count and the second count includes generating an alert in response to the first count not matching the second count.

[0172] A twenty-fifth example provides the method according to the twenty-fourth example, wherein the reminder is in accordance with a predetermined protocol.

[0173] A twenty-sixth example provides the method of any of the first to twenty-fifth examples, wherein at least a portion of the first count is received prior to the medical procedure.

[0174] A twenty-seventh example provides the method of any of the first to twenty-sixth examples, wherein the second count is determined during or after the medical procedure.

[0175] A twenty-eighth example provides a method according to any one of the first to twenty-seventh examples, wherein determining the second count includes analyzing the one or more received images using one or more machine learning models.

[0176] The twenty-ninth example provides a method according to the twenty-eighth example, wherein one or more of the machine learning models incorporates deep learning techniques.

[0177] A thirtieth example provides a system for tracking surgical items, the system comprising:

[0178] One or more processors configured to perform operations including:

[0179] receiving first count of surgical items;

[0180] receiving one or more images, wherein each image has a field of view including one or more surgical objects;

[0181] determining a second count of surgical items based at least in part on the one or more received images; and

[0182] A notification is provided based on a comparison between the first count of surgical items and the second count of surgical items.

[0183] A thirty-first example provides the system of the thirtieth example, wherein determining the second count of surgical items comprises identifying one or more surgical items in the one or more received images.

[0184] A thirty-second example provides a system according to the thirty-first example, wherein at least one of the received images is a depth image.

[0185] The thirty-third example provides a system according to the thirty-second example, wherein the depth image is an infrared image.

[0186] A thirty-fourth example provides a system according to any one of the thirty-first to thirty-third examples, wherein at least one of the received images is a color image.

[0187] A thirty-fifth example provides a system according to any one of the thirty-first to thirty-fourth examples, wherein the identification of the one or more surgical items includes identifying and characterizing a container including one or more compartments in the one or more images.

[0188] A thirty-sixth example provides the system according to the thirty-fifth example, wherein the characterization of the container includes determining one or more parameters of the container.

[0189] A thirty-seventh example provides the system according to the thirty-sixth example, wherein the determining of the one or more parameters of the identified container includes quantifying the one or more compartments of the container.

[0190] A thirty-eighth example provides a system according to the thirty-sixth example or the thirty-seventh example, wherein the determination of the one or more parameters of the identified container includes determining a geometric arrangement of the one or more compartments of the container.

[0191] A thirty-ninth example provides a system according to any one of the thirty-sixth to thirty-eighth examples, wherein determining the one or more parameters of the identified container includes determining the presence or absence of a surgical item in each of the one or more compartments of the container.

[0192] A fortieth example provides a system according to any one of the thirty-fifth to thirty-ninth examples, wherein the container is a flexible container comprising a backing and a plurality of pockets arranged in a rectangular grid.

[0193] A forty-first example provides a system according to the fortieth example, wherein the plurality of pockets comprises a transparent material.

[0194] The forty-second example provides a system according to the fortieth example, wherein the backing comprises an opaque material.

[0195] A forty-third example provides a system according to any one of the thirtieth to forty-second examples, further comprising an optical sensor configured to generate the one or more images received by the processor.

[0196] A forty-fourth example provides a system according to any one of the thirtieth to forty-third examples, further comprising a display screen.

[0197] A forty-fifth example provides a system according to the forty-fourth example, wherein the display screen is configured to display a summary report of the tracked surgical items.

[0198] A forty-sixth example provides the system according to the forty-fourth example or the forty-fifth example, wherein the display screen is configured to display a notification based on a comparison between the first count and the second count.

[0199] A forty-seventh example provides the system according to the forty-sixth example, wherein if the first count does not match the second count, then the notification includes a reminder according to a predetermined protocol.

[0200] A forty-eighth example provides a system according to any one of the thirtieth to forty-seventh examples, wherein the surgical item is a surgical fabric.

[0201] A forty-ninth example provides a system according to any one of the thirtieth to forty-eighth examples, wherein at least one of the one or more processors is in a mobile computing device that is removably mounted to a bracket.

[0202] A fiftieth example provides a method for tracking surgical items, the method comprising:

[0203] At one or more processors:

[0204] receiving an image of a field of view, wherein the field of view includes one or more surgical objects;

[0205] determining the presence of the one or more surgical items in the image; and providing an indication of the determined presence of the one or more surgical items in the image.

[0206] The fifty-first example provides a method according to the fiftieth example, further comprising quantizing the one or more surgical objects in the image based on at least one trained machine learning model.

[0207] The fifty-second example provides a method according to the fifty-first example, wherein the machine learning model is combined with deep learning technology.

[0208] The fifty-third example provides a method according to any one of the fiftieth to fifty-second examples, wherein determining the presence of the one or more surgical items includes detecting the presence of a container in an image, wherein the container includes a plurality of compartments, each compartment being constructed to receive at least one corresponding surgical item.

[0209] The fifty-fourth example provides a method according to the fifty-third example, further comprising classifying each compartment based on the presence or absence of surgical items.

[0210] A fifty-fifth example provides a method according to any one of the fiftieth to fifty-fourth examples, wherein providing an indication of the determined presence includes providing an indication of the presence of at least one surgical object in the determined image.

[0211] The fifty-sixth example provides a method according to any one of the fiftieth to fifty-fifth examples, wherein the one or more surgical items include surgical fabrics.

[0212] The fifty-seventh example provides a method according to any one of the fiftieth to fifty-sixth examples, wherein the one or more surgical items include a surgical instrument.

[0213] The fifty-eighth example provides a carrier medium carrying machine-readable instructions for controlling a machine to perform the operations (eg, method operations) performed in any one of the preceding examples.

[0214] The fifty-ninth example provides a machine-readable medium (e.g., a non-transitory machine-readable storage medium) comprising instructions that, when executed by one or more processors of the machine, cause the machine to perform operations (e.g., method operations) specified by any one of the preceding examples.

Claims

1. A method for tracking a surgical item using a system, the system comprising an imaging system, a display, and one or more processors, the method comprising: capturing, with the imaging system, a video depicting a user handling the surgical item; receiving the video with the one or more processors; performing computational analysis of the video with the one or more processors to detect characteristic arm movements of a user, wherein the characteristic arm movements indicate that the user is moving a surgical item to be counted during a surgical procedure; updating, with the one or more processors, a count of surgical items in response to each detected instance of the characteristic arm movement; as well as The updated count is displayed on the display.

2. The method of claim 1, wherein the characteristic arm motion is a sweeping arm motion.

3. The method of claim 2, wherein the sweeping arm motion comprises an arm motion instructing a user to remove a surgical item from a package to place on a surface.

4. The method of claim 1, wherein the imaging system comprises a camera positioned to capture one of a front view, a side view, a top view, or a combination thereof of the user.

5. The method according to any one of claims 1 to 4, wherein the updated count is a first count, the method further comprising receiving, at the one or more processors, an image depicting at least a subset of the surgical items; determining, with the one or more processors, a second count of surgical items based on the image; as well as A notification is provided on the display based on a comparison of the first count and the second count.

6. A method for tracking surgical items during a surgical procedure, the method comprising receiving input including one or more patient parameters; determining, with one or more processors, a high risk that one or more of the surgical items are left in the patient based on the patient parameters; triggering, with the one or more processors, a protocol in response to: (i) a determination of a high risk and (ii) a number of surgical items counted exceeding a threshold number of surgical items, wherein the protocol includes one or more precautions; displaying the one or more preventive measures on a display; as well as receiving, at the one or more processors, an image depicting at least a subset of the surgical items; tracking, with the one or more processors, compliance with a protocol based on the image; as well as Information about compliance with the protocol is displayed on a display.

7. The method according to claim 6, further comprising: receiving, at the one or more processors, an image depicting at least a subset of the surgical items; determining, with the one or more processors, a first count of surgical items based on the image; as well as The first count is compared with a threshold number of surgical items with the one or more processors.

8. The method of claim 6, wherein the one or more patient parameters is a body mass index (BMI) within a predetermined range.

9. The method of claim 6, wherein the one or more precautions include imaging the patient to ensure that at least one of the surgical items is not left within the patient.

10. A method according to claim 6, wherein the protocol is a counting protocol for a container comprising a collection of compartments, the method comprising displaying on a display one or more visual indications of the order in which the collection of compartments are to be filled with the surgical items to be counted.

11. The method of any of claims 6-10, further comprising receiving an input to enter an emergency mode in which tracking of compliance with the protocol is stopped.

12. The method of claim 11, further comprising providing a notification on a display indicating that an emergency mode has been entered.

13. The method of claim 11, further comprising ceasing to collect item count data in the emergency mode.

14. A method for tracking surgical items, comprising: Displaying the counting protocol on a display; receiving, at one or more processors, an image depicting at least a subset of the surgical items; tracking, with the one or more processors, compliance with the protocol based on the image; as well as displaying information on compliance with the counting protocol on a display; receiving an input to enter an emergency mode in which tracking of compliance with the counting protocol is stopped; as well as A notification is provided on the display indicating that emergency mode has been entered.

15. The method according to claim 14, Wherein the counting protocol is for a container comprising a collection of compartments, the method comprises displaying on a display one or more visual indications of the order in which the collection of compartments are to be filled with the surgical items being counted.

16. The method of claim 15, further comprising tracking, with the one or more processors, compliance with the counting protocol based on the order in which the set of compartments of the container are filled based on the image.

17. The method of claim 14, further comprising ceasing to collect item count data in the emergency mode.

Images