Systems and methods for quantifying blood loss from surgical sponge management systems - Patents.com

Abstract

A system and method for quantifying blood loss with a surgical sponge management system. A data reader detects a tag on the surgical sponge, and one or more processors identify a sponge type of the surgical sponge based on the unique identifier. A presentation window is displayed on the user interface with dimensions of the presentation window based on the identified sponge type. The processor determines whether characteristics of the surgical sponge meet acceptance criteria based on the identified type of the surgical sponge. The acceptance criteria include sponge presence, folded sponge, partial sponge, presentation distance, correct sponge, sponge in motion, mask verification, and sponge monitoring. If the characteristics meet the acceptance criteria, an optical sensor captures a color image of the surgical sponge. The amount of blood or blood components in the surgical sponge is estimated based on the color image and displayed on the user interface.

Description

[Technical field]

[0001] Claiming priority This application claims priority to and the full benefit of U.S. Provisional Patent Application No. 63 / 321,400, filed March 18, 2022, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Managing surgical sponges is a critical area in the modern operating room, where ensuring that surgical sponges (or other objects) are not inadvertently left behind or otherwise misplaced within a patient's body is paramount. In the past, medical personnel relied on manual sorting and counting of surgical sponges. More recently, surgical sponge management systems have utilized electronic devices to assist in counting surgical sponges. One such system is sold by Stryker Corporation (Kalamazoo, Michigan) under the trade names SurgiCount and SurgiCount+, and is disclosed in commonly owned U.S. Patent Application Publication No. 2013 / 0088354, published April 11, 2013, which is incorporated herein by reference in its entirety, in which a radio frequency identification (RFID) reader detects the RFID tag on the surgical sponge.

[0003] The determination of blood loss during surgery can be used to monitor the health of patients during surgery. It is known to estimate blood loss during surgery by visual evaluation of surgical sponges and other fluid-absorbing articles (e.g., surgical gowns, bedding, or drapes), which is inherently subjective and therefore prone to human error. It is also known to estimate blood loss during surgery by weighing surgical sponges in bulk on a scale, which requires the user to carry the surgical sponges to the scale to weigh them. Advances in image processing and computing have made it possible to quantify blood loss by capturing images of surgical sponges. One such system is sold under the trade name Triton by Gauss Surgical, Inc. (Menlo Park, Calif.) and is disclosed in commonly owned U.S. Patent No. 8,897,523, issued November 25, 2014, and U.S. Patent No. 10,424,060, issued September 24, 2019, the entire contents of each of which are incorporated herein by reference.

[0004] Systems that use image-based blood quantification may not adequately compensate for different types of surgical sponges and may not provide the advanced functionality of surgical sponge management systems. Thus, there is a need in the art to provide improved systems and methods that overcome the aforementioned shortcomings and do so with an intuitive and seamless workflow for users familiar with existing surgical protocols. Summary of the Invention

[0005] The present disclosure generally relates to the analysis of surgical objects by image-based processing based on the article type of a known surgical object. The primary embodiment is described with respect to a surgical sponge management system and method for quantifying blood loss of blood-stained surgical sponges during a surgical procedure. However, it is contemplated that the objectives of the present disclosure may be applied to surgical objects other than surgical sponges, including sharps, tools (powered or manual), and other instruments. Among other advantages, the functionality of the present system and method eliminates the need for a user to activate any screen or device to determine, recall, and / or manually input the article type and / or trigger the capture of an image for analysis. In the case of surgical sponges, the system can perform QBL on surgical sponges of different sizes and layers, and can do so automatically in real time without requiring the user to meaningfully change the workflow to count out the surgical sponges. The need to carry the surgical sponge to a separate scale for weighing is likewise eliminated. Additionally, the surgical sponge management system provides QBL without increasing the footprint in the operating room, and accommodates the sponge size being imaged or scanned at a common distance from the camera and at a convenient height for the user. For other surgical objects, the system can verify that the object being presented to the system is the same object type as previously identified. Image processing can further facilitate identification of damaged objects, used vs. unused object status, object cleaning status, automated object counting, etc. Thus, the terminology associated with the sponge-based methods described herein can be modified to apply to non-sponge-based surgical objects as well.

[0006] The surgical sponge management system may include a user interface, a data reader, one or more processors, memory, a communication device, and / or other hardware. The user interface may be a tablet with a touch screen display. A camera including an optical sensor and optionally a depth sensor may be integrated into the tablet. The neural network may be implemented or executed on a processor of the tablet, on one or more processors of hardware coupled to or remote from the tablet, via cloud computing, a combination thereof, etc.

[0007] The data reader can be used as a handheld device or while supported by a cradle. The data reader is configured to detect a tag associated with the surgical sponge. The data reader may be an RFID reader configured to detect radio frequency (RFID) tags. The surgical sponges can be counted by causing the tag to be detected by the data reader. The detected tag transmits identification data to the processor. The tag includes at least one unique identifier indicative of a characteristic of the surgical sponge. The unique identifier may include a size or type of the sponge. The processor is configured to determine the type of surgical sponge based on the unique identifier. The processor can index the counter to reflect that one or more surgical sponges have been counted for the surgical procedure. The counter can be displayed in a user interface.

[0008] The method includes excluding a surgical sponge from the count. A tag on the surgical sponge is detected by a data reader and a unique identifier stored on the tag is received by a processor. The processor identifies a sponge type of the surgical sponge from a database of predetermined sponge types based on the unique identifier. The user interface can index and display counters including updated counts for each sponge type counted for the surgical procedure and excluded from the count for the surgical procedure.

[0009] The method can include triggering a system to image the sponge. The camera can be automatically activated in response to detection of the tag by the data reader. An image feed detected by the camera can be displayed on a user interface. The image feed can be based on data being received from an optical sensor and / or a depth sensor. The image feed can include image frames forming a video of a user holding the sponge within a field of view of the optical sensor. The image feed can include graphical enhancements or indicia based on data from the optical sensor and / or a depth sensor. Displaying the image feed can be performed automatically in response to detection of the tag by the data reader.

[0010] The method can include displaying, in a user interface, a presentation window overlaid on the image feed being captured by the camera. The dimensions of the presentation window are based on an identified sponge type of the surgical sponge corresponding to the detected tag of the surgical sponge being excluded from the count. The processor can determine an acceptance criterion based on the surgical sponge type, and the presentation window can be sized and scaled such that at least a portion of the acceptance criterion is met when the user presents the surgical sponge to at least substantially match or fit the presentation window. Alternatively, a presentation window may not be displayed, and instead the processor may determine a dynamic presentation window based on the actual position of the sponge within the optical sensor and / or depth sensor.

[0011] The image feed is processed or analyzed to meet acceptance criteria or guardrails. The acceptance criteria are generally designed to ensure that the QBL image of the surgical sponge captured by the optical sensor is of high quality. This step includes analyzing each image frame (or every few image frames) of the image feed to ensure, among other acceptance criteria, that the correct surgical sponge is stretched, unfolded, and presented motionless at the proper distance. If one or more acceptance criteria remain unmet, the processor prevents the camera from capturing a QBL image of the surgical sponge for QBL analysis. If the acceptance criteria are determined to be met, the method includes acquiring a QBL image of the surgical sponge by the camera. The QBL image may be a subsequent frame of the image feed output from a sponge recognition algorithm executing a sponge recognition state machine (SRSM) immediately after the acceptance criteria are met. The QBL image is analyzed in a hemoglobin estimation engine to determine the liquid blood components (e.g., hemoglobin) in the surgical sponge. The processor can extract color component values ​​(e.g., redness values) from the QBL images and execute a trained hemoglobin estimation algorithm implementing a hemoglobin estimation engine to determine the mass of blood components on a pixel-by-pixel or other suitable basis. The hemoglobin estimation algorithm may be unique to the sponge type excluded from the count. The hemoglobin estimation algorithm can be trained on an extensive data set for each sponge type to fit the system. Blood loss can be determined based on the blood components. By using the sponge type to determine the presentation window characteristics and acceptance criteria, image-based QBL can be applied to more or most types of surgical cloths, including those of different sizes, layers, and / or colors.

[0012] For some sponge types with fewer layers, it may be necessary to fold the sponge before imaging. A presentation window displayed in the user interface may be used to guide the user to fold the surgical sponge. The method may also include the optional step of displaying a folding protocol to the user in the user interface. The presentation window may be of a size that approximates the size of the surgical sponge when folded according to the folding protocol. The folding protocol may include guidance to facilitate the user folding the surgical sponge in a desired manner. The guidance may be written instructions or graphical indicia and / or audible content.

[0013] The sponge recognition algorithm is configured to process the images of the surgical sponge to detect the sponge and apply acceptance criteria or guardrails, executed by one or more processors. The sponge recognition algorithm may include image pre-processing, a neural network, a labeling logic, a guardrail algorithm, an SRSM, and any image post-processing. There are at least two neural networks: a localization neural network and a segmentation neural network. The localization neural network is configured to detect, among other operations, whether or not a sponge is present in the image and provide a bounding box around the detected sponge. The bounding box may include information about the detected sponge, such as width, height, x-shift, and y-shift. Additionally, the localization neural network may be configured to determine whether or not an unstretched sponge is presented in the presentation window, or whether or not a partial sponge is presented, or whether or not no sponge is presented. The segmentation neural network provides a pixel-by-pixel prediction of the sponge segmentation mask of whether the pixel is a sponge pixel or a background pixel. The segmentation neural network may utilize data from both the optical and depth sensors, or alternatively, only the optical sensor. The localization labels receive the output from the localization neural network, and the segmentation labeler receives the output from the segmentation neural network. These labels are fed into the SRSM to identify a video-based state based on the dominant class prediction over a pre-set number of previous frames. The output of the sponge recognition algorithm is fed into the hemoglobin estimation algorithm.

[0014] First, the sponge recognition algorithm determines whether a surgical sponge is present. Next, the method includes determining whether the surgical sponge is folded by determining whether the surgical sponge is at least substantially square or rectangular. The localization neural network generates a bounding box for the sponge being detected. Based on the outer edge of the bounding box, the localization labeler determines whether the surgical sponge is sufficiently square or rectangular. This determination can be associated with a sponge type. This step may include determining whether an aspect ratio of the presented surgical sponge is within an acceptable range based on sponge types that are excluded from the count. The localization neural network provides a bounding box and compares the determined aspect ratio of the bounding box to an acceptable range of aspect ratios for that sponge type. If the determined aspect ratio of the bounding box is outside the acceptable range, the localization labeler determines that the sponge is a folded sponge and generates a localization label accordingly.

[0015] The method can include determining whether the presented sponge is a partial sponge. If the bounding box of the sponge extends beyond the field of view of the camera, the localization labeler determines the partial sponge condition. Additionally, the localization labeler can determine whether the bounding box is too close to a presentation window. A margin, i.e., a pixel-based distance between the bounding box and the presentation window, can be determined and compared to a margin threshold. If the determined margin is less than the margin threshold, the localization labeler generates a localization label that the presented surgical sponge is a partial sponge.

[0016] The sponge recognition algorithm may include determining whether the sponge is positioned too close or too far from the camera by determining whether the sponge is too big or too small, respectively. The localization labeler determines a pixel-based area of ​​the bounding box and compares the bounding box area to a bounding box area tolerance range unique to each sponge type. The bounding box area tolerance range may approximate a presentation window. If the processor determines that the bounding box area is larger or smaller than the bounding box area tolerance range, the localization labeler generates a localization label that the sponge is too close or too far from the camera, respectively.

[0017] The segmentation neural network outputs a segmentation mask in which sponge pixels and background pixels are separated. The segmentation neural network may run only on image frames of the image feed in which the localization neural network determines that a sponge is present. The segmentation neural network may run in parallel or serially after the localization network to optimize the tradeoff between processing speed and resources. The method includes validating the sponge segmentation mask by determining a number of pixels in the sponge segmentation mask and comparing the determined number of pixels to a pixel tolerance range. The pixel tolerance range is based on a sponge type, which is again identified by the processor by exclusion from counting by the data reader. If the segmentation mask determined by the segmentation neural network does not have a sufficient number of pixels, the segmentation mask is determined to be invalid. The segmentation labeler may generate a segmentation label accordingly.

[0018] A segmentation mask refinement algorithm can eliminate pixels from the segmentation mask that have a different depth value that exceeds a threshold when compared to the majority of the mask pixels. The difference in depth value can be estimated as the distance between each point and a plane fitted to the depth map. The threshold can be the same or different for different sponge types. The depth data can be used to set a threshold for an acceptable presentation distance. The method can include determining whether the presented surgical sponge is too close or too far. The segmentation labeler can output a corresponding segmentation label that the presented surgical sponge is too close or the sponge is too far if the average depth of all pixels belonging to the sponge segmentation mask is less than a minimum threshold or greater than a maximum threshold, respectively.

[0019] The segmentation labeler can utilize the depth map of depth data obtained from the depth sensor to determine the actual dimensions of the presented surgical sponge for evaluating further acceptance criteria. The method can further include determining whether the presented surgical sponge is a correct sponge. By utilizing the depth map to determine the sponge area of ​​the segmentation mask, the segmentation labeler compares the determined actual sponge area to an acceptable range of sponge areas based on the sponge types excluded from the count. If the determined actual sponge area of ​​the segmentation mask is outside the acceptable range of sponge areas, the segmentation labeler outputs an incorrect sponge label. This step can further include comparing the compactness of the sponge to a predetermined compactness threshold. The predetermined compactness threshold can be a percentage of the area of ​​a bounding box that the determined sponge area of ​​the segmentation mask must exceed to be determined as correct.

[0020] The method may further include determining whether the presented surgical sponge is moving. A center of mass of the sponge segmentation mask may be determined, and an average magnitude of movement of the center of mass within a predetermined number of image frames of the image feed may be determined. This step may include comparing the average magnitude of movement to a center of mass change threshold, which may be based on sponge types that are excluded from the count. If the average magnitude of movement is greater than the center of mass change threshold, the segmentation labeler determines that the presented surgical sponge is moving too fast and the acceptance criteria are not met.

[0021] The segmentation and localization neural networks can be combined into a single neural network that outputs both a classification label (e.g., no sponge, partial sponge, folded sponge, and complete sponge) and a segmentation mask and associated bounding box around the detected sponge. Additionally, the segmentation and localization labelers can be combined as a series of condition checks for each of the guardrails based on the features of the detected sponge and the expected sponge type. The segmentation and localization neural networks can be run simultaneously or sequentially.

[0022] The method may include performing sponge monitoring, where a pose estimation neural network is used to detect the user's hands, torso, neck, and other relevant body parts, and a localizer neural network or segmentation neural network is used to detect the position of the presented surgical sponge. Once the person and the presented surgical sponge are detected and located, the pose estimation neural network is configured to determine whether the presented surgical sponge is properly held in front of the torso. The sponge monitoring algorithm may identify a body centerline and a sponge vertical centerline, and further determine a horizontal shift of the position of the presented surgical sponge relative to the body body centerline. One or more of the additional centerlines may be compared to the horizontal sponge centerline of the presented surgical sponge to determine a vertical shift of the presented surgical sponge. The horizontal and / or vertical shift may be compared to respective shift thresholds associated and / or predetermined preferred positions. If the shift exceeds the shift threshold, guidance may be provided in the user interface to instruct the user to present the surgical sponge in front of the user's torso. A depth map from the depth sensor can be used to distinguish between people in the background and people in the foreground, and a pose estimation algorithm determines which of multiple people in the camera's field of view is presenting the sponge.

[0023] The sponge recognition algorithm includes a sponge recognition state machine (SRSM). The SRSM is configured to receive labels from each of the localization labeler and the segmentation labeler and performs further processing before the camera captures the QBL image. The SRSM algorithm determines whether the acceptance criteria or guardrails, which may be performed by the localization labeler and the sponge segmentation labeler separately, have been met by combining the aggregated states from both the localization labeler and the segmentation labeler. The SRSM algorithm may perform a smoothing procedure that aggregates the localization and segmentation labels using a sliding window of the image frame size. The most frequent aggregated sponge label is called the dominant label. If the dominant label in the required consecutive frames meets the acceptance criteria, the SRSM may indicate that the QBL image is ready to be captured and analyzed by the hemoglobin estimation engine. If no dominant label is identified in the last consecutive image frame, the SRSM may generate an indeterminate state state.

[0024] Once a QBL image has been captured, the SRSM algorithm may not allow further capture of QBL images, regardless of the most recent label and the dominant label, until at least the dominant label identifies that the sponge has disappeared or is not present for a requisite number of consecutive image frames of the image feed, at which point the SRSM algorithm is configured to return the localization neural network and the segmentation neural network to a state in which they are prepared to perform the methods disclosed herein. [Brief description of the drawings]

[0025] [Figure 1]1 illustrates a surgical sponge management system, the user interface can be supported on a stand, the system includes an optical sensor that captures a color image of the surgical sponge, and optionally a depth sensor. [Diagram 2] 1 is a flow chart for quantifying blood loss according to an exemplary method of the present disclosure. [Diagram 3] 1 is a perspective view of a user interface at one step of the method. [Figure 4] 11 is a perspective view of a user interface at another step of the method. FIG. [Diagram 5] 11 is a perspective view of a user interface at another step of the method. FIG. [Figure 6] 1 is a flow chart including acceptance criteria according to one step of the method. [Figure 7] FIG. 1 is a schematic diagram of a neural network and algorithms implemented by one or more processors. [Figure 8A] 1 is a flowchart for processing image frames of an image feed using a localization neural network in accordance with an exemplary method of the present disclosure. [Figure 8B] 1 is a flowchart for processing image frames of an image feed using a segmentation neural network according to an exemplary method of the present disclosure. [Figure 9] FIG. 13 is a diagram for explaining the acceptance criteria. [Figure 10] FIG. 13 is a diagram for explaining the acceptance criteria. [Figure 11] FIG. 13 is a diagram for explaining the acceptance criteria. [Figure 12] FIG. 13 is a diagram for explaining the acceptance criteria. [Figure 13] FIG. 13 is a diagram for explaining the acceptance criteria. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The present disclosure relates to a system for managing surgical sponges during a surgical procedure and quantifying blood loss (QBL) associated with blood absorbed by the surgical sponge. With reference to FIG. 1 , a surgical sponge management system 20 can include a stand 22, an electronics subsystem 24, and a dispenser assembly 26. The stand 22 includes a base 28 that is wheeled to move the surgical sponge management system 20 within a medical facility. The stand 22 can include a main support 30 coupled to and extending upwardly from the base 28. The dispenser assembly 26 can be supported on top of the main support 30, and a sponge sorter 32 can be suspended from an articulated arm of the dispenser assembly 26. The electronics subsystem 24 includes a module base 34, a user interface 36, and a data reader 38. The module base 34 can be secured to the stand 22. A mount removably couples the user interface 36 to the module base 34 or the stand 22. The user interface 36 (and / or module base 34) may include one or more processors 42, memory, communication devices, and / or other hardware. For example, the user interface 36 may be a tablet with a touch screen display. The system 20 may include an optical sensor 44 (e.g., a color camera) and may optionally include a depth sensor 46 (e.g., an infrared camera), often collectively referred to herein as a camera 48. The camera 48 may be integrated into the user interface 36, removably coupled to the user interface 36, or may be standalone and in wired or wireless communication with the processor 42. In examples where the camera 48 is integrated on the tablet, the tablet may be oriented upside down in the mount such that the camera 48 is positioned on the lower front surface of the tablet. This arrangement allows for better height alignment so that the user comfortably positions the surgical sponge in front of his or her body to be detected by the camera 48 (see FIGS. 8-12).Furthermore, the described neural networks may be executed on the tablet's processor 42, on one or more processors in hardware coupled to or remote from the tablet, via cloud computing, combinations thereof, etc.

[0027] The data reader 38 can be used as a handheld device or supported in a cradle as shown in FIG. 1. The data reader 38 is configured to detect a tag (T) associated with a surgical sponge (S). In an exemplary embodiment, the data reader 38 is a radio frequency scanner, i.e., an RFID reader configured to detect RFID tags, such as those disclosed in commonly owned International Publication No. WO 2021 / 041795, published March 4, 2021, and WO 2021 / 097197, published May 20, 2021, the entire contents of which are incorporated herein by reference. Optical tags (e.g., bar codes and quick response (QR) codes) and identifiers other than RFID tags, such as those disclosed in commonly owned International Publication No. WO 2017 / 112051, published June 29, 2017, the entire contents of which are incorporated herein by reference, are also contemplated. In embodiments used with surgical articles other than surgical sponges, the identifier may be a tag as described above, or ink printed on the surgical article.

[0028] In a typical surgical procedure in which surgical sponges are utilized, the surgical sponges are counted as they are used during the surgical procedure. The surgical sponges may be counted by having their tags detected by the data reader 38. For example, a user may position a surgical sponge (or a stack of surgical sponges) near the data reader 38. The detected tag transmits identification data to the processor 42, which includes at least one unique identifier that is indicative of a characteristic of the surgical sponge. The unique identifier may include the size or type of sponge (e.g., 2×16 gauze, 4×4 gauze, 4×8 gauze, 4×12 sponge, 12×12 sponge, and 18×18 laparotomy sponge, etc.), material construction (e.g., 4-ply, 8-ply, 16-ply, etc.), absorbency, stretchability, dry weight, etc. The processor 42 is configured to determine the type of surgical sponge based on the unique identifier. Further, based on the data reader 38 detecting the tag, the processor 42 can index the counter 64 to reflect that one or more surgical sponges have been counted for the surgical procedure. The counter 64 can be displayed on the user interface 36 (see FIG. 3). Either during or after the surgical procedure, used and unused surgical sponges are typically excluded from the count. The tag of the surgical sponge is repositioned to be detected by the data reader 38 to identify the surgical sponge as being excluded from the count. The processor 42 indexes the counter 64 accordingly (e.g., subtracts one from the quantity of sponges previously counted) and displays on the user interface 36 the quantity of surgical sponges still being counted. The counter 64 can also display the quantity of surgical sponges originally counted and / or the quantity of surgical sponges already excluded from the count. In practice, after the user confirms in the user interface 36 that the surgical sponge has been successfully excluded from the count, the user immediately places the surgical sponge in one of the pockets of the sponge sorter 32.

[0029] The disclosed system 20 and method 100, 200 facilitate rapid and accurate assessment of fluids, particularly blood, contained in surgical sponges. The associated steps can be performed in a small amount of time between removing the surgical sponge from the count and placing the surgical sponge in the sponge sorter 32, thus minimizing interruption to the surgical workflow. Alternatively, the steps may be performed at other times during the surgical procedure, as requested by the user. Referring now to FIG. 2, a method 100 for assessing bloody surgical sponges in a surgical procedure is provided. The method 100 includes removing the surgical sponge from the count (step 102). As described above, the tag of the surgical sponge is detected by the data reader 38, and data contained on the tag is transmitted from the data reader 38 to the processor 42. In particular, the unique identifier stored on the tag is received by the processor 42. The processor 42 identifies the surgical sponge type from a database of predefined sponge types based on the unique identifier. For example, the processor 42 may identify the surgical sponge as an 18×18 laparotomy sponge. The sponge type may be prompted or populated (e.g., "popped up") in the user interface 36, optionally with an audible indicator, to indicate successful scanning of the sponge for the user to quickly see which sponge types were excluded from the count. The user interface 36 may index and display a counter 64 containing updated counts for each sponge type that was counted for the surgical procedure and excluded from the count for the surgical procedure, as shown in Figure 3. Certain guardrail data described is also based on (e.g., unique to) the sponge type that is excluded from the count, and that data is transmitted, loaded, or otherwise prepared by the processor 42 for image-based processing.

[0030] The method may include triggering the system 20 for sponge imaging (step 104). This step may include automatically activating the camera 48 in response to detection of the tag by the data reader 38. The automatic activation step is optional, and it is contemplated that a user may alternatively provide input to the user interface 36 to activate the camera 48 and perform subsequent steps of the method 100. Alternatively, for procedure types in which a QBL protocol (e.g., methods 100, 200) is utilized, a mode may be selected on the user interface 36 prior to the start of the surgical procedure, after which the camera 48 is automatically activated in response to detection of the tag by the data reader 38.

[0031] This step may further include displaying the image feed 50 detected by the camera 48 on the user interface 36. The image feed 50 may also be considered a sponge capture screen. The image feed 50 may be based on data received from the optical sensor 44 and / or the depth sensor 46. FIG. 4 shows an example of an image feed 50 in which image frames forming a video of a user holding a sponge within the field of view 52 of the optical sensor 44 are shown. The image feed 50 may also include graphical enhancements or indicia based on data from the optical sensor 44 and / or the depth sensor 46. One example includes a presentation window 54 overlaying the image feed 50, and another example includes the image feed 50 being grayscale (based on the depth data) with a segmentation mask 72 of the detected sponge highlighted as green. The user may be trained to "present" the surgical sponge to the camera 48, preferably immediately after excluding the surgical sponge from the count. The user may generally grasp or pinch a top corner of the surgical sponge to expose the maximum area of ​​the surgical sponge to the camera 48, and provide instructions to this effect in the user interface 36. The forward-facing orientation of the camera 48 and the display of the user interface 36 allows the user to utilize an image feed 50 to assist in real-time alignment of the surgical sponge in the manner described.

[0032] The display of image feed 50 may be performed automatically in response to detection of the tag by data reader 38. Thus, camera 48 may be automatically activated and the display of user interface 36 may be automatically updated to include image feed 50. As can be seen generally from FIG. 4, counter 64 has been replaced with image feed 50, and certain other fields of the display have been shifted, resized, or removed. In this manner, image feed 50 is sized so as to be easily viewable by a user standing at an acceptable presentation distance from camera 48.

[0033] The method 100 includes displaying in the user interface 36 a presentation window 54 overlaying the image feed 50 being captured by the camera 48 (step 106). The presentation window 54 is a visual cue to guide the user to present the surgical sponge in an appropriate manner and at an appropriate distance from the camera 48. Depending on the sponge type, the distance may be, for example, 18-36 inches. With continued reference to FIG. 4, the presentation window 54 is depicted as four "corner brackets" representing the four corners of the presentation window 54, although the presentation window 54 need not assume any particular geometric shape. The dimensions of the presentation window 54 are based on the identified type of surgical sponge that corresponds to the detected tag of the surgical sponge that is excluded from the count. The processor 42 may be further configured to determine an acceptance criterion 74 based on the surgical sponge type, and the presentation window 54 may be sized and scaled such that at least a portion of the acceptance criterion 74 is met when the user presents the surgical sponge to at least substantially match or fit the presentation window 54. This step is optional, in which case the processor 42 may determine a dynamic presentation window based on the actual position of the sponge within the optical and / or depth sensors. The processor 42 may dynamically evaluate the acceptance criteria based on the dynamic presentation window.

[0034] The method 100 includes the sponge recognition algorithm analyzing the image feed 50 for whether it meets the acceptance criteria 74 (step 110). This step includes analyzing each image frame (or every few frames) of the image feed 50 to ensure that the correct surgical sponge, complete or stretched, is presented unfolded and motionless at the proper distance, among other acceptance criteria 74, also referred to herein as guardrails (see FIG. 6). If one or more of the acceptance criteria 74 remain unmet, the processor 42 prevents the camera 48 from capturing at least one QBL image of the surgical sponge for QBL analysis. As a result, the characteristics of the QBL image of the surgical sponge captured by the optical sensor 44 are highly consistent, except for the blood content thereon, such that the sponge recognition algorithm trained to analyze the captured QBL images is correspondingly highly accurate.

[0035] Once it is determined that the acceptance criteria 74 have been met, the method 100 includes acquiring a QBL image of the surgical sponge, also referred to herein as a “sponge scan,” using the camera 48 (step 112). In one example, the QBL image may be a subsequent frame of the image feed 50 immediately following the satisfaction of the acceptance criteria 74 analyzed by a sponge recognition algorithm, including output from a sponge recognition state machine (SRSM) 73. The QBL image may be considered separate from the image frame from the image feed 50 that is being analyzed for whether it meets the acceptance criteria. Rather, the QBL image is analyzed using a hemoglobin estimation algorithm to determine blood components (e.g., hemoglobin) of the liquid on the surgical sponge (step 114), for example, in the manner disclosed in the aforementioned U.S. Pat. Nos. 8,897,523 and 10,424,060. In particular, the processor 42 may be configured to extract color component values ​​(e.g., redness values) from the QBL image and execute a hemoglobin estimation algorithm to determine hemoglobin concentration on a pixel-by-pixel or other suitable basis. The hemoglobin estimation algorithm may be unique to the sponge type excluded from the count. In other words, the hemoglobin estimation algorithm may be trained on an extensive data set for each sponge type to fit the system 20. Blood loss may be determined based on blood components and pixel-based or actual area of ​​the surgical sponge.

[0036] The user interface 36 can provide a prompt (e.g., visual and / or audio indication) to notify the user that the QBL image capture was successful and immediately present the itemized blood loss metric 58 for that particular sponge. For example, FIG. 4 represents a surgical sponge that has absorbed 9 milliliters or blood. The user deposits the surgical sponge into the sponge sorter 32. Additionally, the user interface 36 can update in real time to reflect the patient's cumulative blood loss. After a short delay (e.g., 1 or 2 seconds), the user interface 36 can return to the counter 64 (see FIG. 3) where the sponge count is updated as well as the cumulative blood loss metric 60.

[0037] By using sponge type to determine the characteristics and acceptance criteria 74 of the presentation window 54, image-based QBL can be applied to more or most types of surgical cloth, including those of different sizes, layers, and / or colors. In contrast, for example, a smaller sponge type imaged at a closer distance may be mistaken for a larger sponge type presented at a greater distance, or a larger sponge type imaged at a closer distance may have its outer portions cut off, if the system does not know the sponge type (and therefore its size). Similarly, different numbers of layers result in images with different redness values, especially when imaged in a folded state. The system 20 and method of the present disclosure overcome such shortcomings by determining blood loss in different types of surgical sponges in a seamless manner within the surgical procedure, without the need to predefine which surgical sponge types are being used or excluded from the count.

[0038] For some surgical sponges, the presentation window 54 displayed in the user interface 36 may require the user to fold the surgical sponge. One example may include a relatively large or elongated surgical sponge that, if presented unfolded, would require the user to move too far away from the camera 48 for accurate analysis. Furthermore, requiring folding of some surgical sponges results in the dimensions of the respective presentation windows 54 associated with different sponge types being sufficiently different. As a result, the acceptance criteria 74 are sufficiently different for each surgical sponge type such that a user is generally prevented from meeting the acceptance criteria 74 with an incorrect sponge type or an incorrect fold of the sponge.

[0039] In some implementations, the method 100 can include displaying the folding protocol to the user on the user interface 36 (step 108). This step is optional, and alternatively, the presentation window 54 may require that the surgical sponge be presented in an unfolded form. The folding protocol can include at least two aspects. First, the presentation window 54 is sized to approximate the dimensions of the surgical sponge when folded according to the folding protocol. In other words, the presentation window 54 may be too small for the surgical sponge when presented unfolded. Second, the folding protocol can include guidance 62 to facilitate the user folding the surgical sponge in a desired manner. The guidance 62 can be written instructions or graphical indicia, as shown in FIG. 5, and / or audible content. In addition to the folding protocol, it is contemplated that the system 20 can require the user to image both sides of the folded surgical sponge.

[0040] An exemplary workflow will now be described with reference to FIGS. 3-5. A counter 64 displayed in FIG. 3 indicates that two types of surgical sponges have been counted for the surgical procedure: a 4×8 gauze and an 18×18 laparotomy sponge. The user excludes one of the 18×18 laparotomy sponges from the count. The user interface 36 refreshes or updates to include an image feed 50 and a presentation window 54 corresponding to the 18×18 laparotomy sponge, as shown in FIG. 4. The presentation window 54 is a square sized to fit the field of view 52 of the camera 48. The square presentation window for the square shaped surgical sponge indicates (without further guidance being provided) that the user should present the surgical sponge in an unfolded form. If the user aligns the unfolded 18×18 laparotomy sponge to closely fit the presentation window 54, then the acceptance criteria 74 will likely be met and a QBL image will be captured by the camera 48. A unique hemoglobin estimation algorithm is applied to the 18×18 laparotomy sponge to determine blood loss in the sponge. At a later point in the surgical procedure, the user excludes one of the 4×8 gauzes from the count. The user interface 36 again refreshes or updates to include the image feed 50 and displays a presentation window 54 corresponding to the 4×8 gauzes. As can be seen from FIG. 5 , the presentation window 54 is a rectangle sized significantly smaller than the field of view 52 of the camera 48. The user interface 36 also includes guidance 62, i.e., textual instructions for manipulating the surgical sponge in a specified manner (i.e., folding protocol), such that the size of the folded 4×8 gauzes will approximate the size and shape of the displayed presentation window 54. Presenting the surgical sponge in a manner other than that required by the guidance 62 effectively prevents the acceptance criteria 74 for the 4×8 gauzes from being met, and thus prevents the camera 48 from capturing a QBL image.When the folded 4x8 is presented in a manner that approximates the presentation window 54, the acceptance criteria 74 is likely met and a QBL image is captured by the camera 48. A unique hemoglobin estimation algorithm is applied to the 4x8 gauze sponge to determine blood loss in the sponge. An item specific blood loss metric 58, in this case 13 milliliters of blood, is displayed. It should be appreciated that the methods 100, 200 disclosed herein can be performed in just a few seconds without significant interruption to workflow and without requiring the user to select a sponge type and / or manually trigger a sponge scan of the presented surgical sponge within the field of view 52 of the camera 48.

[0041] As described above, the one or more processors 42 implement or execute a sponge recognition algorithm. The sponge recognition includes a neural network configured to process images of surgical sponges to detect the sponge and apply acceptance criteria 74 or guardrails. In an exemplary implementation, there are two neural networks: a localization neural network 66 and a segmentation neural network 68. The localization neural network 66 may be a deep neural network such as a convolutional neural network (e.g., ResNet-50, MobileNet-v1,2, etc.), a transformer-based neural network (e.g., ViT), or a combination thereof. The output of the localization neural network 66 is modified to have both a classification head and a regression head. The localization neural network 66 is configured to determine, among other operations, the image class, i.e., whether a sponge is presented (sponge present), whether it is folded or collapsed (folded sponge), or whether it is only partially visible (partial sponge). The localization neural network 66 is further configured to provide a bounding box 70 around the detected sponge (see FIGS. 10-12). The bounding box 70 may include a width (w), a height (h), and an offset (x, y).

[0042] The segmentation neural network 68 may be an encoder-decoder type convolutional neural network architecture such as U-Net, Mask-R CNN, DeepLab, or a transformer-based neural network architecture that supports image segmentation. The output of the segmentation neural network 68 provides a pixel-by-pixel prediction of whether the pixel is a sponge pixel or a background pixel in a segmentation mask 72 (schematically shown in FIGS. 9-11 ). In an exemplary implementation, the segmentation neural network 68 may utilize data from both the optical sensor 44 and the depth sensor 46. Alternatively, the segmentation neural network 68 may provide the segmentation mask 72 based solely on the color image. The segmentation neural network 68 is configured to provide the sponge segmentation mask 72 and facilitate decisions including acceptance criteria 74, among other operations that will be described.

[0043] As reflected in FIG. 7, the sponge recognition algorithm includes a localization labeler 67, a segmentation labeler 69, and a sponge recognition state machine (SRSM) 73. The localization labeler 67 and the segmentation labeler 68 are configured to generate classification labels, i.e., localization labels and segmentation labels, respectively, based on a respective analysis of certain acceptance criteria 74. The localization labels and segmentation labels are fed to the sponge recognition state machine (SRSM) 73. In particular, based on the labels of the per-image predictions, the SRSM identifies a video-based state. The SRSM uses dominant class predictions over a pre-set number of preceding frames to ensure that the video-based state is correct even if a single frame may be misclassified.

[0044] The localization neural network 66 can run in real-time on slower devices to facilitate real-time user interaction, while the segmentation neural network 68 can be considered as a larger and more accurate convolutional neural network. The dual mode design ensures real-time processing of image frames of the image feed 50 without compromising the accuracy of the sponge segmentation mask 72. Alternatively, both neural networks 66, 68 can be combined into a single hybrid network that outputs both classification labels and the segmentation mask 72 and bounding box 70. Such a hybrid neural network can run in real-time on faster devices with sufficient graphic processing power. The neural network can be trained in the manner disclosed herein to perform the disclosed method with the required accuracy. The output of the sponge recognition algorithm is provided to a hemoglobin estimation algorithm that is unique to the sponge type that is excluded from the count.

[0045] 6-8B, a method 200 for evaluating each of the acceptance criteria 74 or guardrails will now be described in turn. The method 200 includes executing an algorithm by at least one of the neural networks 66, 68. The acceptance criteria 74 can be evaluated in the order shown, or in any suitable order unless otherwise specified. Initially, the method 200 includes determining whether a surgical sponge is present (step 202). The localization neural network 66 determines whether a surgical sponge is present in an image frame of the image feed 50. For example, the surgical sponge may not yet have been brought within the field of view 52 of the camera 48, or is too far from the camera 48 to be detected. If the sponge is not detected, the acceptance criteria 74 is identified as not being met.

[0046] If a surgical sponge is detected in step 202, the method 200 includes determining whether the surgical sponge is folded (step 204). FIG. 9 illustrates a common manner in which a sponge is often improperly held. In particular, a user may not provide enough tension along the top surface of the surgical sponge, causing the top surface to droop due to sagging. Additionally, in this example, a portion of a corner of the lower side of the surgical sponge is folded back. The localization neural network 66 can be trained to recognize and classify folded, sagging, or collapsed sponge images as "folded sponges." Step 204 of determining whether the sponge is folded can also include determining whether the surgical sponge is at least substantially square or rectangular. Referring to FIG. 10, the localization neural network 66 is configured to provide a bounding box 70 around the detected sponge, among other operations. The bounding box can include information about the detected sponge, such as a width, a height, an x-shift, and a y-shift. Based on the outer edge of the bounding box, the localization labeler determines whether the surgical sponge is sufficiently square or rectangular. This determination can be associated with the sponge type. The extent to which the surgical sponge is square or rectangular to meet the guardrail may be based on the localization neural network 66 having been trained on a dataset of sponges that have been folded and correctly held. For example, if an 18×18 laparotomy sponge is excluded from the count, the localization neural network 66 can determine whether the presented surgical sponge is sufficiently square. If a 4×12 gauze is excluded from the count (and should be presented in an unfolded form), the localization neural network 66 can determine whether the presented surgical sponge is sufficiently rectangular.

[0047] Determining whether the sponge is folded, step 204, may further include determining whether the aspect ratio of the presented surgical sponge is within an acceptable range based on the type of sponge being excluded from the count. As described above, the bounding box 70 includes a width and a height, and the ratio of these two is the determined aspect ratio. If the sponge is determined to be sufficiently square or rectangular, but the determined aspect ratio is outside the acceptable range of aspect ratios for that sponge type, the localization neural network 66 determines that the sponge is a folded sponge (step 204). For example, for an 18×18 laparotomy sponge, its manufactured aspect ratio may be approximately 1.0. By training the localization neural network 66 on a data set of 18×18 laparotomy sponges presented in various ways by users, the algorithm implementing the guardrails may have set the acceptable range of aspect ratios to be, for example, between 0.7 and 1.5. These variations account for the elasticity of the sponge material when the user may stretch the sponge, slight warping while being held from an upper corner, etc. If the user excludes an 18×18 laparotomy sponge from the count and presents the sponge in a manner whose determined aspect ratio is outside of the acceptable range, the localization neural network 66 may determine that the 18×18 laparotomy sponge is folded or in a folded state. Additionally or alternatively, a determined aspect ratio outside of the acceptable range may indicate that an incorrect sponge has been presented. If a folded sponge (or an incorrect sponge) is detected, the acceptance criteria 74 is identified as not being met. The localization neural network 66 generates a localization label that the presented surgical sponge is a folded sponge.

[0048] As discussed above, by requiring folding of some surgical sponges, the dimensions of the respective presentation windows 54 for different sponge types are also sufficiently different. By extension, this also applies to the aspect ratio tolerances. In other words, the aspect ratio tolerances are sufficiently different for each type of surgical sponge that a user is generally prevented from filling step 204 with an incorrect sponge type. For example, the tolerance for a 4×8 gauze, when folded according to a folding protocol, may be 1.7-2.2. A conformable sponge type that quickly becomes smaller, e.g., a 2×16 gauze, when folded according to a folding protocol as needed, may be associated with a tolerance range of 4.9-7.8. Similarly, a conformable sponge type that quickly becomes larger, e.g., a 4×12 gauze, when folded according to a folding protocol as needed, may be associated with a tolerance range of 2.5-3.4. These values ​​are merely exemplary, but it should be understood that none of the tolerances overlap between a 2 x 16 gauze, a 4 x 8 gauze, and a 4 x 12 gauze. For square sponges, such as a 12 x 12 sponge and an 18 x 18 laparotomy sponge, the tolerances may overlap, but the correct sponge guardrails described (step 216) will address such considerations.

[0049] The method 200 may include determining whether the presented sponge is a partial sponge by being too close to a boundary (step 206). If the segmented representation of the presented surgical sponge extends beyond the field of view 52 of the camera 48, the localization neural network 66 determines the partial sponge condition. For example, if the outermost pixels of the field of view 52 of the camera 48 are determined to be sponge pixels and / or if the bounding box 70 is determined to extend through the outermost pixels, the localization neural network 66 determines the partial sponge condition. With concurrent reference to FIG. 11, the localization labeler 66 may determine whether the bounding box 70 is too close to the presentation window 54. In particular, the localization labeler 66 determines a margin, i.e., a pixel-based distance between the bounding box 70 and the presentation window 54. Training the localization labeler 66 on the dataset may establish margin thresholds associated with each of the top, bottom, left, and right of the presentation window 54. Alternatively, the margin threshold may be input into the software. In one example, the margin threshold may be set to zero such that if the bounding box 70 intersects with the presentation window 54, the localization neural network 66 will determine a partial sponge condition. FIG. 11 shows that the left side of the bounding box 70 (based on the user's left hand (L)) passes through the presentation window 54, resulting in a negative value for the margin. Similarly, the bottom of the bounding box 70 passes through the presentation window 54. Together, the user in FIG. 11 has submitted the sponge too far to the left and too low to meet the acceptance criteria 74. The localization labeler 67 generates a localization label that the presented surgical sponge is a partial sponge.Of course, the image feed 50 including the presentation window 54 is presented on the user interface 36 so that the user can easily adjust the position of the presented surgical sponge. Prompts or indicia can be provided on the user interface 36 to guide the user's movements accordingly.

[0050] The localization neural network 66 and the localization labeler 67 may be further configured to determine whether the sponge is too close or too far from the camera 48 by determining whether the sponge is too large or too small, respectively (step 208). In particular, the localization labeler 67 is configured to determine a pixel-based area of ​​the bounding box 70 and compare the bounding box area to a bounding box area tolerance range that is unique to each sponge type. The localization labeler 67 may train on a data set to an algorithm that establishes a bounding box area tolerance range, or the tolerance range may be input or otherwise determined. In one example, the bounding box area tolerance range is unique to the sponge type that approximates the presentation window 54 displayed in the user interface 36, which is also excluded from the count. The bounding box area tolerance range is pixel-based such that step 208 can be based solely on optical images from the optical sensor 44. If the processor 42 determines that the bounding box area is larger or smaller than the bounding box area tolerance, the localization labeler 67 generates a localization label that the sponge is too close or too far (from the camera 48), respectively. Figure 12, for example, shows a bounding box 70 having an area significantly smaller than the area of ​​the presentation window 54, such that a presented surgical sponge may be identified as being too far from the camera 48. Again, the image feed 50 is presented on the user interface 36 to allow the user to adjust the position of the sponge, and corresponding prompts can be provided on the user interface 36 to guide the user's movements.

[0051] If the localization neural network 66 determines that the sponge is unfolded and otherwise properly positioned and presented within the presentation window 54, the method 200 includes identifying the presented surgical sponge as a complete sponge (step 210), and the localization labeler 67 responsively generates a localization label for further processing by the SRSM 73.

[0052] Along with processing by the localization neural network 66, a segmentation neural network 68 also processes the image frames of the image feed 50. The dual mode processing may be performed simultaneously or sequentially. The segmentation neural network 68 outputs a segmentation mask 72 in which sponge pixels and background pixels are separated. Figures 9-11 depict that the segmentation mask 72 fits the presented surgical sponge with high accuracy, e.g., pixel-by-pixel. Additionally, the segmentation neural network 68 may utilize a depth map 51 of depth data obtained from the depth sensor 46. A segmentation mask refinement algorithm 71 may eliminate pixels from the segmentation mask 72 that have a different depth value that exceeds a threshold when compared to the majority of the mask pixels. For example, the difference in depth value may be estimated as the distance between each point and a plane fitted to the depth map. The threshold may be the same or different for different sponge types. Additionally, the depth map 51 may be processed to determine the actual dimensions of the presented surgical sponge for evaluating further acceptance criteria 74 including presentation distance (step 214), confirmation of correct sponge (step 216), confirmation of stationary sponge (step 218), and sponge monitoring (step 220).

[0053] 6 and 8B, the segmentation neural network 68 may run only on image frames of the image feed 50 where the localization neural network 66 determines that a sponge is present (see step 202). If not, the segmentation mask 72 may be identified as unavailable or empty. If it is determined that the sponge segmentation mask 72 is not empty, the method 200 includes validating the sponge segmentation mask 72 (step 212). Step 212 may include determining a number of pixels in the sponge segmentation mask 72 and comparing the determined number of pixels to a pixel tolerance range. The pixel tolerance range may be based on the sponge type, which is again identified by the processor 42 by exclusion from counting by the data reader 38. If the segmentation mask 72 determined by the segmentation neural network 68 does not have a sufficient number of pixels, the segmentation mask 72 is determined to be invalid.

[0054] The depth data can be used to establish thresholds for acceptable presentation distances, i.e., lower and upper limits for the actual distance the user must present the surgical sponge from the camera 48. The method 200 can include determining whether the presented surgical sponge is too close or too far (step 214). The segmentation labeler 69 is configured to output that the sponge is either too close or too far away if the average depth of all pixels belonging to the sponge segmentation mask 72 is less than a minimum threshold or greater than a maximum threshold, respectively. As previously mentioned, the localization neural network 66 performs a similar process in which the size of the bounding box 70 is compared to the size of the presentation window 54 (see step 208), which does not depend on the depth data or on the actual dimensions.

[0055] Additionally, if a depth map 51 is available, the actual area of ​​the sponge being imaged can be determined. The segmentation labeler 69 does so by determining the sponge area (in segmentation mask 72) based on the depth map of the image frame of the image feed 50. The determined sponge area is compared to a tolerance range for the sponge area. The determined sponge area can be calculated as the L2 squared norm, which is the square root of all values ​​from the depth map that correspond to pixels in the image frame. Training the segmentation labeler 69 on the data set may establish a minimum and maximum area for each type of surgical sponge, or the tolerance range may be input or otherwise determined.

[0056] The method 200 may further include determining whether the presented surgical sponge is the correct sponge (step 216). As described above, the extent to which the localization neural network 66 can determine the correct sponge is based on a comparison of the determined aspect ratio against a tolerance range of aspect ratios. The segmentation neural network 68 may confirm this determination and further consider scenarios in which some sponge types may have overlapping tolerance ranges. By utilizing the depth map to determine the actual area of ​​the segmentation mask 72, the segmentation labeler 69 may distinguish square-shaped sponges of different dimensions. The segmentation labeler 69 may be trained on a data set to establish a tolerance range for sponge area, or the tolerance range may be input or otherwise determined. For example, at a suitable presentation distance, an upper sponge area threshold for a 12×12 sponge may be 14000 AU, while a lower sponge area threshold for an 18×18 laparotomy sponge may be 16000 AU. As a result, if the user excludes an 18×18 laparotomy sponge from the count and presents a 12×12 sponge that would otherwise meet the acceptance criteria (e.g., to fit into presentation window 54), segmentation labeler 69 will determine that the determined sponge area of ​​the presented surgical sponge is too low for the distance at which it is presented. Segmentation neural network 68 may determine that acceptance criteria 74 is not met and generate a segmentation label accordingly. A corresponding prompt may be presented in user interface 36 to indicate to the user that an incorrect sponge may have been presented.

[0057] The step 216 of whether the presented surgical sponge is a correct sponge can also include comparing the compactness of the sponge to a predetermined compactness threshold. In some respects, the step of comparing the compactness of the sponge can be considered as a cross-check between the segmentation neural network 68 and the localization neural network 66, which again determine the area and aspect ratio of the bounding box 70. The predetermined compactness threshold can be a percentage of the area of ​​the bounding box 70 that the determined sponge area of ​​the segmentation mask 72 must exceed in order to be determined to be correct. For example, the predetermined compactness threshold can be 50% such that if the determined area of ​​the segmentation mask 72 is less than 50% of the determined area of ​​the bounding box 70, the acceptance criteria 74 is determined not to be met. This may indicate an incorrect sponge or an invalid mask (see step 218). A corresponding prompt can be presented on the user interface 36 requesting the user to present the surgical sponge to the camera 48.

[0058] The method 200 may further include determining whether the presented surgical sponge is moving (step 218). If the presented surgical sponge moves too fast, motion blur may impair the accuracy of the QBL analysis. Step 218 may include determining a center of mass of the sponge segmentation mask 72 and further determining an average motion magnitude of the center of mass within a predetermined number of image frames (e.g., five consecutive image frames) of the image feed 50. The average motion magnitude may be calculated as the L2-distance between successive estimated centers of mass. Step 218 may include comparing the average motion magnitude to a center of mass change threshold unique to each sponge type. The center of mass change threshold may be based on training the segmentation labeler 69 with an extensive data set or may be input into the software. If the average motion magnitude is greater than the center of mass change threshold, the segmentation labeler 69 determines that the presented surgical sponge is moving too fast and that the acceptance criteria 74 is not met. An alternative implementation includes tracking the movement of the bounding box 70 of the detected sponge against a threshold value across successive image frames of the image feed 50. A corresponding prompt can be provided in the user interface 36 requesting the user to hold the sponge steady.

[0059] Another acceptance criterion 74 or guardrail of method 200 may include performing sponge monitoring (step 220). As mentioned above, processor 42 may be configured to extract redness values ​​from the blood-stained sponge to determine the concentration of blood constituents on a pixel-by-pixel or other suitable basis. The redness of the pixels of the image is affected by ambient lighting, such as changes in light in a medical facility. It is known to use calibration placards for light normalization to account for linear changes in ambient light intensity and color temperature. While generally effective, light "bleed-through" through the back of a surgical sponge presented from a directional light source may cause an underestimation in the prediction of blood constituents. Light bleed-through may be exacerbated by a user holding the sponge away from the body, thereby exposing a large portion of the surgical sponge to a light source positioned behind the user in a medical facility.

[0060] Step 220 of performing sponge monitoring may include detecting the position of the presented surgical sponge in the manner described above, along with detecting the user's hands, torso, neck, and other relevant body parts using a pose estimation neural network and a pose estimation labeler (not specified). Once the person and the presented surgical sponge are detected, the pose estimation labeler is configured to determine whether the presented surgical sponge is properly held in front of the torso, which tends to limit light bleed-through. With particular reference to FIG. 13, the pose labeler is configured to identify a body centerline 76 and a sponge vertical centerline 78. Based on the identified centerlines 76, 78, step 220 may include determining a horizontal shift of the position of the presented surgical sponge relative to the body centerline 76. Additionally, the pose estimation labeler may be configured to identify centerlines 80 or reference points of other anatomical landmarks, such as the head, neck, arms, hands, legs, etc. One or more of the additional centerlines 80 may be compared to a horizontal sponge centerline 82 of the presented surgical sponge to determine a vertical shift of the presented surgical sponge. The horizontal and / or vertical shifts may be compared to respective shift thresholds determined by a pose estimation algorithm trained on the data set or input or otherwise determined. If the shift exceeds the shift thresholds, guidance may be provided to the user interface 36 to instruct the user to present the surgical sponge in front of the torso. Additionally, the depth map from the depth sensor 46 may be used to distinguish between people in the background and people in the foreground. In conjunction with the known position of the presented surgical sponge within the field of view 52, ​​the pose estimation algorithm is configured to determine which of multiple people within the field of view 52 of the camera 48 the sponge is being presented to.

[0061] If the segmentation neural network 68 determines that the sponge is presented at the correct distance and position relative to the user's body, is the correct sponge, is not moving, and the segmentation mask 72 is otherwise valid, then the segmentation labeler 69 generates segmentation labels for further processing by the SRSM 73. This is in addition to the localization labels generated by the localization labeler 67.

[0062] The SRSM is configured to receive labels from each of the localization labeler 66 and the sponge segmentation labeler 68 and perform further processing before the camera 48 captures the QBL image. The SRSM is configured to determine whether the localization and segmentation labels are equal. In other words, the SRSM determines whether the acceptance criteria 74 or guardrails, which may be performed separately by the localization neural network 66 and the sponge segmentation neural network 68, have been met. Because the image feed 50 is a video and image frames occur in virtually imperceptible rapid succession, the image state of one or more of the acceptance criteria 74 may rapidly change the results. The SRSM may perform a smoothing procedure that aggregates the localization and segmentation labels using a sliding window of the image frame size (L). The sliding window may incorporate prior states from time "tL" to time "t-1" and may not include the most recent state (at time t). The most frequent aggregated sponge label is called the dominant label. If the required consecutive frames (K) containing the most recent label and the dominant label meet the acceptance criteria 74, the SRSM can indicate that the QBL image is ready to be captured and analyzed by the hemoglobin estimation engine. As the number of required consecutive frames increases, the probability of a false positive scan decreases. The parameters L, K can be adjusted based on empirical evaluation. If no dominant label is identified within the last K consecutive image frames, the SRSM may generate an indeterminate video state. For example, an indeterminate video state may be generated if the most frequent aggregated sponge label within the sliding window occurs less frequently than half the sliding window size, or some other suitable value.

[0063] Once a QBL image has been captured, the sponge recognition algorithm is configured to place the system 20 in a temporary "cool down" state in order for the user to remove the presented surgical sponge from the field of view 52 of the camera 48. The SRSM may not allow further capture of QBL images, independent of the immediate and dominant labels, until at least the dominant label identifies the sponge as absent or non-existent over a required number of consecutive image frames (M) of the image feed (see step 202). The parameter M may be adjusted or otherwise determined based on empirical evaluation. Once the sponge has been absent over a required number of consecutive frames, the SRSM is configured to return to a video state in which the localization neural network 66 and the segmentation neural network 68 are prepared to execute the methods 100, 200 disclosed herein. In effect, this allows only one sponge scan to be captured per presented surgical sponge. The user is appropriately instructed to remove the sponge from the field of view 52 of the camera 48 before the next surgical sponge is presented.

[0064] As mentioned above, the neural networks 66, 68 can be trained using supervised, semi-supervised, and / or unsupervised learning methodologies, particularly with the accuracy required to perform the methods of the present disclosure. A diverse and representative set of training data sets is collected to train each of the neural networks for a variety of scenarios both within and outside the expected use of the system 20. Training methods can include data augmentation, pre-trained neural networks, and / or use semi-supervised learning methods to reduce the requirement for labeled training data sets. Data augmentation can increase sample size, such as illumination augmentation (brightness, contrast, hue-saturation shift, PlanckianJitter, etc.), geometric transformations (rotation, homography, inversion, etc.), adding noise and blur, custom augmentation (artificially expanding or reducing the size of the sponge, cropping the sponge), etc.

[0065] The dataset varies parameters including sponge size, sponge saturation, ground truth (blood color), lighting, imaging distance, sponge orientation, and position. For example, the sponge can be saturated by varying the level of summation up to the maximum blood carrying capacity, imaging distance can be varied from 0.5 to 1.5 meters, lighting can be made brighter or brighter, sponge orientation can be rotated in-plane or out-of-plane, and position can be varied with images of folded, unfolded, and slightly folded sponges. Additionally, a set of example non-sponge objects, such as sponges, can be included in the dataset to train the neural network to recognize white or red non-sponge objects as part of the background.

[0066] An additional approach includes training a neural network with both color and depth images to jointly learn a segmentation mask using both color and depth images. Certain architectures such as ESANet can be used to efficiently combine color and depth images. The segmentation neural network 68 can also be trained on images where depth data is not available (e.g., color images only). If depth data is not available, a data set can be generated that includes pixel regions in the segmentation mask that correlate at different imaging distances. From the depth of the sponge scan from the color image, the segmentation mask, and the sponge type being known, the segmentation neural network 68 is configured to determine whether a presented surgical sponge is scanned too close or too far from the system. Thus, the segmentation neural network 68 is configured to provide a segmentation mask 72 based solely on the color image from the optical sensor 44. Finally, for the pose estimation neural network for sponge monitoring, example models can include AlphaPose, MobileHumanPose, MoveNet, OpenPose, etc., which are further trained on application-specific datasets, e.g., nurses imaging sponges.

[0067] The foregoing disclosure is not intended to be exhaustive or to limit the present disclosure to any particular form. The terminology used is intended to be in the nature of words of description rather than of limitation. Many modifications and variations are possible in light of the above teachings, and the invention may be practiced otherwise than as specifically described.

[0068] Further inventive aspects of the present disclosure are disclosed with reference to the following exemplary clauses.

[0069] Clause 1 - A method for evaluating a surgical object using a system including a data reader, one or more processors, a user interface, and an optical sensor, the method including: identifying, by the one or more processors, an article type of the surgical object from a database of predetermined types of surgical objects; detecting the presented surgical object by a field of view of the optical sensor; capturing at least one image of the presented surgical object; determining, by the one or more processors, whether the presented surgical sponge meets at least one acceptance criterion, the acceptance criterion being based on the identified article type scanned by the data reader; and displaying, on the user interface, at least one of a confirmation that the presented surgical object is of the same type as that scanned by the data reader, and information related to a status or condition of the surgical object.

[0070] Clause 2 - The method of clause 1, wherein each of the surgical objects includes a tag that stores a unique identifier, the method further including detecting the tag of the surgical object with a data reader, wherein the unique identifier stored in the tag is received by a processor, and identifying, by one or more processors, the item type based on the unique identifier.

[0071] Clause 3 - The method of clause 1 or 2, wherein the step of determining whether the characteristics of the surgical object meet the acceptance criteria includes generating, by one or more processors, a segmentation mask of the presented surgical object; determining, by one or more processors, a number of pixels in the segmentation mask; comparing, by the one or more processors, the determined number of pixels of the segmentation mask to database images of the surgical object in an unused or undamaged state; and if the determined number of pixels is outside an acceptable range of pixels, displaying information on a user interface indicating that the state of the surgical object is damaged.

[0072] Clause 4 - The method of clause 3, wherein the system includes a depth sensor and the step of determining whether the characteristics of the surgical object meet the acceptance criteria includes: determining, by one or more processors, an actual article area of ​​a segmentation mask based on depth data from the depth sensor; comparing, by the one or more processors, the determined actual sponge area of ​​the segmentation mask to a tolerance range of article area, the tolerance range of article area being based on the identified article type; and identifying, by the one or more processors, the presented surgical object as an incorrect article if the determined actual sponge area of ​​the segmentation mask is outside the tolerance range of article area.

[0073] Clause 5 - A method according to any one of clauses 1 to 4, wherein the step of determining whether the characteristics of the surgical object meet the acceptance criteria includes: generating, by one or more processors, a bounding box near or around the presented surgical object; determining, by one or more processors, an aspect ratio of the bounding box; comparing, by one or more processors, the determined aspect ratio of the bounding box to a tolerance range of aspect ratios, the tolerance range of aspect ratios being based on the identified article type; and identifying, by the one or more processors, the presented surgical object as being an incorrect type of surgical object if the determined aspect ratio of the bounding box is not within the tolerance range of aspect ratios.

[0074] Clause 6 - The method of clause 5, wherein the step of determining whether the characteristics of the surgical object meet the acceptance criteria includes: determining, by one or more processors, an area within a bounding box; comparing, by one or more processors, the determined area of ​​the bounding box to a tolerance range of bounding box areas, the tolerance range of bounding box areas being based on the identified article type; and identifying, by the one or more processors, if the determined area of ​​the bounding box is outside the tolerance range of bounding box areas, the presented surgical object as being too close or too far from the optical sensor.

Claims

1. A method for evaluating surgical sponges contaminated with blood during a surgical procedure, using a sponge management system comprising a data reader, one or more processors, a user interface, and optical sensors, wherein each surgical sponge includes a tag storing a unique identifier, and the method is: The steps include detecting the tag of the surgical sponge using the data reader, wherein the unique identifier stored in the tag is received by one or more processors, The steps include: using one or more processors to identify the sponge type of a surgical sponge from a database of predetermined types of surgical sponges based on the unique identifier; The steps include detecting the presented surgical sponge using the field of view of the optical sensor, A step of determining whether the presented surgical sponge meets at least one acceptance criterion using one or more processors, wherein the acceptance criterion is based on the identified sponge type of the surgical sponges excluded from the count. If it is determined that the acceptance criteria have been met, the optical sensor takes the step of capturing a QBL image of the presented surgical sponge, The steps include: using one or more processors to estimate the amount of blood or blood components based on the QBL image; The steps include displaying the amount of the blood or blood components on the user interface, Methods that include...

2. The method according to claim 1, further comprising the step of displaying a presentation window in the user interface that overlaps with an image feed captured by the optical sensor, wherein the dimensions of the presentation window are based on the identified type of the surgical sponge.

3. The method according to claim 2, further comprising the step of providing the user interface with instructions for operating the presented surgical sponge according to a folding protocol based on the identified sponge type, wherein the dimensions of the presentation window are further based on the dimensions of the surgical sponge according to the folding protocol.

4. The step of determining whether the characteristics of the surgical sponge meet the acceptance criteria is: The steps include generating a label indicating whether the sponge-like object is presented in a form stretched completely inside the presentation window, and a bounding box near or around the presented surgical sponge, using one or more processors. The steps include: determining whether the bounding box of the presented surgical sponge is a square or rectangle within a predetermined geometric threshold using one or more processors; The method according to claim 3, including the method described in claim 3.

5. The step of determining whether the characteristics of the surgical sponge meet the acceptance criteria is: The steps include determining the aspect ratio of the bounding box using one or more processors, A step of comparing the determined aspect ratio of the bounding box with an acceptable range of aspect ratios using one or more processors, wherein the acceptable range of aspect ratios is based on the identified sponge type of the surgical sponge. If the determined aspect ratio of the bounding box is not within the acceptable range of aspect ratios, one or more processors determine that the presented surgical sponge is in a folded state; The steps include preventing the optical sensor from capturing the QBL image sponge of the presented surgical sponge in the folded state, The method according to claim 4, further comprising:

6. The step of determining whether the characteristics of the surgical sponge meet the acceptance criteria is: The steps include determining the margin between the bounding box and the presentation window using one or more processors, The steps include: comparing the margin with a margin threshold using one or more processors; If the determined margin is smaller than the margin threshold, the one or more processors identify the presented surgical sponge as a partial sponge. The steps include preventing the optical sensor from capturing the QBL image sponge of the partial sponge, The method according to claim 5, including the method described in claim 5.

7. The method according to claim 6, wherein the determined margin is the pixel-based distance between the bounding box and the presentation window.

8. The step of determining whether the characteristics of the surgical sponge meet the acceptance criteria is: The steps include determining the area within the bounding box using one or more processors, A step of comparing the determined area of ​​the bounding box with an acceptable range of bounding box area using one or more processors, wherein the acceptable range of bounding box area is based on the identified sponge type of the surgical sponge. If the determined area of ​​the bounding box is outside the allowable range of the bounding box area, one or more processors determine that the presented surgical sponge is too close or too far from the optical sensor. The steps include preventing the optical sensor from capturing a QBL image of the presented surgical sponge that is too close or too far away, The method according to claim 4, including the method described in claim 4.

9. The step of determining whether the characteristics of the surgical sponge meet the acceptance criteria is: The steps include generating a segmentation mask of the presented surgical sponge using one or more of the aforementioned processors, The steps include determining the number of pixels in the segmentation mask using one or more of the aforementioned processors, A step of comparing the number of pixels determined in the segmentation mask with a pixel tolerance range using one or more processors, wherein the tolerance range of the number of pixels is based on the identified sponge type of the surgical sponge. The step of determining that the number of pixels determined in the segmentation mask is less than the lower limit of the acceptable range of the number of pixels, by one or more processors, that the mask is in an invalid state. The steps include preventing the optical sensor from capturing the QBL image of the presented surgical sponge in an invalid mask state, The method according to claim 1, including the method described in claim 1.

10. The system includes a depth sensor, and the step of determining whether the properties of the surgical sponge meet the acceptance criteria is: The steps include generating a segmentation mask of the presented surgical sponge using one or more of the aforementioned processors, The steps include: determining the actual sponge area of ​​the segmentation mask based on depth data from the depth sensor using one or more of the aforementioned processors; A step of comparing the determined actual sponge area of ​​the segmentation mask with an acceptable range of sponge area, wherein the acceptable range of sponge area is based on the identified sponge type of the surgical sponge, If the actual sponge area determined by the segmentation mask is outside the acceptable range of the sponge area, one or more processors identify the presented surgical sponge as an incorrect sponge. The steps include: preventing the optical sensor from acquiring the incorrect QBL image of the sponge using one or more processors; The method according to claim 1, including the method described in claim 1.

11. The method according to claim 10, further comprising the step of removing pixels from the segmentation mask by one or more processors if the estimated difference in depth values ​​between each point and a plane fitted to a depth map is outside a threshold.

12. The method according to claim 11, wherein the threshold may be the same for all sponge types.

13. The step of determining whether the characteristics of the surgical sponge meet the acceptance criteria is: The steps include generating a segmentation mask of the presented surgical sponge using one or more of the aforementioned processors, The steps include determining the center of mass of the segmentation mask using one or more of the aforementioned processors, The steps include determining the average magnitude of the movement of the center of mass within a predetermined number of image frames of the image feed using one or more processors, The steps include: comparing the magnitude of the average movement of the center of mass determined by one or more processors with a center of mass change threshold; If the magnitude of the average movement of the center of mass determined is greater than the center of mass change threshold, one or more processors identify the presented surgical sponge as a moving sponge. The steps include preventing the optical sensor from capturing the QBL image of the moving sponge, The method according to claim 1, including the method described in claim 1.

14. The step of determining whether the characteristics of the surgical sponge meet the acceptance criteria is: The steps include: detecting the user's torso within the field of view of the optical sensor using one or more processors; The steps include using one or more processors to determine the centerline of the user's torso and the vertical centerline of the presented surgical sponge, The steps include determining the horizontal shift of the vertical center line relative to the center line of the body using one or more processors, The steps include: comparing the horizontal shift with a horizontal shift threshold using one or more processors; If the horizontal shift exceeds the horizontal shift threshold, the user interface displays guidance instructing the user to move the presented surgical sponge. The method according to claim 1, including the method described in claim 1.

15. The step of determining whether the characteristics of the surgical sponge meet the acceptance criteria is: The steps include: using one or more processors to identify the user's additional centerline and the horizontal centerline of the presented surgical sponge; The steps include determining the vertical shift of the horizontal center line relative to the additional center line using one or more processors, The steps include: comparing the vertical shift with a vertical shift threshold using one or more processors; If the vertical shift exceeds the vertical shift threshold, the user interface displays guidance instructing the user to move the presented surgical sponge. The method according to claim 14, further comprising:

16. The steps include: one or more processors receiving localization labels from a localization neural network and receiving segmentation labels from a segmentation neural network; The one or more processors include the step of generating aggregated sponge labels, The steps include: one or more processors performing a sliding window protocol in which the most frequently aggregated sponge label is identified as the dominant label; If the dominant label indicates that the acceptance criteria have been met for a predetermined number of recent consecutive image frames of the image feed, one or more processors determine that the QBL image is ready to be captured. The method according to claim 1, further comprising:

17. Data reader and, User interface and Optical sensors and One or more processors that electronically communicate with the data reader, the user interface, and the optical sensor, and are configured to perform the method according to any one of claims 1 to 16. A surgical sponge management system equipped with [features / equipment].

18. A non-temporary computer-readable medium for storing instructions configured to perform the method according to any one of claims 1 to 16 when executed by one or more processors.

Images