System and method for quantifying blood loss volume using surgical sponge management system

Abstract

Systems and methods for quantifying blood loss using a surgical sponge management system. A data reader detects a tag of the surgical sponge, and one or more processors identify a sponge type of the surgical sponge based on the unique identifier. A presentation window is displayed on the user interface, wherein a size of the presentation window is based on the identified sponge type. The processor determines whether a feature of the surgical sponge meets acceptance criteria according to the identified type of surgical sponge. The acceptance criteria may include sponge presence, folded sponge, partial sponge, display distance, correct sponge, moved sponge, mask verification, and sponge supervision. Once the feature meets acceptance criteria, the optical sensor captures a color image of the surgical sponge. The amount of blood or blood component on the surgical sponge is estimated from the color image and displayed on a user interface.

Description

[0001] Priority claim

[0002] This application claims priority to and all benefits of U.S. Provisional Patent Application No. 63 / 321,400, filed on March 18, 2022, the entire contents of which are incorporated herein by reference. Background Art

[0003] Managing surgical sponges is an important area of ​​the modern operating room, with the most important being ensuring that no surgical sponges (or other objects) are inadvertently left inside a patient or misplaced. In the past, healthcare professionals relied on manual sorting and counting of surgical sponges. More recently, surgical sponge management systems have utilized electronic devices to assist in counting surgical sponges. Stryker Corporation (Kalamazoo, Michigan) sells one such system under the trade names SurgiCount and SurgiCount+, which is disclosed in co-owned U.S. Patent Publication No. 2013 / 0088354, published on April 11, 2013, the entire contents of which are incorporated herein by reference, wherein a radio frequency identification (RFID) reader detects an RFID tag on a surgical sponge.

[0004] Determining blood loss during surgery can be used to monitor the health status of patients during surgery. It is well known that blood loss during surgery is estimated by visually assessing surgical sponges and other fluid-absorbing items (e.g., surgical gowns, bedding, or surgical sheets), which is essentially subjective and therefore prone to human error. It is also known to estimate blood loss during surgery by weighing surgical sponges in batches on a scale, which requires the user to transport the surgical sponges to the scale for weighing. Advances in imaging and computing technology have provided methods for quantifying blood loss by capturing images of surgical sponges. This system sold by Gauss Surgical, Inc. (Menlo Park, California) is called Triton, which is disclosed in co-owned U.S. Patent No. 8,897,523 announced on November 25, 2014 and U.S. Patent No. 10,424,060 announced on September 24, 2019, wherein the entire contents of each patent are incorporated herein by reference.

[0005] Systems using image-based blood quantification may not adequately compensate for different types of surgical sponges and may not implement advanced features of surgical sponge management systems. Therefore, there is a need in the art to provide an improved system and method that overcomes the above-mentioned shortcomings and can provide an intuitive, seamless workflow for users who may be accustomed to existing surgical protocols. Summary of the invention

[0006] The present disclosure generally relates to analyzing surgical items by image-based processing, which is based on the known item type of the surgical item. The main embodiments are described in the context of a surgical sponge management system and a method for quantifying blood loss on a surgical sponge stained with blood during a surgical procedure. However, it is contemplated that the purposes of the present disclosure may also be applied to surgical items other than surgical sponges, including sharps, tools (electric or manual), and other instruments. Among other advantages, the functions of the system and method do not require a user to determine, call, and / or manually enter the item type, and / or do not require a user to use any screen or device to trigger the capture of an image for analysis. For surgical sponges, the system is capable of performing QBL on surgical sponges of different sizes and layers, and can do so automatically and in real time without the user purposefully changing the workflow of counting surgical sponges. Similarly, there is no need to transport surgical sponges to a separate scale for weighing. In addition, the surgical sponge management system can achieve QBL without increasing the floor space in the operating room, and accommodate sponge sizes that are imaged or scanned at a distance that is commonly seen from the camera and at a height that is convenient to the user. For other surgical items, the system is able to confirm that the item presented to the system is the same type of item as previously identified. Image processing can also help identify damaged items, used vs. unused item status, item cleaning status, automatic item counting, etc. Therefore, the terminology associated with the sponge-based methods described herein can be modified to also apply to non-sponge-based surgical items.

[0007] The surgical sponge management system may include a user interface, a data reader, one or more processors, memory, communication devices, and / or other hardware. The user interface may be a tablet computer with a touch screen display. A camera including an optical sensor and an optional depth sensor may be integrated on the tablet computer. The neural network may be implemented or executed on a processor of the tablet computer, on one or more processors of hardware connected to the tablet computer or remote from the tablet computer, through cloud computing, a combination thereof, and the like.

[0008] The data reader can be used as a handheld device or when supported by a bracket. The data reader is configured to detect a tag associated with a surgical sponge. The data reader can be a radio frequency (RFID) reader configured to detect an RFID tag. A surgical sponge can be counted by having the tag detected by the data reader. The detected tag sends identification data to a processor. The tag includes at least one unique identifier indicating a characteristic of the surgical sponge. The unique identifier can include a sponge size or type. The processor is configured to determine the type of surgical sponge based on the unique identifier. The processor can index a counter to reflect that one or more surgical sponges are counted into a surgical procedure. The counter can be displayed on a user interface.

[0009] The method includes counting out surgical sponges. A tag of the surgical sponge is detected by a data reader, and a unique identifier stored on the tag is received by a processor. The processor identifies a sponge type of the surgical sponge from a database of predetermined sponge types based on the unique identifier. A user interface can index and display a counter, including an updated count of each sponge type that has been counted in and out of surgical procedures.

[0010] The method may include triggering a system to image the sponge. The camera may be automatically activated in response to the data reader detecting the tag. The image supply detected by the camera may be displayed on the user interface. The image supply may be based on data received from the optical sensor and / or the depth sensor. The image supply may include image frames forming a video of the user holding the sponge within the field of view of the optical sensor. The image supply may include graphic enhancements or markings based on the data from the optical sensor and / or the depth sensor. The display of the image supply may be automatically performed in response to the tag being detected by the data reader.

[0011] The method may include displaying a presentation window on a user interface overlaid on the image supply captured by the camera. The size of the presentation window is based on an identified sponge type of the surgical sponge that corresponds to the detected label of the surgical sponge being counted. The processor may determine acceptance criteria based on the type of surgical sponge - and may adjust the size and scale of the presentation window so that if the user presents the surgical sponge in a manner that at least substantially matches or conforms to the presentation window, at least some of the acceptance criteria are met. Alternatively, the presentation window may not be displayed, and instead, the processor may determine a dynamic presentation window based on the actual positioning of the sponge within the optical and / or depth sensor.

[0012] The image supply is processed or analyzed to determine whether the acceptance criteria or guardrails are met. The acceptance criteria are generally designed to ensure that the QBL image of the surgical sponge captured by the optical sensor is of high quality. This step includes analyzing each image frame (or every few image frames) of the image supply to ensure that the stretched, correct surgical sponge is displayed at a suitable distance, is not folded, is not moved, and meets other acceptance criteria. If any one or more of the acceptance criteria are still not met, the processor will prevent the camera from capturing the QBL image of the surgical sponge for QBL analysis. Once it is determined that the acceptance criteria are met, the method includes the step of acquiring a QBL image of the surgical sponge using a camera. The QBL image may be a subsequent frame of the image supply that is input immediately after the acceptance criteria are met, and is output from a sponge recognition algorithm that executes a sponge recognition state machine (SRSM). The QBL image is analyzed using a hemoglobin estimation engine to determine the blood component (e.g., hemoglobin) of the liquid on the surgical sponge. The processor can extract color component values ​​(e.g., red values) from the QBL image and execute a trained hemoglobin estimation algorithm that implements the hemoglobin estimation engine to determine the quality of the blood component pixel by pixel or other suitable manner. The hemoglobin estimation algorithm may be unique to the type of sponge that has been counted. The hemoglobin estimation algorithm may be trained for each sponge type on a broad dataset to be compatible with the system. The amount of blood loss may be determined based on the blood composition. By using the sponge type to determine the characteristics of the presentation window and acceptance criteria, image-based QBL may be applied to more or most types of surgical textiles, including textiles of different sizes, layers, and / or colors.

[0013] For some sponge types with fewer or lower layers (1ower plys), it may be necessary to fold the sponge before imaging. The display window displayed on the user interface can be used to guide the user to fold the surgical sponge. The method can also include an optional step of displaying a folding protocol to the user on the user interface. The size of the display window can be designed to be close to the size of the surgical sponge when folded according to the folding protocol. The folding protocol can include guidance to facilitate the user to fold the surgical sponge in a desired manner. The guidance can be text instructions or graphic marks, and / or voice content.

[0014] The sponge identification algorithm is executed by one or more processors and is configured to process images of surgical sponges to detect sponges and apply acceptance criteria or guardrails. The sponge identification algorithm may include image preprocessing, neural networks, labeling logic, guardrail algorithms, SRSMs, and any post-image processing. There may be at least two neural networks: a localization neural network and a segmentation neural network. The localization neural network, among other actions, is configured to detect the presence of a sponge in the image and provide a bounding box around the detected sponge. The bounding box may include information about the detected sponge, such as width, height, x-shift, and y-shift. In addition, the localization neural network may be configured to determine whether an unstretched sponge, a partial sponge, or no sponge is displayed within the display window. The segmentation neural network provides a pixel-by-pixel prediction of the sponge segmentation mask, i.e., whether a pixel is a sponge pixel or a background pixel. The segmentation neural network may utilize data from both the optical sensor and the depth sensor, or alternatively utilize data from only the optical sensor. The localization marker receives the output from the localization neural network, and the segmentation marker receives the output from the segmentation neural network. The marker is provided to the SRSM for identifying a video-based state based on the dominant category prediction of a preset number of previous frames. The output of the sponge identification algorithm is provided to the hemoglobin estimation algorithm.

[0015] First, a sponge recognition algorithm determines whether a surgical sponge is present. Next, the method includes determining whether the surgical sponge is folded by determining whether the surgical sponge is at least substantially square or rectangular. The localization neural network generates a bounding box for the detected sponge. Based on the outer edges of the bounding box, the localization marker determines whether the surgical sponge is sufficiently square or rectangular. The determination can be associated with the sponge type. This step may include determining whether the aspect ratio of the displayed surgical sponge is within an acceptable range based on the sponge type being counted. The localization neural network provides a bounding box and compares the determined aspect ratio of the bounding box with the acceptable range of aspect ratios for the sponge type. If the determined aspect ratio of the bounding box is outside the acceptable range, the localization marker determines that the sponge is a folded sponge and generates a localization marker accordingly.

[0016] The method may include determining whether the displayed sponge is a partial sponge. If the bounding box of the sponge extends beyond the field of view of the camera, the localization marker determines a partial sponge condition. In addition, the localization marker may determine whether the bounding box is too close to the display window. A margin, i.e., a pixel-based distance between the bounding box and the display window, may be determined and compared to a margin threshold value. If the determined margin is less than the margin threshold value, the localization marker generates a localization marker that the displayed surgical sponge is a partial sponge.

[0017] The sponge identification algorithm may include determining that the sponge is positioned too close or too far from the camera by determining whether the sponge is too large or too small, respectively. The location marker determines a pixel-based area of ​​a bounding box and compares the bounding box area to an acceptable range of bounding box areas unique to each sponge type. The acceptable range of bounding box areas may approximate the display window. If the processor determines that the bounding box area is larger or smaller than the acceptable range of bounding box areas, the location marker generates a location marker that the sponge is too close or too far from the camera, respectively.

[0018] The segmentation neural network outputs a segmentation mask in which the sponge pixels are separated from the background pixels. The segmentation neural network may only be run on the image supply image frames in which the localization neural network has determined that the sponge exists. The segmentation neural network can be run in parallel or sequentially after the localization network to optimize processing speed and resource trade-offs. The method includes validating the sponge segmentation mask by determining the number of pixels in the sponge segmentation mask and comparing the determined number of pixels with an acceptable range of pixels. The acceptable range of pixels depends on the sponge type, which is identified by the processor when it is counted using the data reader. If the segmentation mask determined by the segmentation neural network does not have a sufficient number of pixels, the segmentation mask is determined to be invalid. The segmentation marker can generate a segmentation marker accordingly.

[0019] The segmentation mask refinement algorithm can exclude pixels from the segmentation mask that have different depth values ​​that exceed a threshold compared to most mask pixels. The difference in depth values ​​can be estimated as the distance between the corresponding point and the plane fitted to the depth map. The threshold value may be the same or different for different sponge types. The depth data can be used to establish an acceptable threshold for display distance. The method can include determining whether the displayed surgical sponge is too close or too far. If the average depth of all pixels belonging to the sponge segmentation mask is less than or greater than a minimum threshold or a maximum threshold, respectively, the segmentation marker can output a corresponding segmentation marker that the displayed surgical sponge is too close or the sponge is too far.

[0020] The segmentation marker can use the depth map of the depth data obtained from the depth sensor to determine the true size of the displayed surgical sponge to assess other acceptance criteria. The method may also include determining whether the displayed surgical sponge is the correct sponge. By using the depth map to determine the sponge area of ​​the segmentation mask, the segmentation marker compares the determined true sponge area with an acceptable range of sponge areas, and the acceptable range of sponge areas is based on the calculated sponge type. If the true sponge area of ​​the determined segmentation mask is outside the acceptable range of sponge areas, the segmentation marker outputs an incorrect sponge marker. This step may also include comparing the sponge compactness with a predetermined compactness threshold. The predetermined compactness threshold may be a portion of the area of ​​the bounding box, and the sponge area of ​​the determined segmentation mask must exceed this portion to be determined as correct.

[0021] The method may also include determining whether the displayed surgical sponge is moving. A center of mass of the sponge segmentation mask may be determined, and an average magnitude of motion of the center of mass may be determined over a predetermined number of image frames of the image supply. This step may include comparing the average magnitude of motion to a center of mass change threshold, which may be based on the type of sponge being counted. If the average magnitude of motion is greater than the center of mass change threshold, the segmentation marker determines that the displayed surgical sponge is moving too fast and does not meet the acceptance criteria.

[0022] The segmentation and localization neural networks can be combined into a single neural network that outputs not only classification labels (e.g., no sponge, partial sponge, collapsed sponge, and complete sponge), but also a segmentation mask and an associated bounding box around the detected sponge. In addition, the segmentation and localization labelers can be combined to check each guardrail as a series of condition checks based on the detected sponge features and the expected sponge type. The segmentation and localization neural networks can be executed simultaneously or sequentially.

[0023] The method may include performing sponge supervision, wherein a posture estimation neural network is used to detect the user's hands, torso, neck and other associated body parts, and a positioning neural network or a segmentation neural network is used to detect the positioning of the displayed surgical sponge. Once the person and the displayed surgical sponge are detected and located, the posture estimation neural network is configured to determine whether the displayed surgical sponge is correctly held in front of the torso. The sponge supervision algorithm can identify the body centerline and the sponge vertical centerline, and further determine the horizontal displacement of the displayed surgical sponge relative to the positioning of the body centerline. One or more of the additional centerlines can be compared with the horizontal sponge centerline of the displayed surgical sponge to determine the vertical displacement of the displayed surgical sponge. The horizontal displacement and / or the vertical displacement can be compared with a corresponding relative displacement threshold and / or a predetermined preferred position. If (one or both) displacements exceed the displacement threshold, guidance can be provided on the user interface, instructing the user to display the surgical sponge in front of his torso. The depth map from the depth sensor can be used to distinguish between people in the background and foreground, and the posture estimation algorithm determines which of the multiple people in the camera field of view is displaying the sponge.

[0024] The sponge recognition algorithm includes a sponge recognition state machine (SRSM). The SRSM is configured to receive markers from each of the positioning marker and the segmentation marker and perform further processing before the camera captures the QBL image. The SRSM algorithm determines whether the acceptance criteria or guardrails performed by the positioning marker and the sponge segmentation marker, respectively, are met by combining the aggregation states from both the positioning and segmentation markers. The SRSM algorithm can perform a smoothing procedure to aggregate the positioning and segmentation markers using a sliding window with the size of the image frame. The most frequently aggregated sponge marker is called the dominant marker. If the dominant marker in the required consecutive frames meets the acceptance criteria, the SRSM can indicate that the QBL image is ready to be captured and continued to be analyzed by the hemoglobin estimation engine. If the dominant marker is not identified in the last consecutive image frame, the SRSM may generate an uncertain condition state.

[0025] Once a QBL image is captured, the SRSM may not allow further capture of QBL images, regardless of the proximity markers and the dominant marker, at least until the dominant marker confirms that the sponge is no longer present or absent for a desired number of consecutive image frames of the image supply. Once the sponge is no longer present for a desired number of consecutive frames, the SRSM algorithm is configured to return to a state in which the localization neural network and the segmentation neural network are ready to perform the methods disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1A surgical sponge management system is shown. A user interface may be supported on a stand, and the system includes an optical sensor for capturing a color image of the surgical sponge, and optionally a depth sensor.

[0027] Figure 2 is a flow chart for quantifying blood loss according to an exemplary method of the present disclosure.

[0028] Figure 3 is a perspective view of the user interface at a certain step of the method.

[0029] Figure 4 is a perspective view of a user interface in another step of the method.

[0030] Figure 5 is a perspective view of a user interface in another step of the method.

[0031] Figure 6 is a flow chart of the steps according to the method including acceptance criteria.

[0032] Figure 7 is a schematic illustration of a neural network and algorithms implemented by one or more processors.

[0033] Fig. 8A is a flow chart of processing an image frame of an image supply using a positioning neural network according to an exemplary method of the present disclosure.

[0034] Figure 8B is a flow chart of processing an image frame of an image supply using a segmentation neural network according to an exemplary method of the present invention.

[0035] Fig. 9 is a schematic diagram used to explain the acceptance criteria.

[0036] Fig.10 is a schematic diagram used to explain the acceptance criteria.

[0037] Fig.11 is a schematic diagram used to explain the acceptance criteria.

[0038] Fig.12 is a schematic diagram used to explain the acceptance criteria.

[0039] Fig.13 is a schematic diagram used to explain the acceptance criteria. DETAILED DESCRIPTION

[0040] The present disclosure relates to a system for managing surgical sponges during surgical procedures and quantifying the quantity of blood loss (QBL) associated with blood absorbed on the surgical sponge. Figure 1, the surgical sponge management system 20 may include a support 22, an electronic subsystem 24, and a dispenser assembly 26. The support 22 includes a base 28 with wheels to facilitate maneuvering the surgical sponge management system 20 within a medical facility. The support 22 may include a main support 30 coupled to the base 28 and extending upward from the base 28. The dispenser assembly 26 may be supported on the top of the main support 30, and the sponge sorter 32 may be suspended from an articulated arm of the main support 30. The electronic subsystem 24 includes a module base 34, a user interface 36, and a data reader 38. The module base 34 may be fastened to the support 22. The mount removably couples the user interface 36 to the module base 34 or the support 22. The user interface 36 (and / or the module base 34) may include one or more processors 42, memory, communication devices, and / or other hardware. For example, the user interface 36 may be a tablet computer with a touch screen display. The system 20 includes an optical sensor 44 (e.g., a color camera), and may optionally include a depth sensor 46 (e.g., an infrared camera), collectively referred to herein as a camera 48. The camera 48 may be integrated into or removably coupled to the user interface 36, or may be separate and in wired or wireless communication with the processor 42. In the case where the camera 48 is integrated into the tablet, the tablet may be oriented upside down within the mount so that the camera 48 is positioned on the front lower side of the tablet. This arrangement provides better height alignment for the user, who may comfortably position the surgical sponge in front of their body to be detected by the camera 48 (see FIGS. 8-12 ). In addition, the neural network to be described may be executed on the processor 42 of the tablet, on one or more hardware processors coupled to or remote from the tablet, via cloud computing, via a combination thereof, and the like.

[0041] The data reader 38 can be used as a handheld device or as a Figure 1As shown supported by a bracket. The data reader 38 is configured to detect a tag (T) associated with a surgical sponge (S). In an exemplary embodiment, the data reader 38 is a wireless scanner, i.e., an RFID reader configured to detect an RFID tag, such as those disclosed in co-owned International Publication No. 2021 / 041795, published on March 4, 2021, and International Publication No. 2021 / 097197, published on May 20, 2021, the entire contents of which are hereby incorporated by reference herein. Identifiers other than RFID tags are also contemplated, including optical tags (e.g., bar codes and quick response (QR) codes), and those disclosed in co-owned International Publication No. 2017 / 112051, published on June 29, 2017, the entire contents of which are hereby incorporated by reference herein. In an embodiment used with a surgical item other than a surgical sponge, the identifier may be the aforementioned tag, or ink printed on the surgical item.

[0042] In a typical surgical procedure using surgical sponges, the surgical sponges are counted for use during the surgical procedure. The surgical sponges can be counted by causing a tag to be detected by the data reader 38. For example, a user can position a surgical sponge (or a bundle of surgical sponges) near the data reader 38. The detected tag sends identification data to the processor 42, and the identification data includes at least one unique identifier indicating a characteristic of the surgical sponge. The unique identifier may include sponge size or type (e.g., 2x16 gauze, 4x4 gauze, 4x8 gauze, 4x12 sponge, 12x12 sponge, and 18x18 laparotomy sponge, etc.), material construction (e.g., four layers, eight layers, sixteen layers, etc.), absorbency, stretchability, dry weight, etc. The processor 42 is configured to determine the type of surgical sponge based on the unique identifier. In addition, based on the data reader 38 detecting the tag, the processor 42 can index the counter 64 to reflect that the one or more surgical sponges are counted into the surgical procedure. The counter 64 can be displayed on the user interface 36 (see Figure 3 ). Used and unused surgical sponges are typically counted out, either during or after a surgical procedure. The tag of the surgical sponge is again positioned to be detected by the data reader 38 to identify the surgical sponge as being counted out. The processor 42 can index the counter 64 accordingly (e.g., subtract one from the number of sponges previously counted) and display the remaining number of surgical sponges counted on the user interface 36. The counter 64 can also display the original number of surgical sponges counted, and / or the number of surgical sponges that have been counted out. In practice, the user places the surgical sponge into one of the pockets of the sponge sorter 32 immediately after confirming on the user interface 36 that the surgical sponge has been successfully counted out.

[0043] The disclosed system 20 and methods 100, 200 facilitate rapid and accurate assessment of fluids, particularly blood, contained in surgical sponges. The relevant steps can be completed in the short time between the steps of counting out surgical sponges and placing the surgical sponges into the sponge sorter 32, thereby minimizing disruption to the surgical workflow. Alternatively, these steps can be performed at other points during the surgical procedure as desired by the user. Now referring to Figure 2 , a method 100 for evaluating surgical sponges for blood during surgical procedures is provided. The method 100 includes counting out surgical sponges (step 102). As described above, a tag of the surgical sponge is detected by a data reader 38, and the data contained therein is sent from the data reader 38 to a processor 42. Specifically, the processor 42 receives a unique identifier stored on the tag. The processor 42 identifies the type of surgical sponge from a database of predetermined sponge types based on the unique identifier. For example, the processor 42 may identify the surgical sponge as an 18x18 laparotomy sponge. The sponge type may be prompted or filled in (e.g., "popped up") on the user interface 36, and optionally with an audible indicator indicating that the sponge was successfully scanned so that the user can quickly determine that the sponge type has been counted out. The user interface 36 may index and display a counter 64, including an updated count of each sponge type that has been counted into and out of surgical procedures, such as Figure 3 Certain guardrail data to be described is also based on (eg, is unique to) the type of sponge being counted, and this data is transmitted, loaded or otherwise prepared by processor 42 for image-based processing.

[0044] The method may include triggering the system 20 to perform sponge imaging (step 104). This step may include automatically activating the camera 48 in response to the data reader 38 detecting the tag. The step of automatic activation is optional, and it is alternatively contemplated that the user may provide input to the user interface 36 to activate the camera 48 to perform subsequent steps of the method 100. Alternatively, for the type of procedure that will use the QBL protocol (e.g., methods 100, 200), a mode may be selected on the user interface 36 prior to commencing the surgical procedure, after which the camera 48 is automatically activated in response to the data reader 38 detecting the tag.

[0045] This step may also include displaying an image supply (or supplied image) 50 detected by the camera 48 on the user interface 36. The image supply 50 may also be considered a sponge capture screen (or sponge grab). The image supply 50 may be based on data being received from the optical sensor 44 and / or the depth sensor 46. Figure 4An example of an image feed 50 is shown, wherein an image frame is formed of a video of a user holding a sponge in the field of view 52 of the optical sensor 44. The image feed 50 may also include graphical enhancements or markings based on data from the optical sensor 44 and / or the depth sensor 46. One example includes a presentation window 54 overlaying the image feed 50, while another example includes the image feed 50 being grayscale (based on the depth data) and the segmentation mask 72 of the detected sponge highlighted in green. The user may be trained to "present" the surgical sponge to the camera 48, preferably immediately after the surgical sponge is measured out. The user typically grasps or pinches the upper corners of the surgical sponge to expose the maximum area of ​​the surgical sponge to the camera 48, and instructions to this effect may be provided on the user interface 36. The forward orientation of the camera 48 and the display of the user interface 36 allow the user to utilize the image feed 50 to help align the surgical sponge in real time in a manner to be described.

[0046] The display of the image supply 50 may be automatically performed in response to the data reader 38 detecting the tag. Thus, the camera 48 may be automatically activated and the display of the user interface 36 automatically updated to include the image supply 50. Figure 4 It can be generally seen that the counter 64 has been replaced by the image supply 50, and some other fields of the display have been shifted, resized or removed. In this way, the size of the image supply 50 is designed to make it easy for users standing at an acceptable display distance from the camera 48 to see it.

[0047] The method 100 includes displaying a presentation window 54 on the user interface 36 overlaying the image feed 50 captured by the camera 48 (step 106). The presentation window 54 is a visual marker that guides the user to present the surgical sponge in an appropriate manner and at an appropriate distance from the camera 48. For example, for some sponge types, the distance is between eighteen and thirty-six inches. Figure 4 , the display window 54 is depicted as four "corner brackets" representing the four corners of the display window 54, but the display window 54 need not present any particular geometric shape. The size of the display window 54 is based on the identified type of surgical sponge corresponding to the detected tag of the surgical sponge being counted. The processor 42 can also be configured to determine the acceptance criteria 74 based on the type of surgical sponge - and the size and proportions of the display window 54 can be designed so that if the user presents the surgical sponge in a manner that at least substantially matches or conforms to the display window 54, at least some of the acceptance criteria 74 are met. This step is optional, in which case the processor 42 can determine a dynamic display window based on the actual positioning of the sponge within the optical and / or depth sensor. The processor 42 can dynamically assess the acceptance criteria based on the dynamic display window.

[0048] The method 100 includes a sponge identification algorithm that analyzes the image supply 50 to determine whether the acceptance criteria 74 are met (step 110). This step includes analyzing each image frame (or every few image frames) of the image supply 50 to ensure that a correct surgical sponge is shown intact or fully stretched at a suitable distance, not folded and not moving, and other acceptance criteria 74, also referred to herein as guardrails (see Figure 6 ). If any one or more of the acceptance criteria 74 are still not met, the processor 42 prevents the camera 48 from capturing at least one QBL image of the surgical sponge for QBL analysis. Therefore, the features of the QBL images of the surgical sponge captured by the optical sensor 44 are highly consistent (except for the blood content thereon), so that the sponge recognition algorithm trained to analyze the captured QBL images has a correspondingly high degree of accuracy.

[0049] Once it is determined that the acceptance criteria 74 are met, the method 100 includes the step of acquiring a QBL image of the surgical sponge using the camera 48 (step 112), also referred to herein as a "sponge scan". In one example, the QBL image can be a subsequent frame of the image supply 50 immediately following the acceptance criteria 74 analyzed by the sponge recognition algorithm, including output from the sponge recognition state machine (SRSM) 73. The QBL image can be considered to be different from those image frames of the image supply 50 that are analyzed for meeting the acceptance criteria. Instead, the QBL image is analyzed using a hemoglobin estimation algorithm to determine the blood component (e.g., hemoglobin) of the fluid on the surgical sponge (step 114), for example, in the manner disclosed in the aforementioned U.S. Patent No. 8,897,523 and U.S. Patent No. 10,424,060. Specifically, the processor 42 can be configured to extract color component values ​​(e.g., red values) from the QBL image and execute the hemoglobin estimation algorithm to determine the hemoglobin concentration on a pixel-by-pixel or other suitable manner. The hemoglobin estimation algorithm may be unique to the type of sponge being metered. In other words, a hemoglobin estimation algorithm may be trained on a large dataset for each sponge type to be compatible with the system 20. The amount of blood loss may be determined based on the blood composition as well as the pixel-based or true area of ​​the surgical sponge.

[0050] The user interface 36 may provide a prompt (e.g., a visual and / or audible marker) to notify the user that the QBL image capture was successful and immediately display an item-specific blood loss metric 58 for that particular sponge. Figure 4 The surgical sponge absorbs 9 ml of blood. The user deposits the surgical sponge into the sponge sorter 32. In addition, the user interface 36 can be updated in real time to reflect the patient's cumulative blood loss. The user interface 36 can - after a short delay (e.g., one or two seconds) - return to the counter 64 (see Figure 3), where the cumulative blood loss metric 60 is updated along with the sponge count.

[0051] By using sponge type to determine the characteristics of the display window 54 and acceptance criteria 74, image-based QBL can be applied to more or most types of surgical textiles, including textiles of different sizes, numbers of layers, and / or colors. In contrast, for example, if the system is unaware of the sponge type (and therefore its size), a smaller sponge type imaged at a close distance may be mistaken for a larger sponge type displayed at a farther distance, or the exterior of a larger sponge type imaged at a closer distance may be truncated. Similarly, different numbers of layers may result in images having different red values, especially when imaged in a folded state. The systems 20 and methods of the present disclosure overcome these shortcomings by determining the amount of blood loss on different types of surgical sponges in a seamless manner during a surgical procedure without the need to predefine the type of surgical sponge being used or counted.

[0052] For some surgical sponges, the display window 54 displayed on the user interface 36 may require the user to fold the surgical sponge. One situation may include a large or elongated surgical sponge that, if displayed undone, would require the user to be too far away from the camera 48 for accurate analysis. In addition, the need for folding of some surgical sponges may result in the sizes of the corresponding display windows 54 associated with different sponge types being sufficiently different. Therefore, for each type of surgical sponge, the acceptance criteria 74 are sufficiently different that the user is generally prevented from using an incorrect sponge type or an incorrect folding of the sponge to meet the acceptance criteria 74.

[0053] In some embodiments, method 100 may include a step of displaying a folding protocol to the user on user interface 36 (step 108). This step is optional, and alternatively, display window 54 may require that the surgical sponge be displayed in an unfolded configuration. The folding protocol may include at least two aspects. First, the size of display window 54 is designed to be close to the size of the surgical sponge when folded according to the folding protocol. In other words, if the surgical sponge is displayed in an unfolded form, the display window 54 may be too small for the surgical sponge. Second, the folding protocol may include instructions 62 to facilitate the user to fold the surgical sponge in the desired manner. Instructions 62 may be textual operating instructions or graphic markings (such as Figure 5 In addition to the folding protocol, it is contemplated that the system 20 may require the user to image both sides of the folded surgical sponge.

[0054] refer to Figure 3-5 An exemplary workflow is described. Figure 3The counter 64 shown in FIG. 5 shows that two types of surgical sponges have been counted into the surgical procedure - a 4x8 gauze and an 18x18 laparotomy sponge. The user counts out one of the 18x18 laparotomy sponges. The user interface 36 is refreshed or updated to include the image supply 50 and the display window 54 corresponding to the 18x18 laparotomy sponge, as shown in FIG. Figure 4 as shown. The display window 54 is square and sized to coincide with the field of view 52 of the camera 48. The square shaped display window for the square shaped surgical sponge (without providing further instructions) instructs the user to display the surgical sponge in an unfolded configuration. Once the user aligns the unfolded 18x18 laparotomy sponge with the display window 54, the acceptance criteria 74 are likely met and a QBL image will be captured by the camera 48. A hemoglobin estimation algorithm unique to the 18x18 laparotomy sponge is applied to determine the amount of blood loss on the sponge. Later in the surgical procedure, the user measures out a 4x8 gauze. The user interface 36 is again refreshed or updated to include the image supply 50 and displays the display window 54 corresponding to the 4x8 gauze. From Figure 5 As can be seen, the display window 54 is rectangular and is significantly smaller than the field of view 52 of the camera 48. The user interface 36 also includes instructions 62, i.e., text instructions for manipulating the surgical sponge in a specified manner (i.e., a folding protocol). This is done so that the size of the folded 4x8 gauze approximates the size and shape of the displayed display window 54. Displaying the surgical sponge in a manner different from that required by the instructions 62 effectively prevents the acceptance criteria 74 for the 4x8 gauze from being met, thereby preventing the camera 48 from capturing a QBL image. Once the folded 4x8 is displayed in a manner that approximates the display window 54, the acceptance criteria 74 is likely to be met and a QBL image will be captured by the camera 48. Applying a hemoglobin estimation algorithm unique to the 4x8 gauze sponge, the amount of blood loss on the sponge is determined. A blood loss metric 58 specific to that item is displayed, in this case thirteen milliliters of blood. It should be understood that the methods 100, 200 disclosed herein can be performed in a few seconds without significantly interrupting the workflow and without requiring the user to select a sponge type and / or manually trigger a sponge scan of the surgical sponge displayed within the field of view 52 of the camera 48.

[0055] As described above, the one or more processors 42 implement or execute a sponge identification algorithm. Sponge identification includes configuring a neural network to process an image of a surgical sponge to detect the sponge and apply acceptance criteria 74 or guardrails. In an exemplary embodiment, there are two neural networks: a localization neural network 66 and a segmentation neural network 68. The localization neural network 66 can be a deep neural network, such as a convolutional neural network (e.g., ResNet-50, MobileNet-vl,2, etc.), a Transformer (Transformer architecture)-based neural network (e.g., ViT), or a combination thereof. The output of the localization neural network 66 is modified to have both a classification head and a regression head. The localization neural network 66 is configured to, among other actions, determine the image category: whether a sponge is being shown (sponge present), whether the sponge is folded or broken (folded sponge), and whether the sponge is only partially visible (partial sponge). The localization neural network 66 is also configured to provide a bounding box 70 around the detected sponge (see Figure 10-12 ). The bounding box 70 may include a width (w), a height (h), and an offset (x, y).

[0056] The segmentation neural network 68 can be an encoder-decoder convolutional neural network architecture, such as U-Net, Mask-RCNN, DeepLab, or a Transformer-based neural network architecture that supports image segmentation. The output of the segmentation neural network 68 provides a pixel-by-pixel prediction of the segmentation mask 72 (in Figure 9-11 ), i.e., whether the pixel is a sponge pixel or a background pixel. In an exemplary embodiment, the segmentation neural network 68 can utilize data from both the optical sensor 44 and the depth sensor 46. Alternatively, the segmentation neural network 68 can provide a segmentation mask 72 based solely on the color image. The segmentation neural network 68 is configured to provide a sponge segmentation mask 72 and facilitate determinations involving acceptance criteria 74, among other actions to be described.

[0057] like Figure 7 As shown, the sponge recognition algorithm includes a positioning marker (1abeler) 67, a segmentation marker 69 and a sponge recognition state machine (SRSM) 73. The positioning marker 67 and the segmentation marker 68 are respectively configured to generate a classification label (1abel)-i.e., a positioning label and a segmentation label based on their respective analysis of certain acceptance criteria 74. The positioning and segmentation labels are provided to the sponge recognition state machine (SRSM) 73. Specifically, based on the labels predicted for each image, the SRSM can identify a video-based state. The SRSM uses the dominant category predictions of a preset number of previous frames to ensure that the video-based state is correct even if a single frame may be misclassified.

[0058] The localization neural network 66 can run in real time on slower devices and facilitate real-time user interaction, while the segmentation neural network 68 can be viewed as a larger, more accurate convolutional neural network. The dual-mode design ensures real-time processing of image frames of the image supply 50 without affecting the accuracy of the sponge segmentation mask 72. Alternatively, the neural networks 66, 68 can be combined into a single hybrid network that outputs classification labels, segmentation masks 72, and bounding boxes 70. Such a hybrid neural network can run in real time on faster devices with sufficient graphics processing power. These neural networks can be trained in the manner disclosed herein to perform the methods of the present disclosure with the necessary accuracy. The output of the sponge identification algorithm is provided to a hemoglobin estimation algorithm that is unique to the type of sponge being counted.

[0059] Reference now Figure 6-8B , a method 200 for assessing each acceptance criterion 74 or guardrail is described in turn. The method 200 includes executing an algorithm by at least one of the neural networks 66, 68. Unless otherwise specified, the acceptance criteria 74 can be assessed in the order shown or in any suitable order. First, the method 200 includes determining whether a surgical sponge is present (step 202). The localization neural network 66 determines whether a surgical sponge is present in an image frame of the image supply 50. For example, the surgical sponge may not have entered the field of view 52 of the camera 48, or is too far away from the camera 48 to be detected. If a sponge is not detected, it is determined that the acceptance criterion 74 is not met.

[0060] If a surgical sponge is detected in step 202 , method 200 includes determining whether the surgical sponge is folded (step 204 ). Fig. 9 204 can also include determining whether the surgical sponge is at least substantially square or rectangular. Fig.10, the localization neural network 66 is configured to, among other actions, provide a bounding box 70 around the detected sponge. The bounding box may include information about the detected sponge, such as width, height, x-shift, and y-shift. Based on the outer edges of the bounding box, the localization marker determines whether the surgical sponge is sufficiently square or rectangular. This determination may be associated with the type of sponge. The degree to which the surgical sponge is square or rectangular to meet the guardrail can be based on the localization neural network 66 being trained on a data set of folded and properly held sponges. For example, if an 18x18 laparotomy sponge is counted out, the localization neural network 66 can determine whether the displayed surgical sponge is sufficiently square. If a 4x12 gauze is counted out (and displayed in an unfolded configuration), the localization neural network 66 can determine whether the displayed surgical sponge is sufficiently rectangular.

[0061] The step 204 of determining whether the sponge is folded may also include determining whether the aspect ratio of the displayed surgical sponge is within an acceptable range based on the calculated sponge type. As described above, the bounding box 70 includes a width and a height - the ratio of the two is the determined aspect ratio. If the sponge is determined to be sufficiently square or rectangular, but the determined aspect ratio exceeds the acceptable range of aspect ratios for the sponge type, the positioning neural network 66 determines that the sponge is a folded sponge (step 204). For example, for an 18x18 laparotomy sponge, the manufactured aspect ratio may be approximately 1.0. Since the training of the positioning neural network 66 on the 18x18 laparotomy sponge dataset is performed by the user in different ways, the algorithm implementing the guardrail may have established an acceptable range of aspect ratios, such as between 0.7 and 1.5. These differences account for situations where the user may stretch the sponge, the stretchability of the sponge material, slight distortion when holding from an upper corner, etc. If the user measures out an 18x18 laparotomy sponge and displays the sponge in a manner that the determined aspect ratio is outside of the acceptable range, the localization neural network 66 may determine that the 18x18 laparotomy sponge is folded or in a folded state. Additionally or alternatively, the determined aspect ratio being outside of the acceptable range may indicate that the displayed sponge is the wrong sponge. If a folded sponge (or wrong sponge) is detected, then the acceptance criteria 74 is determined to be not met. The localization neural network 66 generates a localization mark that the displayed surgical sponge is a folded sponge.

[0062] As mentioned above, some surgical sponges need to be folded to cause the size of the corresponding display window 54 of different sponge types to have enough differences.By extension, this is applicable to the acceptable range of aspect ratio.In other words, the acceptable range of aspect ratio has enough differences for each type of surgical sponge, so that the user can usually avoid using incorrect sponge types to meet step 204.For example, if folded according to the folding protocol, the acceptable range of 4x8 gauze may be between 1.7 and 2.2.Adjacent smaller compatible sponge types-for example, 2x16 gauze, if folded according to the folding protocol-may be associated with an acceptable range between 4.9 and 7.8.Equally, adjacent larger compatible sponge types-for example, 4x12 gauze, if folded according to the folding protocol-may be associated with an acceptable range between 2.5 and 3.4.Although these values ​​are merely exemplary, it should be appreciated that the acceptable ranges between 2x1 6 gauze, 4x8 gauze and 4x12 gauze do not overlap. For square shaped sponges - for example, a 12x12 sponge and an 18x18 laparotomy sponge - the acceptable ranges may overlap; however, the correct sponge guard rail (step 216), to be described, resolves these issues.

[0063] The method 200 may include determining whether the displayed sponge is a partial sponge due to being too close to a boundary (step 206). If the displayed segmented representation of the surgical sponge extends beyond the field of view 52 of the camera 48, the localization neural network 66 determines a partial sponge condition. For example, if the outermost pixels of the field of view 52 of the camera 48 are determined to be sponge pixels, and / or if the bounding box 70 is determined to extend through the outermost pixels, the localization neural network 66 determines the partial sponge condition. Also referring to Fig.11 , the positioning marker 66 can determine whether the bounding box 70 is too close to the display window 54. Specifically, the positioning marker 66 determines the margin, that is, the pixel-based distance between the bounding box 70 and the display window 54. Training of the positioning marker 66 on the data set can establish a margin threshold value associated with each of the top, bottom, left and right sides of the display window 54. Alternatively, the margin threshold value can also be input into the software. In one example, the margin threshold value can be set to zero so that: if the bounding box 70 intersects the display window 54, the positioning neural network 66 determines the partial sponge condition. Fig.11 The left side of the bounding box 70 (according to the user's left hand (L)) is shown to pass through the display window 54, resulting in a negative margin. Similarly, the bottom of the bounding box 70 passes through the display window 54. In general, Fig.11In the example of FIG. 7 , the user has placed the sponge too far to the left and too low to meet the acceptance criteria 74. The positioning marker 67 generates a positioning marker that the displayed surgical sponge is a partial sponge. Of course, the image supply 50 (including the display window 54) is displayed on the user interface 36 so that the user can easily adjust the position of the displayed surgical sponge. Hints or markings can be provided on the user interface 36 to guide the user's movements accordingly.

[0064] The positioning neural network 66 and the positioning marker 67 can be further configured to determine whether the sponge is positioned too close or too far from the camera 48 by determining whether the sponge is too large or too small, respectively (step 208). Specifically, the positioning marker 67 is configured to determine a pixel-based area of ​​the bounding box 70 and compare the bounding box area to an acceptable range of bounding box areas that is unique to each sponge type. The positioning marker 67 can be trained on a data set for the algorithm to establish an acceptable range of bounding box areas, or the acceptable range can be input or otherwise determined. In one example, the acceptable range of the bounding box area approximates the display window 54 displayed on the user interface 36, which is also unique to the type of sponge being counted. The acceptable range of the bounding box area is pixel-based, so step 208 can be based solely on the optical image from the optical sensor 44. If the processor 42 determines that the bounding box area is larger or smaller than the acceptable range of the bounding box area, the positioning marker 67 generates a positioning marker that the sponge is too close or too far (from the camera 48), respectively. Fig.12 The area of ​​the illustrated bounding box 70 is significantly smaller than the area of ​​the display window 54, so that, for example, the displayed surgical sponge may be deemed to be too far from the camera 48. Furthermore, the image feed 50 is displayed on the user interface 36 so that the user can adjust the position of the sponge, and corresponding prompts may be provided on the user interface 36 to guide the user's movement.

[0065] If the positioning neural network 66 determines that the sponge is displayed unfolded and properly positioned within the display window 54, the method 200 includes identifying the displayed surgical sponge as a complete sponge (step 210), and the positioning marker 67 accordingly generates a positioning marker for further processing by the SRSM 73.

[0066] Having been processed using the localization neural network 66, the image frames of the image supply 50 are also processed by the segmentation neural network 68. The dual mode processing can occur simultaneously or sequentially. The segmentation neural network 68 outputs a segmentation mask 72 that separates the sponge pixels from the background pixels. Figure 9-11214. The segmentation mask 72 is shown in FIG. 215. The segmentation mask 72 is shown in FIG. 216. The segmentation mask 72 is shown in FIG. 217. The segmentation mask 72 is shown in FIG. 218. The segmentation mask 72 is shown in FIG. 219. The segmentation mask 72 is shown in FIG. ...8. The segmentation mask 72 is shown in FIG. 219.

[0067] refer to Figure 6 and 8B , the segmentation neural network 68 may only be run on image frames of the image supply 50 where the localization neural network 66 has determined that a sponge exists (see step 202). Otherwise, the segmentation mask 72 may be determined to be unavailable or empty. If it is determined that the sponge segmentation mask 72 is not empty, the method 200 includes validating the sponge segmentation mask 72 (step 212). Step 212 may include determining the number of pixels in the sponge segmentation mask 72 and comparing the determined number of pixels to an acceptable range of pixels. The acceptable range of pixels is based on the sponge type, which is identified by the processor 42 when it is counted using the data reader 38. If the segmentation mask 72 determined by the segmentation neural network 68 does not have a sufficient number of pixels, the segmentation mask 72 is determined to be invalid.

[0068] The depth data can be used to establish thresholds for acceptable display distances; that is, lower and upper limits on the actual distances that a user must be from the camera 48 to display the surgical sponge. The method 200 may include determining whether the displayed surgical sponge is too close or too far (step 214). The segmentation marker 69 is configured to output that the sponge is too close or too far if the average depth of all pixels belonging to the sponge segmentation mask 72 is less than or greater than a minimum threshold or a maximum threshold, respectively. As previously described, the localization neural network 66 performs a similar process, comparing the size of the bounding box 70 to the size of the display window 54 (see step 208), which does not rely on depth data or actual size.

[0069] In addition, if the depth map 51 is available, the true area of ​​the imaged sponge can be determined. The segmentation marker 69 does this by determining the sponge area (of the segmentation mask 72) based on the depth map of the image frame of the image supply 50. The determined sponge area is compared to an acceptable range of sponge areas. The determined sponge area can be calculated as the squared root L2-squarednorm of all values ​​in the depth map corresponding to the pixels of the image frame. Training of the segmentation marker 69 on the data set can establish the minimum and maximum areas for each type of surgical sponge, or the acceptance range can be input or otherwise determined.

[0070] Method 200 may also include determining whether the displayed surgical sponge is the correct sponge (step 216). As described above, the degree to which the positioning neural network 66 can determine the correct sponge depends on the comparison of the determined aspect ratio with the acceptable range of the aspect ratio. The segmentation neural network 68 can confirm this determination and further consider the situation where certain sponge types may have overlapping acceptable ranges. By using the depth map to determine the true area of ​​the segmentation mask 72, the segmentation marker 69 is able to distinguish square sponges of different sizes. The segmentation marker 69 can be trained on the data set to establish an acceptable range for the sponge area, or the acceptable range can be input or otherwise determined. For example, at a suitable display distance, the upper threshold of the sponge area of ​​a 12x12 sponge can be 14000AU, while the lower threshold of the sponge area of ​​a 18x18 laparotomy sponge can be 16000AU. Thus, if the user counts out an 18x18 laparotomy sponge and displays a 12x12 sponge in a manner that satisfies the acceptance criteria (e.g., fits within the display window 54), the segmentation marker 69 will determine that the determined sponge area of ​​the displayed surgical sponge is too low relative to the distance it is displayed. The segmentation neural network 68 determines that the acceptance criteria 74 are not met and can generate a segmentation marker accordingly. A corresponding prompt can be provided on the user interface 36 to indicate to the user that the wrong sponge may be displayed.

[0071] The step 216 of determining whether the displayed surgical sponge is the correct sponge may also include comparing the sponge compactness with a predetermined compactness threshold. In some aspects, the step of comparing the sponge compactness may also be viewed as a cross-check between the segmentation neural network 68 and the localization neural network 66, which also determine the area and aspect ratio of the bounding box 70, as described above. The predetermined compactness threshold may be a portion of the area of ​​the bounding box 70 that the determined sponge area of ​​the segmentation mask 72 must exceed to be determined to be correct. For example, the predetermined compactness threshold may be 50%, so that if the determined area of ​​the segmentation mask 72 is less than 50% of the determined area of ​​the bounding box 70, it is determined that the acceptance criteria 74 are not met. This may indicate an erroneous sponge, or an invalid mask (see step 218). A corresponding prompt may be provided on the user interface 36 to request the user to re-display the surgical sponge to the camera 48.

[0072] The method 200 may also include determining whether the displayed surgical sponge is moving (step 218). If the displayed surgical sponge moves too fast, motion blur may result in affecting the accuracy of the QBL analysis. Step 218 may include determining the centroid of the sponge segmentation mask 72, and further determining the average motion magnitude of the centroid within a predetermined number of image frames (e.g., five consecutive image frames) of the image supply 50. The average motion magnitude may be calculated as the L2-distance between consecutive estimated centroids. Step 218 may include comparing the average motion magnitude with a centroid change threshold, which is unique to each sponge type. The centroid change threshold may be based on training the segmentation marker 69 using a wide range of data sets, or may be input into the software. If the average motion magnitude is greater than the centroid change threshold, the segmentation marker 69 determines that the displayed surgical sponge is moving too fast and does not meet the acceptance criteria 74. An alternative embodiment includes tracking the motion of the detected bounding box 70 of the sponge on consecutive image frames of the image supply 50 relative to the threshold. A corresponding prompt may be provided on the user interface 36, requiring the user to hold the sponge steadily.

[0073] Another acceptance criterion 74 or guardrail of method 200 may include performing sponge supervision (step 220). As described above, the processor 42 may be configured to extract red values ​​from the bloody sponge to determine the concentration of blood components on a pixel-by-pixel or other suitable basis. The red color of the image pixels is affected by ambient lighting, such as light differences within a medical facility. It is known to perform light normalization using calibration plaques to account for linear differences in ambient light intensity and color temperature. While generally effective, light "bleeding through" from a directional light source through the back of a displayed surgical sponge may cause the prediction of blood composition to be underestimated. Light bleed-through may be exacerbated if the user holds the sponge farther from the body, exposing a larger portion of the surgical sponge to the light source located behind the user within the medical facility.

[0074] Step 220 of performing sponge supervision may include detecting the user's hands, torso, neck, and other associated body parts using a pose estimation neural network and a pose estimation marker (not identified), as well as detecting the positioning of the displayed surgical sponge in the manner previously described. Once the person and the displayed surgical sponge are detected, the pose estimation marker may be configured to determine whether the displayed surgical sponge is properly held in front of the torso, which tends to limit (or reduce) light transmission. Specifically, reference Fig.13 , the posture marker is configured to identify the body centerline 76 and the sponge vertical centerline 78. Based on the identified centerlines 76, 78, step 220 may include determining the horizontal offset of the positioning of the displayed surgical sponge relative to the body centerline 76. In addition, the posture estimation marker can be configured to identify the centerlines 80 or reference points of other anatomical landmarks, such as the head, neck, arms, hands, legs, etc. One or more of the additional centerlines 80 can be compared with the horizontal sponge centerline 82 of the displayed surgical sponge to determine the vertical displacement of the displayed surgical sponge. The horizontal displacement and / or the vertical displacement can be compared with a corresponding displacement threshold value, which is determined by a posture estimation algorithm trained on the data set or input or otherwise determined. If the displacement exceeds the displacement threshold value, guidance can be provided on the user interface 36 to instruct the user to display the surgical sponge in front of his torso. In addition, the depth map from the depth sensor 46 can be used to distinguish between people in the background and the foreground. In conjunction with the known location of the displayed surgical sponge within field of view 52 , the pose estimation algorithm is configured to determine which of the multiple persons within field of view 52 of camera 48 is displaying the sponge.

[0075] If the segmentation neural network 68 determines that the sponge is presented at the appropriate distance and position relative to the user's body, is the correct sponge, has not moved, and the segmentation mask 72 is valid, then the segmentation marker 69 generates a segmentation marker for further processing by the SRSM 73. This is in addition to the positioning marker generated by the positioning marker 67.

[0076] The SRSM is configured to receive the markers from each of the positioning marker 66 and the sponge segmentation marker 68 and perform further processing before the camera 48 captures the QBL image. The SRSM is configured to determine whether the positioning marker and the segmentation marker are the same. In other words, the SRSM determines whether the acceptance criteria 74 or guardrails performed by the positioning neural network 66 and the sponge segmentation neural network 68, respectively, have been met. Since the image supply 50 is a video and the image frames actually occur in an imperceptibly rapid succession, the image state of one or more of the acceptance criteria 74 may change the result rapidly. The SRSM can perform a smoothing procedure to aggregate the positioning and segmentation markers using a sliding window with an image frame size (L). The sliding window takes the state between the previous time "tL" and time "t-1", but may not include the most immediate state (at time t). The sponge marker that is most frequently aggregated is called the dominant marker. If the required consecutive frames (K) (including the close markers and the dominant marker) meet the acceptance criteria 74, the SRSM can indicate that the QBL image is ready to be captured and analyzed by the hemoglobin estimation engine. As the number of required consecutive frames increases, the probability of false positive scans decreases. The parameters L, K can be adjusted based on empirical evaluation. The SRSM may generate an uncertain video state if it is identified as a dominant marker in the latest K consecutive image frames. For example, an uncertain video state may be generated if the frequency of the most frequently aggregated sponge marker in the sliding window is less than half the sliding window size or other suitable values.

[0077] Once a QBL image is captured, the sponge recognition algorithm is configured to immediately place the system 20 into a "cooldown" state so that the user can remove the displayed surgical sponge from the field of view 52 of the camera 48. Regardless of the proximity marker and the dominant marker, the SRSM may not allow further QBL images to be captured, at least until the dominant marker confirms that the sponge is no longer present or absent in a desired number of consecutive image frames (M) of the image supply (see step 202). The parameter M can be adjusted or otherwise determined based on empirical evaluation. Once the sponge is no longer present in the desired number of consecutive frames, the SRSM is configured to return to a video state in which the localization neural network 66 and the segmentation neural network 68 are ready to perform the methods 100, 200 disclosed herein. In practice, this allows only one sponge scan to be captured for each displayed surgical sponge. The user is appropriately instructed to remove the sponge from the field of view 52 of the camera 48 before the next surgical sponge is displayed.

[0078] As described above, supervised, semi-supervised and / or unsupervised learning methods can be used to train neural networks 66, 68 in order to perform the methods of the present disclosure, particularly with the necessary accuracy. A diverse and representative set of training data sets is collected to train each neural network in different scenarios within and outside the intended use of the system 20. The training method may include data enhancement, pre-training neural networks and / or using semi-supervised learning methods to reduce the requirements for labeled training data sets. Data enhancement can increase the sample size, such as lighting enhancement (brightness, contrast, hue saturation shift, Planckian Jitter, etc.), geometric transformation (rotation, homography, flip, etc.), noise and blur addition, custom enhancement (artificially enlarging or reducing the size of the sponge, cropping the sponge), etc.

[0079] The dataset was varied to include the following parameters: sponge size, sponge saturation, ground truth (blood color), lighting, imaging distance, sponge orientation, and position. For example, the sponge can be saturated by changing the level of the total value until the maximum blood carrying capacity is reached; the imaging distance can be varied from 0.5 to 1.5 meters; the lighting can be brighter or darker; the sponge orientation can be rotated in-plane or out-of-plane; and the position can be varied with images of folded, unfolded, and slightly folded sponges. In addition, a set of examples of sponge-like non-sponge objects can be included in the dataset to train the neural network to recognize white or red non-sponge objects as part of the background.

[0080] Other approaches include training a neural network using both color images and depth data to jointly learn segmentation masks using both color images and depth images. Specific architectures such as ESANet can be used to effectively combine color images and depth data. The segmentation neural network 68 can also be trained for images without depth data (e.g., only color images). If depth data is not available, a data set can be generated that includes pixel areas within the segmentation mask associated at different imaging distances. Based on the depth of the sponge scan in the color image, the segmentation mask, and the known sponge type, the segmentation neural network 68 is configured to determine whether the displayed surgical sponge is scanned too close or too far from the system. Therefore, the segmentation neural network 68 is configured to provide a segmentation mask 72 based only on the color image from the optical sensor 44. Finally, for sponge-supervised pose estimation neural networks, example models can include AlphaPose, MobileHumanPose, MoveNet, OpenPose, etc., which can be further trained on application-specific data sets, such as nurses imaging sponges.

[0081] The foregoing disclosure is not intended to be exhaustive or to limit the disclosure to any particular form. The terms used are intended to be descriptive rather than restrictive. Based on the above teachings, many modifications and variations can be made, and the present invention can be practiced in a different manner from the specific description.

[0082] Other inventive aspects of the present disclosure are disclosed with reference to the following exemplary clauses.

[0083] Item 1 - A method for evaluating a surgical item using a system including a data reader, one or more processors, a user interface, and an optical sensor, the method comprising: identifying, using the one or more processors, an item type of the surgical item from a database of predetermined types of surgical items; detecting a displayed surgical item using a field of view of the optical sensor; capturing at least one image of the displayed surgical item; determining, using the one or more processors, whether the displayed surgical sponge meets at least one acceptance criterion, wherein the acceptance criterion is based on the identified item type scanned with the data reader; and displaying on the user interface at least one of: confirmation that the displayed surgical item is the same type as scanned with the data reader, and information related to the status or condition of the surgical item.

[0084] Item 2 - A method according to Item 1, wherein each of the surgical items includes a tag storing a unique identifier, the method further comprising: detecting the tag of the surgical item using a data reader, wherein the unique identifier stored on the tag is received by a processor; and identifying the item type based on the unique identifier using one or more processors.

[0085] Clause 3 - A method according to clause 1 or 2, wherein the step of determining whether the characteristics of the surgical object meet acceptance criteria includes: generating a segmentation mask of the displayed surgical object using one or more processors; determining the number of pixels in the segmentation mask using one or more processors; comparing the determined number of pixels of the segmentation mask with a database image in which the surgical object is in an unused or undamaged condition using one or more processors; and if the determined number of pixels is outside an acceptable range of pixels, displaying information on a user interface indicating that the condition of the surgical object is damaged.

[0086] Clause 4 - A method according to clause 3, wherein the system includes a depth sensor, and the step of determining whether the features of the surgical item meet the acceptance criteria includes: using one or more processors to determine the real item area of ​​the segmentation mask based on depth data from the depth sensor; using one or more processors to compare the real sponge area of ​​the determined segmentation mask with an acceptable range of the item area, wherein the acceptable range of the item area is based on the identified item type; if the real sponge area of ​​the determined segmentation mask is outside the acceptable range of the item area, using one or more processors to identify the displayed surgical item as an incorrect item.

[0087] Clause 5 - A method according to any one of clauses 1-4, wherein the step of determining whether a feature of a surgical object meets acceptance criteria comprises: using one or more processors to generate a bounding box around or surrounding a displayed surgical object; using one or more processors to determine an aspect ratio of the bounding box; and using one or more processors to compare the determined aspect ratio of the bounding box with an acceptable range of aspect ratios, wherein the acceptable range of aspect ratios is based on the identified object type; if the determined aspect ratio of the bounding box is not within the acceptable range of aspect ratios, using one or more processors to identify the displayed surgical object as a surgical object of the wrong type.

[0088] Clause 6 - A method according to clause 5, wherein the step of determining whether the characteristics of the surgical object meet the acceptance criteria includes: using one or more processors to determine an area within a bounding box; using one or more processors to compare the determined bounding box area with an acceptable range of the bounding box area, wherein the acceptable range of the bounding box area is based on the identified object type; if the determined bounding box area is outside the acceptable range of the bounding box area, then using one or more processors to identify the displayed surgical object as being too close or too far from the optical sensor.

Claims

1. A method of evaluating surgical sponges stained with blood during a surgical procedure using a sponge management system, the sponge management system comprising a data reader, one or more processors, a user interface, and an optical sensor, wherein each of the surgical sponges comprises a tag storing a unique identifier, the method comprising: using the data reader, detecting a tag of a surgical sponge, wherein a unique identifier stored on the tag is received by the one or more processors; identifying, using the one or more processors, a sponge type of surgical sponge from a database of predetermined types of surgical sponges based on the unique identifier; detecting the displayed surgical sponge using the field of view of the optical sensor; determining, using the one or more processors, whether the displayed surgical sponge meets at least one acceptance criterion, wherein the acceptance criterion is based on an identified sponge type of the counted surgical sponge; If it is determined that the acceptance criteria are met, capturing a QBL image of the displayed surgical sponge using an optical sensor; estimating, using the one or more processors, an amount of blood or a blood component based on the QBL image; as well as The amount of the blood or blood component is displayed on the user interface.

2. The method of claim 1 , further comprising displaying a presentation window on the user interface overlaying the image feed captured by the optical sensor, wherein a size of the presentation window is based on an identified sponge type of the surgical sponge.

3. The method of claim 2, further comprising providing operating instructions on the user interface to manipulate the displayed surgical sponge according to a folding protocol based on the identified sponge type, wherein the size of the display window is also based on the size of the surgical sponge following the folding protocol.

4. The method according to claim 2 or 3, wherein: The steps to determine whether the characteristics of the surgical sponge meet the acceptance criteria include: generating, with the one or more processors: a label indicating whether the sponge-like object is fully displayed within the display window in a stretched configuration, and a bounding box around or surrounding the displayed surgical sponge; and Using the one or more processors, a determination is made as to whether a bounding box of the displayed surgical sponge is square or rectangular within a predetermined geometric threshold.

5. The method according to claim 4, wherein: The steps to determine whether the characteristics of the surgical sponge meet the acceptance criteria also include: determining, with the one or more processors, an aspect ratio of the bounding box; and comparing, with the one or more processors, the determined aspect ratio of the bounding box to an acceptable range of aspect ratios, wherein the acceptable range of aspect ratios is based on an identified sponge type of the surgical sponge; If the determined aspect ratio of the bounding box is not within an acceptable range of aspect ratios, determining, using the one or more processors, that the displayed surgical sponge is in a folded condition; and The optical sensor is prevented from capturing a QBL image of the surgical sponge shown in a folded condition.

6. The method according to claim 4 or 5, wherein: The steps to determine whether the characteristics of the surgical sponge meet the acceptance criteria include: determining, with the one or more processors, a margin between the bounding box and the presentation window; comparing, using the one or more processors, the margin to a margin threshold value; If the determined margin is less than the margin threshold value, determining, using one or more processors, that the displayed surgical sponge is a partial sponge; and Block the optical sensor from capturing the QBL image of the portion of the sponge.

7. The method according to claim 6, wherein: The margin determined is the pixel-based distance between the bounding box and the presentation window.

8. The method according to any one of claims 4 to 7, wherein: The steps to determine whether the characteristics of the surgical sponge meet the acceptance criteria include: determining, using the one or more processors, an area within the bounding box; comparing, with the one or more processors, the determined area of ​​the bounding box to an acceptable range of bounding box areas, wherein the acceptable range of bounding box areas is based on an identified sponge type of the surgical sponge; If the determined bounding box area is outside of an acceptable range of bounding box areas, determining, using one or more processors, that the presented surgical sponge is too close or too far from the optical sensor; and The optical sensor is prevented from capturing a QBL image of a surgical sponge that is presented too close or too far away.

9. The method according to any one of claims 1 to 8, wherein: The steps to determine whether the characteristics of the surgical sponge meet the acceptance criteria include: generating, using the one or more processors, a segmentation mask of the displayed surgical sponge; determining, using the one or more processors, a number of pixels in a segmentation mask; comparing, with the one or more processors, the determined number of pixels of the segmentation mask to an acceptable range of pixels, wherein the acceptable range of pixels is based on an identified sponge type of the surgical sponge; If the determined number of pixels of the segmentation mask is less than a lower limit of an acceptable range of pixel numbers, determining, using one or more processors, an invalid mask condition; and The optical sensor is prevented from capturing a QBL image of the displayed surgical sponge with an invalid mask condition.

10. The method according to any one of claims 1 to 8, wherein: The system includes a depth sensor, and the step of determining whether a characteristic of the surgical sponge meets an acceptance criterion includes: generating, using the one or more processors, a segmentation mask of the displayed surgical sponge; determining, with the one or more processors, a true sponge region of the segmentation mask based on depth data from the depth sensor; comparing, using one or more processors, the determined true sponge area of ​​the segmentation mask to an acceptable range of sponge areas, wherein the acceptable range of sponge areas is based on an identified sponge type of the surgical sponge; If the determined true sponge region of the segmentation mask is outside the acceptable range of the sponge region, identifying the displayed surgical sponge as an erroneous sponge using one or more processors; and Utilizing the one or more processors prevents the optical sensor from capturing a QBL image of an erroneous sponge.

11. The method of claim 10, further comprising excluding, using one or more processors, pixels from the segmentation mask if a difference in depth value between the estimated corresponding point and a plane fitted to the depth map exceeds a threshold.

12. The method according to claim 11, wherein: The threshold may be the same for all sponge types.

13. The method according to any one of claims 1 to 8, wherein: The steps to determine whether the characteristics of the surgical sponge meet the acceptance criteria include: generating, using the one or more processors, a segmentation mask of the displayed surgical sponge; determining, using the one or more processors, a centroid of the segmentation mask; determining, with the one or more processors, an average magnitude of motion of the centroid within a predetermined number of image frames of the image supply; comparing, with the one or more processors, the determined average magnitude of motion of the center of mass to a center of mass change threshold; If the determined average magnitude of movement of the center of mass is greater than a center of mass change threshold, identifying, using one or more processors, the displayed surgical sponge as a moving sponge; and The optical sensor is prevented from capturing a QBL image of the moving sponge.

14. The method according to any one of claims 1 to 13, wherein: The steps to determine whether the characteristics of the surgical sponge meet the acceptance criteria include: detecting, with the one or more processors, a torso of a user in a field of view of an optical sensor; identifying, with the one or more processors, a body centerline of a torso of a user and a vertical centerline of a displayed surgical sponge; determining, with the one or more processors, a horizontal displacement of the vertical centerline relative to a body centerline; utilizing the one or more processors, comparing the horizontal shift to a horizontal shift threshold; and If the horizontal shift exceeds a horizontal shift threshold, guidance is displayed on the user interface to instruct the user to move the displayed surgical sponge.

15. The method according to claim 14, wherein: The steps to determine whether the characteristics of the surgical sponge meet the acceptance criteria also include: identifying, with the one or more processors, an additional centerline of the user and a horizontal centerline of the displayed surgical sponge; determining, with the one or more processors, a vertical displacement of the horizontal centerline relative to the additional centerline; comparing, with the one or more processors, the vertical shift to a vertical shift threshold; and If the vertical displacement exceeds the vertical displacement threshold, guidance is displayed on the user interface to instruct the user to move the displayed surgical sponge.

16. The method according to any one of claims 1 to 15, further comprising: The one or more processors receive the localization markers from the localization neural network and the segmentation markers from the segmentation neural network; The one or more processors generate a polymeric sponge tag; The one or more processors perform a sliding window protocol in which the most frequently aggregated sponge tag is identified as the dominant tag; as well as The one or more processors determine that the QBL image is ready for capture if the dominant indicia indicates that the acceptance criteria have been met for a preset number of most recent consecutive image frames supplied for the image.

17. A method of evaluating surgical sponges stained with blood during a surgical procedure using a sponge management system, the sponge management system comprising a data reader, one or more processors, a user interface, a depth sensor, and an optical sensor, wherein each of the surgical sponges comprises a tag storing a unique identifier, the method comprising: using a data reader, detecting a tag of the surgical sponge, wherein a unique identifier stored on the tag is received by the one or more processors; using the one or more processors, identifying that the surgical sponge is discharged from a surgical procedure; identifying, using the one or more processors, a sponge type of a surgical sponge from a database of predetermined sponge types based on the unique identifier; Using the optical sensor, capturing a QBL image of the surgical sponge; using the one or more processors, estimating an amount of blood or blood components on the surgical sponge based on the color component values ​​of the QBL image and the identified sponge type; and The amount of the blood or blood component is displayed on the user interface.

18. The method according to claim 17, wherein: A hemoglobin estimation algorithm is trained for each of the predetermined sponge types, wherein the step of estimating the amount of the blood component further comprises applying, using one or more processors, a selected one of the hemoglobin estimation algorithms corresponding to the identified sponge type.

19. A method of evaluating a surgical sponge for blood during a surgical procedure using a sponge management system, the sponge management system comprising a data reader, one or more processors, a user interface, an optical sensor, wherein: Each of the surgical sponges includes a tag storing a unique identifier, the method comprising: detecting, using a data reader, a tag of the surgical sponge, wherein a unique identifier stored on the tag is received by the one or more processors; identifying, with the one or more processors, a sponge type of surgical sponge from a database of predetermined types of surgical sponges based on the unique identifier; displaying a presentation window on the user interface overlaying the image captured by the optical sensor, the presentation window having a size based on the identified sponge type of the surgical sponge; Once the surgical sponge is displayed in the display window, a QBL image of the surgical sponge is captured using an optical sensor; estimating, with the one or more processors, an amount of blood or blood components on the surgical sponge based on the QBL image; and The amount of the blood or blood component is displayed on the user interface.

20. A method of evaluating surgical sponges stained with blood during a surgical procedure using a sponge management system, the sponge management system comprising a data reader, one or more processors, a user interface, and an optical sensor, wherein each of the surgical sponges comprises a tag storing a unique identifier, the method comprising: using the data reader, detecting a tag of a surgical sponge, wherein a unique identifier stored on the tag is received by a processor; identifying, using the one or more processors, a sponge type of surgical sponge from a database of predetermined types of surgical sponges based on the unique identifier; activating an optical sensor based on detecting the tag; displaying, on the user interface, a supply of images captured by the optical sensor; capturing a QBL image of the surgical sponge using the optical sensor; using the one or more processors, estimating an amount of blood or blood components on the surgical sponge based on the QBL image; and The amount of the blood or blood component is displayed on the user interface.

21. The method according to claim 20, wherein: The step of displaying the image provision is automatically performed based on detecting the tag.

22. The method according to claim 20 or 21, wherein: The system includes a depth sensor, and wherein the method further includes automatically activating the depth sensor based on detecting the tag.

23. The method according to any one of claims 20 to 22, further comprising: displaying a counter on a user interface, the counter including the number of surgical sponges counted into and out of a surgical procedure; and Updated the user interface to remove the display of the counter and include displaying the image feed captured by the optical sensor.

24. The method according to claim 23, further comprising: In response to the QBL image being captured by the optical sensor, displaying an item-specific blood loss metric adjacent to the image supply; and updating the user interface to remove the display of the image supply and to include the display counter again; and A cumulative blood loss measure is displayed adjacent to the counter.

25. The method according to claim 23, wherein: The step of updating the user interface to display the counter occurs after a delay of at least one second.

26. A surgical sponge management system, comprising: Data reader; user interface; Optical sensors; and One or more processors in electronic communication with the data reader, the user interface, and the optical sensor, wherein the one or more processors are configured to perform the method of any one of claims 1-25.

27. A non-transitory computer readable medium storing instructions which, when executed on one or more processors, are configured to perform the method of any one of claims 1-25.

Images