Method for assessing the amount of blood constituents in surgical fabrics

Abstract

A system and method for detecting, counting, and analyzing blood content in surgical tissue is provided. [Solution] A system for measuring blood content is provided, comprising: a color imaging system configured to acquire color images; an infrared imaging system configured to acquire infrared images; and a processor configured to identify a shape of a surgical fabric using the infrared images, generate shape information from the infrared images, use the shape information to identify surgical fabric pixels in the color images, and extract blood information from the surgical fabric pixels.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 387,222, filed December 23, 2015. This application is also related to U.S. Patent Application No. 13 / 544,664, filed July 9, 2012, which is incorporated by reference in its entirety. [Background technology]

[0002] The present invention relates generally to the field of blood loss management, and more specifically to a novel and useful method for assessing the amount of blood constituents within surgical fabrics in the field of blood loss management. Summary of the Invention [Problem to be solved by the invention]

[0003] The present invention relates generally to the field of blood loss management, and more particularly to a novel and useful method for assessing the amount of blood constituents in surgical fabrics in the field of blood loss management. A system and method for detecting, counting and analyzing blood content in surgical fabrics is provided, utilizing an infrared or depth camera for color imaging. [Means for solving the problem]

[0004] In one embodiment, a system for measuring blood content is provided, comprising a color imaging system configured to acquire color images, an infrared imaging system configured to acquire infrared images, and a processor configured to identify a shape of a surgical fabric using the infrared images, generate shape information from the infrared images, and use the shape information to identify surgical fabric pixels in the color images and extract blood information from the surgical fabric pixels. The processor may be further configured to transform the infrared images into a coordinate system corresponding to the color images, generate a depth image from one of the infrared images, identify a set of planar pixels in the depth image, and / or detect a perimeter of the image of the surgical fabric in the depth image. Detection of the perimeter of the image of the surgical fabric may be performed by looking for abrupt vertical depth drops in the depth images. The processor may be further configured to: down-resolution process the depth image or the infrared image, generate a mask image from the down-resolution processed depth image, up-resolution process the mask image, perform filling of missing infrared pixel regions in the infrared image, generate the mask image using the detected perimeter, and / or transform the mask image into the coordinate system of the color image. The detected perimeter may be the top and bottom edges of the image of the surgical fabric, or may be side edges of the image of the surgical fabric and / or corners of the image of the surgical fabric.

[0005] In another embodiment, a method for analyzing an object containing a blood component is provided, comprising obtaining a depth image and a color image of an object containing a blood component, determining the presence of the object in a field of view using the depth image, and evaluating a property of the blood component using the color image. The method may further comprise generating an image mask using the depth image and applying the image mask to the color image to remove background pixels. The method may further comprise applying at least one of the following image processes to the depth image: erosion and dilation. This may be included.

[0006] In yet another embodiment, a method of assessing the amount of a fluid component in a surgical fabric is provided, comprising: acquiring a first infrared image of a field of view at a first time; detecting a shape or surface characteristic of the surgical fabric in the first infrared image; indexing a frame counter when the shape or surface characteristic is detected in the first infrared image; acquiring a second infrared image of the field of view at a second time subsequent to the first time; detecting a surface characteristic of the surgical fabric in the second infrared image; indexing a frame counter when the shape or surface characteristic is detected in the second infrared image; and indexing a non-detection counter when the shape or surface characteristic of the surgical fabric is not detected in the second infrared image. The method may further comprise clearing the frame counter and the non-detection counter when a pre-specified value is reached. The method may further comprise comparing the frame counter to a threshold value; acquiring a color image of the field of view when the frame counter may be equal to or greater than the threshold value; converting an area of ​​the color image corresponding to the surgical fabric to an assessment of the amount of a blood component; and clearing the frame counter. The method may further include generating an image mask using the second infrared image and using the image mask on the color image to identify an area of ​​the color image that corresponds to the surgical fabric.

[0007] In another embodiment, a method for performing fluid monitoring is provided and includes acquiring an infrared image of a field of view, acquiring a color image of the field of view, generating an image mask using the infrared image, and using the image mask on the color image to identify a region of the color image that corresponds to the surgical fabric. The method may further include converting the infrared image to a geometric perspective view of the color image. The method may further include low resolution processing of the infrared image before generating the image mask. The method may also further include high resolution processing of the image mask before using the image mask to identify a region of the color image. [Brief description of the drawings]

[0008] [Figure 1]1 is a flow chart representation of a method for analyzing a surgical fabric. [Diagram 2] 1 is a flowchart representation of one variation of a method for identifying and counting surgical textiles for a particular characteristic. [Diagram 3] 1 is a schematic representation of an exemplary depth imaging system. [Figure 4] 1 illustrates an exemplary computing device and user display highlighting a surgical fabric. [Diagram 5] 1 is a flow chart representation of a method for detecting removal of a surgical fabric from a field of view. [Figure 6] 1 is a flowchart representation of a method for unwarping and re-evaluating an image. [Figure 7] 1 is a flowchart representation of a method for generating an image mask. [Figure 8] 11 is another flowchart representation of a method for generating an image mask using an erosion and dilation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Overestimation and underestimation of patient blood loss is a major contributor to the high cost of surgery and surgical care for hospitals, clinics, and other medical facilities. Specifically, overestimation of patient blood loss can result in wasted transfusion grade blood, driving up surgery costs for medical institutions, and can lead to blood shortages. Underestimation of patient blood loss is a significant factor in delayed resuscitation and transfusion during severe bleeding events, and is associated with a large number of avoidable patient infections, re-admissions, and annual litigation. Thus, the surgical field needs new and useful systems and methods for assessing extracorporeal blood volume in a physical sample.

[0010] The systems and methods described herein utilize a color camera system to acquire images of surgical fabrics or other operating room surfaces and evaluate the blood content within each image. Additionally, the system can track multiple surgical textiles used during surgery to assist the surgeon or surgical staff in verifying that all surgical textiles are present and not left in the patient prior to surgical closure. Surgical textiles may include items such as absorbent surgical gauze sponges, surgical clothing, surgical towels, surgical gowns, scrubs or other clothing, tables, beds or bed sheets, walls, floor surfaces, external skin surfaces, surgical gloves, surgical instruments, or any other surface, material, substrate, or object.

[0011] To facilitate presentation of objects to a color camera system for analysis and / or counting, infrared or other types of cameras configured to provide depth information may be used to detect presentation of objects for analysis. Depth information from such systems may more easily distinguish objects located in the foreground from other objects that are not intended to be analyzed and would be located in the background. Examples of optical systems that can provide depth information include range-gated time-of-flight (ToF) cameras, RF modulated ToF cameras, pulsed light ToF, and projected light stereo cameras, among others.

[0012] To improve the accuracy of the detection process, other features of the object may also be detected to reduce the risk of false positive results. The reliability of the detection process may also be improved by identifying the presence of the object in multiple consecutive frames before analyzing the object and providing a result.

[0013] 1.Application 1, various methods described herein may be performed by a computing device 100 to acquire one or more images 102 using an optical sensor system 104 to detect a surgical fabric 106a, such as a surgical gauze, surgical sponge, or surgical towel, within a field of view 108, as shown in S110. At S111, the image 102 is analyzed to confirm the presence of a surgical fabric (image) 106b across a sequence of frames output by the optical sensor system, and if its presence is confirmed, at S125, a color image 110 of the surgical fabric 106b is automatically acquired or the last color image currently captured of the image 102 is utilized. The computing device 100 can then index the local surgical fabric counter 112 in S150 and locally convert the color values ​​of the pixels of the color image 110 to an estimate of the amount of blood components (e.g., hemoglobin mass, total blood volume) within the surgical fabric 106b in S140, or pass the color image 110 to a distal computing system for such processing.

[0014] In particular, the computing device 100 may be configured to detect a possible surgical fabric image 106 in a first frame (e.g., an infrared image), index the frame counter 114 per block S112, and either re-index the frame counter 114 if the possible surgical fabric 106 is detected again in the next frame, or clear the frame counter 114 per block S130 if the surgical fabric image 106b is not detected in the next frame. This process is repeated and the frame counter 114 indexed as long as the possible surgical fabric 106b is detected in the next frame and until the frame counter 114 contains a value equal to a certain threshold value 116. Once the frame counter contains a value equal to the threshold value 116, the computing device may record a color image 110 of the surgical fabric 106, process the color image 110 to determine its blood constituent (e.g., hemoglobin) content, block S140, and / or update the fabric counter 112 with respect to the detected surgical fabric 106, block S150. For example, the computing device 100 may satisfy the threshold value 116 of "7," which correlates to seven consecutive frames that include an area determined to likely correspond to the surgical fabric 106. Additionally, the computing device 100 may update the frame counter 114 to a value of "7." , a digital color photographic image 110 of the surgical fabric 106b can be obtained and the image 110 can be processed to determine the mass or volume of hemoglobin or total blood components within the surgical fabric 106. In other embodiments, the threshold can be set in a range of about 3 to about 100, or about 5 to about 70, or about 30 to about 50, or about 8 to about 20. By tracking the surgical fabric 106 through a sequence of frames and only confirming the presence of the surgical fabric 106b when it is detected within a minimum number of consecutive frames, the computing device 100 may minimize or eliminate false positive confirmations of the surgical fabric 106b, thereby minimizing or eliminating erroneous fabric counter events, as well as automated collection and processing of images that do not include the surgical fabric. This tracking process may also improve computational efficiency by reducing the need to perform a blood content determination on every color image.

[0015] In further embodiments, the computing device 100 may also be configured to automatically detect the surgical textile 106a within its field of view 108 (i.e., within the field of view of one or more optical sensors of the optical sensor system 104 incorporated and / or connected to the computing device 100), thereby reducing or eliminating the need for a user to manually trigger the acquisition of the blood-containing surgical textile, such as by depressing a foot pedal, touch control, gesture control, or voice control. Additionally, the computing system 100 may be configured to provide real-time visual feedback through an incorporated or connected display 118, such as a visual indicator that the surgical textile 106a has been detected within the field of view 108 of the computing device 100, that an image of the surgical textile has been acquired, and / or a prompt to manipulate the surgical textile 106a to provide a satisfactory or high quality image.

[0016] The methods described herein may be performed locally by a computing device 100 installed or located in an operating room, transport room, ambulance, or any other medical or emergency response space or unit. However, the methods may also be performed locally and / or remotely, such as by a remote server. The blocks of the methods are described herein as detecting a surgical sponge, and acquiring and processing images of the surgical sponge. However, the blocks of the methods may additionally or alternatively be implemented to detect surgical gauze, surgical towels, or any other textile, and to acquire and process images of such textiles.

[0017] 2. Exemplary Methods 1, a schematic method S100 for assessing the amount of blood constituents in a surgical fabric includes, at a first time, acquiring a first infrared or depth image 102 of a field of view 108 using an optical system 104 in block S110. In this particular embodiment, the optical system 104 comprises a computing device 100 having an integrated display 118 and a color camera or color imaging system 120 on the same side of the computing device as the display 118, although in other embodiments, the computing device 100 may be configured to use a camera on the opposite side of the display 118. To acquire the depth image 102, an infrared camera or depth imaging system 122 is mounted directly on the computing device 100, although in other embodiments, the depth imaging system 122 may be mounted at a location remote from the computing device 100. In some variations, the infrared or depth camera 120 is configured to be mounted on the particular computing device 100 such that the optical axis of this second camera 121 is aligned vertically and parallel to the optical axis of the color camera 120. However, in other embodiments, the optical axis of the second camera 121 may be aligned with the horizontal axis of the color camera 120 or may not be aligned with any axis of the color camera. The second camera 122 may alternatively be located about 15 cm to 20 cm from the color camera 120. The optical axis of the color camera 120 may be configured to intersect at a distance of about 0.00 cm, or within a range of about 30 cm to about 100 cm.

[0018] The image 102 is then analyzed in S111 to detect surface or shape characteristics corresponding to the surgical fabric 106b in the first infrared image 102. If the presence of the surgical fabric 106b is confirmed, the frame counter 114 is indexed in S112. At a second or subsequent time point following the first, a second or subsequent infrared image 102 of the field of view 108 is acquired and the frame counter 114 is again indexed in response to detecting the surface or shape characteristics of the surgical fabric 106b in the second or subsequent infrared image 102. This detection is performed repeatedly until a threshold value 116 is reached or the frame counter is cleared in S130 in response to failing to detect the surface or shape characteristics of the surgical fabric in the second or subsequent infrared image. When the frame counter contains a value equal to the threshold value in S113, a color image 110 of the field of view 108 may be acquired by the color camera 120. In variations where a color image is acquired in the infrared or depth image 102, the last color image 110 therefrom acquired during block S110 or block S111 may be used. Regions of the color image 110 corresponding to the surgical fabric 106b are identified, and these regions are then further analyzed to identify portions containing blood. These portions are then converted to an estimate of the amount of blood components in block S140. To this end, the system clears the frame counter in S130.

[0019] A further variation of the method is also shown in FIG. 1. For example, at a first time point, a first image 102 of the field of view 100 is acquired by the optical system 104 in S110, and a rendered form of the first image 102 can be provided to the display 118 in S120. This displayed image 102 can correspond to a depth image or a color image. In response to detecting a characteristic of a first surface or shape of the surgical fabric 106b in the first image 102, the frame counter 114 is indexed in S112 and highlights on the display 118, in block S122, an area of ​​the first image 102 that corresponds to the first surface or shape of the surgical fabric 106b. This variation of the method can also include acquiring a second or subsequent image 102 of the field of view 108 at a second or subsequent time point following the first time point, and rendering the form of the second or subsequent image 102 on the display 118. Then, in response to detecting the second surface or shape characteristic of the surgical fabric 106b in the second image 102, the frame counter 114 is indexed in S112 and an area of ​​the second image 102 corresponding to the second surface or shape is highlighted on the display 118 in block S122 based on the value of the frame counter 114. If there is no detection of the second surface or shape characteristic of the surgical fabric 106b in the second or subsequent image 102, the frame counter 114 is cleared in S130 or in response to the frame counter 114 containing a value equal to the threshold value 116, the fabric counter 112 is indexed in block S150 and the frame counter 114 is cleared. In procedures or practices utilizing different surgical fabrics, multiple surgical fabric counters may be provided and the counters for the corresponding surgical fabrics 106 may be indexed in S150.

[0020] 3. System 1, computing device 100 includes an imaging system 104 configured to define a field of view 108 and acquire depth and color data from the field of view 108. In particular, computing device 100 may include a depth imaging camera or system 122 that outputs depth information or map 102 of the field of view 108, and a color imaging camera or system 120 that outputs a color image 110 of the field of view 108.

[0021] As shown in FIG. 3, in one embodiment, the depth imaging system 122 includes an infrared ("IR") light projector 302 that defines a projection field and projects infrared light in a pattern of dots into the projection field. The computer device 100 includes a housing 300 having a structure of a casing 102, an IR camera 304 configured to define an infrared field of view substantially the same as the projected field of view and to acquire a sequence of infrared images (or "infrared frames") of the infrared field of view, and an internal processor 306 configured to convert the distorted dot pattern detected by the IR camera 304 recorded in each infrared image into a depth map 102, such as based on the density or displacement of dots across each infrared image. The IR projector may include an IR laser or diode. U.S. Patent Application No. 8,374,397 is an example of this IR depth technology, which is incorporated by reference in its entirety. Thus, during operation of the computer device 100, the depth imaging system can output a sequence of depth maps 102, each of which includes an indication of the distance between the IR camera 122 and a surface in the infrared field of view at a corresponding time. Of course, in addition to IR-based cameras, other depth cameras may be used, including, but not limited to, other types of similarly configured cameras, stereo cameras in which stereo triangulation may be used on pairs of stereo images to determine the distance and focal depth of the camera, and other known types of depth cameras. In some variations, the depth imaging system 122 may include a color camera system 308, which may be used in place of a color camera system internal to the computing device.

[0022] The color imaging system may include a color camera 120, such as a CCD or CMOS camera supporting red, green, and blue color channels. The color camera 120 may also define a color field of view that is substantially similar to (i.e., substantially overlapping with) the infrared field of view (and thus the projected field of view) of the IR camera 122. As described below, the computing device 100 may trigger or activate the depth imaging system 122 and the color imaging system 120 to generate the depth map 102 and acquire the color image 110, respectively, during each sampling period while performing the method, such as at a rate of 30 Hz. In other embodiments, the sampling rate of the camera(s) may range from about 0.1 Hz to about 1 kHz, or from about 60 Hz to about 240 kHz. The computing device 100 may thus acquire one depth map 102 and one color image 110 ("frame pair") for each sampling period, and process both the depth and color data for each sampling period to detect the presence of a surgical sponge or other surgical fabric 106 within the field of view of the computing device 100 at the corresponding time. Alternatively, the computing device 100 may sample the depth imaging system and process the depth map data to detect the presence of a surgical sponge or other surgical fabric within the field of view 108 of the computing device 100. The computing system 100 may also acquire a color image 110 through the color imaging system when a value in a frame counter 114 reaches a threshold value 116, as described below. In still other embodiments, the ratio of depth image 102 to color image 110 may be fixed or predetermined, and may be from about 3:2 to about 100:1, or from about 2:1 to about 50:1, or from about 30:1 to about 50:1.

[0023] The computing device 100 may also include a display 118 adjacent to the IR camera 122 and the color camera 120. The display 118 may define a viewing area approximately perpendicular to the field of view of the IR and color cameras, such that a user (e.g., a nurse, technician, anesthesiologist) holding a surgical sponge may stand in front of the IR camera 122 and the color camera 120 while viewing the contents of the display 118. For example, as described below, the display 118 may render in near real time any one or more of: a) an indicator that a sponge has been detected; b) an indicator that an image of the sponge has been recorded; c) a sponge counter value; and d) a prompt to handle the sponge before an image is captured. In one embodiment, the computing device includes a tablet computer with a forward-facing color camera and an integrated display, as well as a display that displays a front-facing color camera and ... , a tablet computer including an IR projector and an IR camera mounted on the tablet and facing outward from the tablet, defining a projected field of view and an infrared field of view that are substantially similar to the color field of view of the color camera, but the computing device may include any other optical, imaging, and / or processing components incorporated or connected thereto, configured in any other suitable manner.

[0024] 4. Exemplary User Experiences In one embodiment, the computing device 100 performs the method illustrated in FIG. 1 in the form of an in situ blood loss monitoring application installed in an operating room and utilized during an operation, surgical procedure, or procedure. In this embodiment, the computing device 100 continuously acquires infrared images 102 and / or color images 110 of its field of view 108 and discards these images if the surgical sponge 106a is not detected in the image or frame. The computing device 100 may render these frames or images on the display 118 in near real time. The rendered image is typically a color image 110, but may also be a depth image 102 or a combination of color and depth images. In operation, a user (e.g., a nurse, technician, anesthesiologist) may hold the surgical sponge 106a in front of the imaging system 104. Now, with the surgical sponge 106a within the field of view 108 of the imaging system 104, the imaging system 104 acquires a first depth map 102 and / or a first color image 110 (a "first frame pair" or, if more than two images or maps are grouped, a "frame set") for a first sampling period and then tracks the presence of the surgical sponge 106a within the field of view 108 of the computing device 100, S110. The computing device 100 then performs the machine vision techniques described herein to correlate areas of the first depth map 102 (and corresponding areas of the first color image 110) with the surgical sponge 106a and indexes a frame counter 114 from "0" to "1", S112. In response to detecting the image 106b of the surgical sponge within the first depth map 102 (or first color image 110), the computing device 100 renders the first infrared image 102 (or first color image 110) on the display 118 with a colored (e.g., green, red, yellow, or other distinct color) line or shape overlay 400 of a first width along the perimeter of the detected surgical sponge and / or a colored overlay area of ​​a first opaque portion over the detected area of ​​the surgical sponge.

[0025] In successive sampling periods, the imaging system 104 acquires a second depth map 102 and / or a second color image 110 (a "second frame pair" or "second frame set"), and the computing device 100 again performs machine vision techniques to correlate areas of the second depth map 102 (and corresponding areas of the first color image 110) with the surgical sponge 106a and index the frame counter 114 from "1" to "2." In response to detecting an image 106b of the surgical sponge in the second frame pair, the computing device 100 renders the second infrared image 102 on the display with a wider colored line or shape overlay 402 of a second width or line length - larger than or different than the first width or line length - along the perimeter of the detected surgical sponge 106b, and / or a second opaque colored overlay area - larger than or different than the first opaque - over the detected area of ​​the surgical sponge 106b.

[0026] The computing device 100 can repeat this process to acquire frame pairs during successive sampling periods, detect the surgical sponge 106b within the frame pairs, index the frame counter 114 accordingly, and refresh the display 118 with the current depth map 102 (or color image 110), the line widths 400, 402, and opacity around and / or in the area of ​​the overlay; 10 and 110 show the frame counter values, respectively. A larger overlay width or other changing indicia may be provided to dynamically demonstrate the process of verifying the presence of the surgical fabric 106b in the images 102, 110. For example, the overlay color may change from red to yellow to green. When the value on the frame counter 114 reaches a threshold value (e.g., "7"), the computing device acquires a color image through the color camera, renders a visual indicator that the color image has been acquired on the display, and resets the frame counter to "0" without additional manual input. For example, with an imaging system operating at 30 Hz and a threshold preset in which the frame counter is set to "7", the computing device may implement the methods and techniques described above to detect, verify, and acquire an image of the surgical sponge within less than 0.3 seconds of the surgical sponge entering the field of view of the computing device. In other embodiments, the threshold value may be set in a range of about 5 to 100, or about 5 to 50, or about 4 to about 30, or about 20 to about 40. The computing device may be further configured to perform an analysis regarding movement of the surgical fabric 106a from the field of view 108, as shown at S500 and in FIG.

[0027] Computing device 100 may be configured to implement methods and techniques described in U.S. patent application Ser. No. 13 / 544,664 to convert color values ​​of pixels in the image into an estimate of the amount of blood constituents in the imaged surgical sponge in block S140. In response to identifying surgical sponge 106b in the field of view 108 of computing device 100 and acquiring a color image 110 of surgical sponge 106a, computing device 100 may also index a sponge or surgical fabric counter 112 in block S150 to maintain a total count of surgical sponges that pass computing device 100 during the surgery or procedure.

[0028] 5. Classifier Block S110 of the method depicts acquiring a first image 102 of a field of view 108 at a first time point. Generally, during each sampling period, the computing device 100 operates to acquire an infrared image 102 and / or a color image 110 through the imaging system 104 in S110. As described above, the computing device 100 may convert the infrared image 102 into or extract a depth map therefrom of the field of view. The computing device 100 may repeat this process over time during its operation, such as at a sampling rate (or "frame rate") of 30 Hz, or other sampling rates described herein. However, the computing device may acquire depth and color data of its field of view in any other manner and through any other suitable type of sensor, and at any other sampling rate, in block S110.

[0029] After acquiring the image(s) 102 and / or 110, the computing device 100 analyzes the image(s) 102 and / or 110 to determine the presence of one or more image characteristics therein, S111, or otherwise identifies the image(s) 102 and / or 110 acquired during the current sampling period as including or not including the sponge or surgical fabric 106b. Based on this determination, the computing device 100 may then index a frame counter, S112, or clear the frame counter, S130, respectively. Specifically, the computing device 100 applies one or more classifiers acquired during the sampling period to the depth map 102 and / or color image 110, as shown in FIG. 1 and FIG. 2, to determine the presence or absence of the surgical sponge or fabric 106a within the field of view 108 of the computing device 100 during the sampling period, and manipulates the frame counter 114 accordingly.

[0030] In one exemplary method S200, computing device 100 acquires IR image 102 and / or color image 110, as in S110 of FIG. 1, and then, optionally, converts image 102 and / or image 110 to a depth map in S210. In some embodiments, no conversion may be necessary since the IR "image" output from IR imaging system 122 is a depth map. Alternatively, the depth map may be extracted from the alpha channel (or other channel) of the infrared image without the need for any conversion or calculation. In S220, computing device 100 applies a perimeter classifier to the depth map acquired during the sampling period to correlate areas in the depth map to potential surgical sponges. In one embodiment, to compensate for surgical sponges that may not be held in a substantially perpendicular position relative to the infrared camera, the computing device can scan the depth map - along both the X and Y axes of the depth map - to identify depth changes along the Z axis in the depth map that exceed a threshold of 10% or more, 20% or more, 30% or more, or 40% or more over a line of 10 adjacent pixels. In this embodiment, the computing device 100 can correlate each of the short adjacent lines of pixels over which the depth value exceeds a peripheral condition by 10%, and then cluster adjacent similar peripheral conditions in the depth map around a distinct surface in the field of view of the computing device. Once these distinct surfaces are thus detected, the computing device can perform template matching, surface matching, or other machine vision techniques to identify distinct, approximately linear surfaces in the depth map of possible surgical sponges. The computing device may thus scan the depth map for an area bounded by a generally straight perimeter characterized by an abrupt increase in depth (from the imaging system) from the interior of the area to the exterior of the area. In this embodiment, the computing device may implement a geometric shape identifier that compensates for the looseness (e.g., non-rigidity) of the surgical sponge, including the curvature of one or more sides of a straight sponge when held vertically in front of the imaging system.The slack of the surgical sponge may vary with the volume of fluid within the surgical sponge, the tension exerted on the surgical sponge by the user, and the like.

[0031] In the above embodiment, the computing device may also reject all detected surfaces that do not match predefined points or regions in the depth map, such as all surfaces that do not encompass a single pixel or a small cluster of pixels in the lateral and longitudinal centers of the depth image. The computing device may thus apply a 2D positional classifier to the depth map to reject surfaces that are not substantially centered within the field of view of the imaging system. However, the computing device may implement any other geometric classifier in any other manner to detect the approximately linear perimeter of nearby surfaces in the depth map and correlate the surfaces with possible surgical sponges.

[0032] In other embodiments, the perimeter detection may be configured to identify more abrupt changes or decreases in depth, such as, for example, greater than 15%, 20%, 30%, 40%, 50%, 60%, or 70%. While the perimeter detection may be performed for all four edges of a square or rectangular surgical fabric, in other embodiments, the computing device 100 may perform a scan of the edge only on the X-axis or Y-axis for changes in depth, but not on the other axis, and thus estimate the edge along the other axis by connecting each end point of the edge to the estimated edge. Thus, in some variations, only the top and bottom edges may be detected, and the lateral edges may be estimated from the ends of the top and bottom edges. Alternatively, the lateral edges may be detected, and the top and bottom edges estimated.

[0033] In another embodiment, the computing device 100 may optionally apply a flatness discriminator to selected surfaces of the depth map at S230 - as described above - to discriminate between possible In this embodiment, the computing device 100 may select a set of pixels in the depth map (e.g., 3, 4, 5, 6, or 10), map a virtual plane to the set of pixels, and then estimate a histogram of the vertical distance from the virtual plane to other pixels in the depth map that correspond to the selected surface. Thus, if the histogram indicates that the selected surface remains within the virtual plane's distance threshold across its height and width (optionally with outliers removed to reduce distortion), the computing device 100 may determine or analyze whether the selected surface is sufficiently flat to represent a surgical sponge held by its upper corners. However, if the histogram indicates that an area of ​​the selected surface violates the distance threshold from the virtual plane, the computing device 100 may discard the selected surface as not corresponding to a surgical sponge held by its upper corners. In this embodiment, the computing device can implement distance thresholds to compensate for folds, wrinkles, and / or creases in the surgical sponge, such as distance thresholds of ±0.1 inches, ±0.25 inches, ±0.5 inches, ±0.75 inches, or ±1 inch. In this embodiment, if the selected surface is rejected under the flatness classifier, the computing device can generate textual and / or diagrammatic prompts to stretch the surgical sponge to pass the flatness classifier. The computing device can render this prompt on the display in near real time.

[0034] In yet another embodiment, the computing device applies a perpendicularity classifier to the depth map in S240 to ensure that the selected surface is sufficiently perpendicular to the field of view of the imaging system to be suitable for imaging and processing in block S140 of FIG. 1. For example, the computing device 100 estimates a virtual line perpendicular to the above-mentioned virtual plane, rejects the selected surface if the virtual line forms a maximum angle with the center of the field of view of the imaging system that exceeds a threshold angle, and passes the selected surface if the maximum angle formed between the virtual line and the field of view of the imaging system remains within the threshold angle. In an alternative method S600 shown in FIG. 6, after application of the perpendicularity classifier in S240, instead of rejecting images that are not sufficiently perpendicular to the computing device 100 or the optical system 104, the computing device 100 may estimate or undistort the depth map in S602. To undistort the depth map, the computing device 100 may detect the four corners and / or four edges of the surgical fabric 106b in the image 102 and apply a 3D rotation or translation matrix to smooth the corners or edges to the same plane in the Z axis.

[0035] After unwarping, the computing device 100 may again apply the flatness classifier to the unwarped surgical fabric 106b and then reject the selected surface if the maximum change in distance values ​​for pixels across the selected surface in the warped depth image exceeds a threshold distance due to folds, wrinkles, and / or creases in the surgical sponge, or pass the selected surface to continue further classification and / or processing using the unwarped depth image if the maximum change in distance values ​​for pixels across the selected surface in the warped depth image is at a threshold distance. The threshold distance may be in the range of about -1 cm to about +1 cm, about -2 cm to about +2 cm, or about -3 cm to about +3 cm, or about -4 cm to about +4 cm. If the warped image does not pass the flatness classifier at S604 after warping at S602, the computer system 100 may reset the frame counter at S130 with or without providing a notice or message on the display 118 of the computing device 100 to encourage repositioning the surgical fabric 106a within the field of view 108 to a relatively more vertical orientation. Alternatively, if the selected surface is rejected under the verticality classifier: The computing device can generate textual and / or diagrammatic prompts to adjust the surgical sponge to pass the perpendicularity identifier for an additional certain number of frames, and the computing device can render the prompts on the display in near real-time, where the number of frames can be in the range of about 15 frames to 120 frames, or about 30 frames to about 60 frames, for example.

[0036] 2, the computing device may optionally apply a distance discriminator to the depth map at S250 to ensure that the selected surface is within an appropriate depth of the imaging system 104. For example, if the computing device 100 determines that the selected surface is sufficiently flat at S230 and sufficiently perpendicular to the field of view of the imaging system at S240, respectively, the computing device 100 may reject the selected surface if more than a minimum number (e.g., 1%) of pixels in the depth image 102 showing the selected surface of the surgical fabric 106b are outside a predetermined depth range (e.g., 0.5 meters to 1.25 meters, or 30 cm to about 100 cm, or about 40 cm to about 75 cm) from the imaging system 104. In this embodiment, if the selected surface is rejected under the distance identifier in S250, the computing device 100 may optionally generate a textual and / or diagrammatic prompt on the display 118 to move the surgical sponge 106a closer or farther away from the optical system 104 of the computing device 100 with the further intent of passing the distance identifier in S250. The computing device 100 may render this prompt on the display 118 in near real-time.

[0037] In the above-described embodiment, the computing device 100 may also convert the pixel length and pixel width of the selected surface in the depth image to actual length and actual distance based on the pixel distance value in the depth image. The computing device may then reject the selected surface if its actual length or actual width is outside the preset length range or preset width range, respectively, otherwise confirm that such a surgical fabric is being used. Preset dimensions for the surgical fabric may include 2 inches by 2 inches, 3 inches by 3 inches, 4 inches by 4 inches, 4 inches by 8 inches, 4 inches by 12 inches, 4 inches by 16 inches, 4 inches by 18 inches, 12 inches by 12 inches, 16 inches by 16 inches, 16 inches by 24 inches, 18 inches by 30 inches, 15 inches by 24 inches, 16 inches by 23 inches, 12 inches by 13 inches, 13 inches by 18 inches, 24 inches by 24 inches, 36 inches by 8 inches, 36 inches by 36 inches, etc. For each type of sponge, a user may input the type and / or size of the sponge at the start of a surgical procedure or procedure, and the computing device 100 may be configured to detect the dimensions of the surgical fabric, for example, within 0.5 inch or 0.75 inch along the width or length for dimensions less than 5 inches, 8 inches, or 10 inches, and within 0.5 inch, 1 inch, 1.5 inch, or 2 inches for dimensions greater than 8 inches, 10 inches, 12 inches, or 15 inches.

[0038] In another embodiment, the computing device applies a texture identifier to the depth map in S260 to determine whether the actual texture of the selected surface depicted in the depth map sufficiently corresponds to a surgical fabric. In one embodiment, the computing device 100 may be configured to project or define a pixel cluster (e.g., a 25 pixel by 25 pixel cluster) in the depth map - corresponding to the selected surface - onto a virtual plane and then apply machine vision texture analysis methods and techniques to quantify the texture of the smoothed region of the depth map corresponding to the selected surface. Thus, the quantified value of the texture of the selected surface may be within a single range of values, such as a dry surgical sponge, a wet surgical sponge, or a wet surgical sponge. If the range of distance values ​​indicative of a surgical sponge and saturated surgical sponge is within the range, the computing device may accept the selected surface. Otherwise, the computing device may reject the selected surface. In other examples, a fast Fourier transform or wavelet transform may be applied to the texture 106b in the image(s) 102, 110 to evaluate the frequency domain of the texture in one or more image channels, such as the RGB or depth channels. In some variations, if the detected texture characteristics match a surgical textile in the library of the computing device 100 but do not match the surgical textile(s) selected by the user at the start of the surgical operation or procedure, the computing device 100 may prompt the user to confirm that a new or different surgical textile is indicated as being used and may prompt the initialization or activation of a different surgical textile counter for that surgical textile.

[0039] In another embodiment, the computing device 100 applies a color classifier to the color image 110 at S270 to verify that the selected surface includes one or more colors indicative of a dry or bloody surgical sponge. In one example, the computing device 100 accepts the selected surface if at least a threshold percentage (e.g., 70%) of pixels in the region of the color image corresponding to the selected surface include colors within a red value range (e.g., [ff 00 00] to [ff 40 00]) or a near-white value range (e.g., around [ff ff ff]), as shown in FIG. 1. Otherwise, the computing device may reject the selected surface.

[0040] However, the computing device 100 may apply any other classifier parameters to one or more selected surfaces in the frame pairs and / or the depth map or color image of each frame pair. In operation, the computing device 100 may thus apply one or more classifier parameters to the current frame pair, and may confirm the selected surface in the frame pair as an indication of a surgical sponge if all applied classifier parameters pass the selected surface or pass the selected surface with sufficient certainty. Alternatively, the computing device 100 may confirm the selected surface in the frame pair as an indication of a surgical sponge if at least a threshold number of applied classifier parameters pass the selected surface or the combined results of all applied classifier parameters reach at least a threshold certainty. The computing device may thus index the frame counter in block S112, such as from "0" to "1" or from "4" to "5". In some further variations, a user may adjust the sensitivity and / or specificity of the computing device 100 via a user interface by adjusting the number or weights of classifiers that are applied or required to pass an image pair or set of images.

[0041] Otherwise, if the aforementioned conditions are not met, the computing device 100 may discard the frame pair as not including an area indicative of the surgical sponge 106b and reset the frame counter 114 to "0" at S130. The computing device 100 may also discard the frame pair and reset the frame counter to "0" if the surface selected in the current frame pair is significantly different in location, size, texture, depth, etc. from the surface selected in the previous frame pair. For example, even if a surface indicative of a surgical sponge is detected in the current frame pair, the computing device may reject the current frame pair and reset the frame counter if the center of the selected surface in the current frame pair is offset in any direction from the center of the selected surface in the previous frame pair by more than 0.5 inches. The computing device may thus reject surgical sponges and other surfaces that move through the field of view of the imaging system and remain approximately static within the field of view of the imaging system. It is only possible to confirm the sponge sample of any image that has been held.

[0042] Although the identifiers S230, S240, S250, S260, and S270 are shown performed in a particular order in S200, those skilled in the art will understand that a different order of these and other identifiers may be used, or that many identifiers may be performed in parallel, such as in a multi-core and / or multi-threaded processor of computing device 100.

[0043] 6. Check the sponge Then, in response to the frame counter 114 containing a value equal to the threshold value 116, the computing device records a color image 110 of the field of view 108, either before or after further processing the image 110 to assess blood content, and clears the frame counter 114 in block S130. Generally, for each frame pair in the sequence of frame pairs, the computing device 100 analyzes the frame pair independently of the previous frame pair, as shown in Figures 1 and 2, to either identify the surgical sponge or fabric 106b in the frame pair or confirm that the surgical sponge or fabric 106b is not present in the frame pair, and indexes or clears the frame counter accordingly. When the frame counter 114 contains a value equal to the threshold 116, a total number of consecutive frame pairs equal to the threshold has been determined to include the surgical sponge or fabric 106b, and the computing device 100 may therefore determine that the surgical sponge or fabric 106a is present within the field of view 108 of the imaging system 104, and that when the frame counter 114 contains a value equal to the threshold 116, that area within the field of view is known to an appropriate degree of certainty.

[0044] The computing device may implement a static, pre-set threshold, such as one of "7", "10", or "15", or may otherwise be in the range of about 5 to 100, or about 10 to 70, or about 30 to 50. Alternatively, the computing system may implement custom threshold settings by the user. For example, the user may reduce the threshold to reduce the time of reviewing and processing images of the surgical sponge, and the user may increase the threshold to reduce false positive detection of the surgical sponge in the field of view of the imaging system. In yet another variation, the computing device may include a discontinuity counter. The discontinuity counter is for each instance during a surgical procedure or procedure where a certain minimum number of consecutive frames is met, but the threshold number of frames is not met. Depending on the value of the discontinuity counter over time, the computing device may adjust the threshold to increase or decrease to change the sensitivity of the detection method. This may accommodate the position of the light or other environmental conditions that may affect the accuracy of the method S100.

[0045] When the computing device thus confirms the presence of the surgical sponge 106a within its field of view 108, the computing device 100 may record the last depth map 102 and / or color image 110 acquired by the imaging system, or may trigger the imaging system to acquire a new depth map 102 and / or color image 110. The computing device 100 may then process this image pair as described below.

[0046] In some variations, to reduce the risk of inadvertently re-counting or re-analyzing the surgical fabric 106a that was just presented to the optical system 104 before resetting the frame counter 114 in S130 or before resuming scanning of a new surgical fabric 106a in S110, the computing device may, as shown in S500 of FIG. 5, wait until the surgical fabric 106a is removed from the field of view 108 before acquiring an image to identify the next surgical fabric 106a. After determining the assessed blood content at S140 and / or incrementing the textile counter 112 at S150, the computing device 100 may acquire a new depth map 102 and / or color image 110 at S510 and determine at S520 whether the surgical textile 106a has been removed from the field of view 108 or is otherwise not detected in the image 102 or image 110. If the surgical textile 106b is still present in the image 102 or image 110, either the user has not yet removed the previous surgical textile or has not yet removed the surgical textile for a sufficient number of images or frames, the computing device resets the "no textile" counter to zero.

[0047] If the surgical textile 106b is not identified in the frame or image, then at S540 a "no textile" counter is incremented and the counter is then checked against a "no textile" threshold to determine whether a sufficient number of consecutive "no textile" frames or images have been identified. If the threshold is met, the "no textile" counter may be reset and the computing device may then reset the frame counter at S130. In some embodiments, the "no textile" threshold may be in the range of about 5 to 100, or about 5 to 50, or about 4 to about 30, or about 20 to about 40. The "no textile" threshold at S500 may be the same as or different from threshold 116 at S100 of FIG. 1.

[0048] 7. Dynamic Classifier In one embodiment, the computer system applies the same order set of identifier parameters to each frame pair obtained through its operation. Alternatively, the computer system can apply a set of identifier parameters to a frame pair based on the current value in the frame counter. For example, the computer device 100 may apply a perimeter identifier and a lateral / longitudinal position identifier to the frame pair if the frame counter includes a value of "0", and then apply a perimeter identifier and a verticality identifier to the frame pair if the frame counter includes a value of "1". The perimeter identifier and the distance identifier may be applied to the frame pair if the frame counter includes a value of "2" or more, the perimeter identifier and the texture identifier may be applied to the frame pair if the frame counter includes a value of "6", and the perimeter identifier and the color identifier may be applied to the frame pair if the frame counter includes a value of "7". However, the computer device can apply any other single or combination of identifier parameters to the frame pair to identify the surgical sponge in the frame pair or to discard the frame pair. In some other variations, the perimeter and / or position classifiers may be applied to all frames, while the remaining classifiers are applied to different frames, in some variations the different frames for the remaining classifiers are predefined, while in other variations the different frames may depend on processing time or completion of a previous classifier.

[0049] 8. User Interface Block S120 of the method illustrates rendering a morphology of the first image on a display, and block S122 of the method illustrates highlighting an area of ​​the first image on the display that corresponds to the first surface. Generally, the computing device renders a morphology of a depth map (or color image) from a current frame pair (i.e., the frame pair most recently acquired by the imaging system) on the display in block S120, and renders a virtual overlay over the depth map in block S122 to visually indicate that a surface, possibly corresponding to a surgical sponge, has been detected in the frame pair, as shown in FIG.

[0050] In one embodiment, the computing device dewarps the depth map in real time. , and renders these warped depth maps on a display. As each depth map is processed, as described above, the computing device can identify, in near real-time, regions (e.g., demarcated areas) with the depth map that may correspond to a surgical sponge, and then render a visual indicator of this potential surgical sponge over the corresponding depth map on the display. For example, the computing device can overlay a colored outline over the perimeter of the potential surgical sponge in the rendered depth map on the display. In this example, the computing device can set a line width of the outline proportional to the current value on the frame counter. Similarly, the computing device can set the color of the outline based on the current value of the frame counter, such as red if the current value of the frame counter is "0", orange if the current value of the frame counter is "2", yellow if the current value of the frame counter is "4", and green if the current value of the frame counter is "6". In another example, the computing device can overlay an area of ​​semi-transparent color overlapping over a region of the depth map - rendered on the display - that corresponds to a potential surgical sponge. In this example, the computing device may set the opacity of an area of ​​the overlay that corresponds to the current value of the frame counter, such as 12% opacity if the current value of the frame counter is "0," 24% opacity if the current value of the frame counter is "1," 36% opacity if the current value of the frame counter is "3," ... and 84% opacity if the current value of the frame counter is "7." Similarly, the computing device may set the color of an area of ​​the overlay based on the current value on the frame counter, such as those described above.

[0051] Thus, in blocks S120 and S122, the computing device can provide real-time visual feedback regarding detection of the surgical sponge within the field of view of the imaging system. The computing device can additionally or alternatively implement any of the methods and techniques described above to render color images and / or overlays on a display in near real-time to provide such feedback to a user.

[0052] 9. Mask Once the presence of surgical sponge or fabric 106b is thus confirmed in images 102 and / or 110, computing device 100 applies an image mask or places the perimeter of the selected and confirmed area in the depth map onto the corresponding color image in the same frame pair to select a region of the color image for processing in block S140 and / or block S150.

[0053] In some variations, the method S700 of forming and applying a mask to a color image may be optionally used to facilitate computational speed, efficiency, and / or accuracy. For example, in FIG. 7, the depth imaging system 122 may be used to obtain a depth image in S710 along with an associated image from the color imaging system 120. Depth channel information is then extracted or obtained in S720, and if necessary, obtained or calculated from the infrared image. In S730, a perimeter or edge identifier, which may be similar or the same as the perimeter or edge identifier in S220, may be used to identify the perimeter of the surgical fabric. The image may then be undistorted in S740, if necessary, or converted to a perspective view of the coordinate system of the color image, as described elsewhere herein. The color image may also be processed for color and / or contrast equalization in S750. For example, generating histogram distributions or color curves for the RGB channels and equalizing the color image to a different histogram or curve. A corresponding image of the surgical fabric is thus applied to the color image as a mask in S760. ,Remove background color pixels from the analysis that are unnecessary.

[0054] In another embodiment of the mask method S800, shown in FIG. 8, depth and color images are obtained from an infrared system and a color system in S810. In S810, a first mask is generated from the depth channel of the infrared image by treating the maximum foreground pixel value as positive and all other pixels as negative. This may be a result of the surgical fabric being presented to the IR camera at or below the minimum depth range of the IR camera. In S830 and S840, the first mask undergoes erosion and dilation. These are threshold-based morphological image processing operations that may be used to reduce noise and false negative pixels. For example, to perform erosion, each pixel in the mask is replaced with the minimum value of the surrounding pixels, which may be configured or defined as, for example, a 3×3, 5×5, 7×7, 3×5, or 5×3 pixel zone or window. This erosion process may be applied to the first mask as a single pass or through multiple passes to generate a second mask, and different pixel windows may be used on different passes. This erosion process may be used to reduce background noise or other false positive pixels in the first mask. A dilation process may then be applied to the first mask to generate a third mask, where each pixel in the first mask is replaced with the maximum value in a group of surrounding pixels, for example, a 3×3, 5×5, 7×7, 3×5, or 5×3 pixel zone or window. The grouping may be the same as or different from that used for the erosion process, and may also be performed in a single pass or multiple passes, again with the same or different grouping for each pass. This dilation process may be used to potentially return certain false negative pixels. Using these three masks, a fourth mask is optionally generated at S850, with two bits per pixel of the mask, where all pixels from the second mask are flagged as "11" or otherwise flagged as strictly foreground pixels. Pixels that are not in the third mask are flagged as "00" or otherwise flagged as strictly background pixels.Pixels that are in the first mask but not in the second mask are labeled "10" as potential foreground pixels, and pixels that are in the third mask but not in the first mask are labeled "01", i.e., potential background pixels.

[0055] Depending on the desired sensitivity and / or specificity required, the first, second, or fourth mask may be used to identify corresponding pixels in the color image for further processing. In a mask, such as the first mask or the fourth mask, further processing, such as perimeter or edge detection, may be applied to further refine the mask prior to application to the color image.

[0056] 10. Hand Removal The computing device may also identify areas of the color image that correspond to the user's hand - holding the surgical sponge - and discard these areas from processing in blocks S140 and S150. For example, the computing device may scan a selected region of the depth map in a frame pair identified as showing a surgical sponge, correlate areas in the depth image that are close to corners of the selected region and indicate a distance from the imaging system reduced by the hand, and then designate the corresponding area of ​​the color image for removal from processing in block S140. The computing device may also remove this area from the outline of the surgical sponge shown on the display. The computing device may apply the above methods and techniques to the frame pair designated for processing only after the surgical sponge is identified in the field of view of the imaging system. Alternatively, the computing device may apply the above methods and techniques to each frame pair and each depth or color image rendered on the display.

[0057] 11. Blood component volume and sponge counter In block S140, once a color image has been acquired and recorded that is confirmed to include an area indicative of a sufficiently flat surgical sponge at the proper distance and perpendicularity of the field of view of the imaging system, the computing device can implement methods and techniques described in U.S. patent application Ser. No. 13 / 544,664 to convert color values ​​contained in pixels in the color image into an estimate of mass, volume, or other quantitative measurement of the amount of whole blood, red blood cells, hemoglobin, and / or other blood constituents in the actual surgical sponge depicted in the color image, the contents of which are incorporated herein by reference in their entirety.

[0058] In some variations, prior to converting pixel color values ​​into estimates of blood constituent values, the image (with or without any masks described herein) may also undergo ambient light compensation as described in U.S. Patent Application No. 15 / 154,917, which is incorporated herein by reference in its entirety.

[0059] In block S150, the computing device may also implement the methods and techniques described in U.S. Patent Application Serial No. 13 / 544,664 to index a counter indicating the total number of surgical sponges identified and counted during operation in which the computing device is operating. In this variation, the computing device may render a value contained in the sponge counter on a display as a surgical sponge is identified within the field of view of the imaging system, and update the rendered value substantially in real time. For example, the computing device may make the updated sponge counter value appear on the display as the surgical sponge is identified and counted. Similarly, when a possible surgical sponge is detected within a frame pair, the computing device may overlay the value of the next sponge counter (i.e., the current sponge count + 1) over the rendered depth (or color) image on the display, and the computing device may increase the opacity of the overlaid sponge counter value as the value in the frame counter increases for the next frame pair.

[0060] In another embodiment, the color image is further segmented or masked based on the areas of the image containing blood after segmentation of the surgical fabric to exclude areas of the surgical sponge that do not contain blood. This can be done by using an algorithm based on the following rules:

number

number

[0061] 12. Audio sponge counting In one variation, the computing device may acquire and store audio data along with image and blood constituent quantity data regarding surgical sponges detected and imaged during operation. In particular, the computing device may record a user (e.g., two users, such as a nurse, an anesthesiologist, and a technician) continually indicating an audio sponge count.

[0062] In one embodiment, if a possible sponge is detected in a frame pair and indexed as 1 in a frame counter, the computing device may begin recording audio by sampling a microphone connected to or integrated into the computing device. If the frame counter value then reaches a threshold value and thus the surgical sponge is identified, the computing device may continue recording audio after 5 seconds until the sponge is identified or a new sponge is detected within the field of view of the imaging system, whichever occurs first. The computing device may then store and link this audio recording with a corresponding color image of the surgical sponge. For example, the computing device may record a color image of the surgical sponge, convert the color image to an estimate of the amount of blood components (e.g., hemoglobin volume) in the surgical sponge, and store this color image and the estimate of the amount of blood components along with the time the image was acquired, the value of the sponge counter for the surgical sponge, and a recording of the audio of two users reading and confirming the sponge count value.

[0063] In another embodiment, the computing device may record a single audio track throughout the procedure, then splice the audio track into multiple separate audio segments, match each audio segment to a recorded color image and an assessment of the amount of blood components based on the time of the audio segment and the time the color image was acquired, and then store each audio segment with a tag or other pointer to its corresponding color image or associated data.

[0064] 12. Rejection of duplicate sponges – RFID In one variation, the computing device further includes a radio frequency identification ("RFID") reader and an RFID antenna. In this variation, the computing device may index an image counter with "1" (or some other value below a threshold) to indicate a possible surgical sponge in the field of view of the imaging system, and the computing device may emit an excitation signal through the RFID antenna and then sample the RFID reader. If a sponge in the field of view of the imaging system includes an RFID tag, the RFID tag may emit a data packet back to the RFID reader. The computing device may apply receipt of this data packet from the surgical sponge to identify the sponge, and then obtain and process a color image of the sponge as soon as flatness, distance, location, and / or other requirements are met, as described above, and before the image counter reaches a threshold. The computing device may additionally or alternatively implement supervised machine learning techniques to train an optical sponge detection model based on this RFID data packet received from the surgical sponge and the depth map and / or color image of the surgical sponge. Alternatively, the computing device may emit an excitation signal and read the incoming data packet from the proximal sponge only when the frame counter reaches a threshold and the surgical sponge is thus optically identified.

[0065] In the above-described embodiment, the computing device may compare the RFID data packets received from the current surgical sponge with previously received data packets from other sponges during the operating period to detect duplicate scans of the same surgical sponge. Specifically, if the RFID data packet received from the surgical sponge is identical to a previously received RFID data packet during the same operating period, the computing device may detect a duplicate. The surgical sponge may then be rejected as such, the corresponding frame pair discarded, and the assessment of the amount of blood constituent associated with the surgical sponge discarded.

[0066] Further, in this variation, to prevent detection of other proximate RFID tags, the computing device can vary the power output of the RFID antenna depending on the distance between the computing device and the sponge as assessed from the current depth map. Specifically, the computing device can convert the distance value in the area of ​​the current depth map corresponding to the possible or confirmed surgical sponge into a power level of the excitation signal that is then emitted by the RFID antenna. However, in this variation, if no data packet is received following the conversion of the excitation signal, the computing device can proceed according to each block of the method described above to confirm the surgical sponge and process the color image of the sponge. Furthermore, if no RFID data packet is received following the emission of the excitation signal, the computing device can tag the surgical sponge (e.g., the value of the sponge counter of the surgical sponge, the color image of the surgical sponge, etc.) as not RFID enabled.

[0067] In further embodiments, the computing device may identify any barcodes, QR codes, or other visible codes on the surgical textile. For example, the processor may scan the color image for the four corners of the QR code or barcode and then identify the coded region. In block S410, the processor then decodes the coded region to determine a code string. The code string may then be used to uniquely identify the detected surgical textile, avoid double counting if the same surgical textile is presented to the system again, and provide a warning to the user or surgeon.

[0068] 13. Rejection of overlapping sponges – reflective IR coating In another variation, the computing device implements machine vision techniques to detect reflective material on the surgical sponge shown in the frame pair and (uniquely) identify the sponge based on the detected reflective material. For example, the computing device may include a second IR emitter configured to illuminate the field of view of the IR camera with electromagnetic radiation that is reflected by material printed on or woven into the surgical sponge and detected by the IR camera. During the sampling period, the computing device may thus acquire and process a second infrared image to detect a standard or substantially unique pattern of reflective material within the surgical sponge in the field of view of the imaging system. The computing device may then implement the methods and techniques described above to confirm the presence of the surgical sponge in the field of view of the imaging system, uniquely identify the image based on the pattern of reflective material, reject duplicate surgical sponges, trigger earlier acquisition and processing of color images of the surgical sponge, and / or improve the optical surgical sponge detection model in accordance with supervised machine learning techniques, etc.

[0069] The systems and methods described herein may be implemented and / or performed at least in part as a machine configured to receive a computer readable medium having computer readable instructions stored thereon. The instructions may be executed by computer executable components in conjunction with an application, applet, host, server, network, website, communication server, communication interface, hardware / firmware / software elements of a user computer or mobile device, wristband, smartphone, or any suitable combination thereof. Other systems and methods of embodiments may be implemented and / or performed at least in part as a machine configured to receive a computer readable medium having computer readable instructions stored thereon. The instructions may be executed by computer executable components in conjunction with devices and networks of the types described above. The instructions may be executed by computer-executable components embedded in the computer-readable medium. The computer-readable medium may be stored on any suitable computer-readable medium, such as a RAM, a ROM, a flash memory, an EEPROM, an optical disk (CD or DVD), a hard drive, a floppy drive, or any suitable device. The computer-executable component may be a processor, although any suitable dedicated hardware device may (alternatively or additionally) execute the instructions.

[0070] In this description, terms such as component, module, device, etc. may refer to any type of logical or functional circuit, block, and / or process that may be implemented in various ways. For example, the functions of various circuits and / or blocks may be combined with each other into any other number of devices. Or, the device may comprise programming instructions transmitted via a carrier wave to a general purpose computer or processing / graphics hardware. Again, the device may be implemented as hardware logic circuits that perform the functions encompassed by the novelty herein. Finally, the device may be implemented using special purpose instructions (SIMD instructions), field programmable logic arrays, or a mixture thereof that provides the desired level of performance and cost.

[0071] Aspects of the methods and systems described herein, such as logic, may be implemented as functionally programmed into any of a variety of circuits, including field programmable gate arrays ("FPGAs"), programmable array logic ("PAL") devices, electrically programmable logic and memory devices, as well as programmable logic devices ("PLDs"), such as standard cell-based devices, and application specific integrated circuits. Some other variations for implementing the aspects include memory devices, microcontrollers with memory (such as EEPROMs), embedded microprocessors, firmware, software, and the like. Additionally, the aspects may be implemented in microprocessors with software-based circuit emulation, discrete logic (sequential and concatenative), custom devices, fuzzy (neural) logic, quantum devices, as well as hybrids of any of the above types of devices. The basic device technology may be provided in a variety of component types, for example, metal-oxide-semiconductor field effect transistor ("MOSFET") technologies such as complementary metal-oxide-semiconductor ("CMOS"), bipolar technologies such as emitter-coupled logic ("ECL"), polymer technologies (e.g., silicon-conjugated polymer and metal-conjugated polymer-metal structures), mixed analog and digital, etc.

[0072] As those skilled in the art will appreciate from the foregoing detailed description, and from the drawings and claims, changes and modifications can be made to the embodiments of the invention without departing from the scope of the invention, which is defined in the appended claims.

[0073] The foregoing description of the embodiments of the present invention is not intended to limit the invention to these embodiments, but rather to enable those skilled in the art to make and use the present invention. The variations, configurations, embodiments, exemplary embodiments, and examples described herein are optional and are not exclusive to the variations, configurations, embodiments, exemplary embodiments, and examples they describe. The invention described herein can include any and all permutations of these variations, configurations, embodiments, exemplary embodiments, and examples. In order to maintain the disclosure matters at the time of filing of the present application, the contents of claims 1 to 26 at the time of filing of the present application are added as follows. (Claim 1) a color imaging system configured to capture a color image; an infrared imaging system configured to acquire an infrared image; 1. A processor comprising: Using the infrared image to identify a shape of the surgical fabric; generating shape information from the infrared image; using said shape information to identify surgical fabric pixels within said color image; and the processor configured to extract blood information from the surgical fabric pixels. (Claim 2) The system of claim 1 , wherein the processor is further configured to transform the infrared image into a coordinate system corresponding to the color image. (Claim 3) The system of claim 1 , wherein the processor is further configured to generate a depth image from one of the infrared images. (Claim 4) The system of claim 3 , wherein the processor is further configured to identify a set of flat pixels in the depth image. (Claim 5) The system of claim 4 , wherein the processor is further configured to detect a perimeter of an image of a surgical fabric in the depth image. (Claim 6) The system of claim 5 , wherein the detection of the perimeter of the image of the surgical fabric is performed by looking for an abrupt vertical drop in depth in the depth image. (Claim 7) The system of claim 3 , wherein the processor is further configured to down-resolution process the depth image or the infrared image. (Claim 8) The system of claim 7 , wherein the processor is further configured to generate a mask image from the reduced resolution depth image. (Claim 9) The system of claim 8 , wherein the processor is further configured to perform high resolution processing of the mask image. (Claim 10) The system of claim 1 , wherein the processor is further configured to perform filling of missing infrared pixel regions in the infrared image. (Claim 11) The system of claim 5 , wherein the processor is further configured to generate a mask image using the detected perimeter. (Claim 12) The system of claim 11 , wherein the processor is further configured to transform the mask image to a coordinate system of the color image. (Claim 13) The system of claim 11 , wherein the detected perimeter is a top and bottom edge of an image of a surgical fabric. (Claim 14) The system of claim 11 , wherein the detected perimeter comprises opposing edges of an image of a surgical fabric. (Claim 15) The system of claim 11 , wherein the detected perimeter comprises a corner of an image of a surgical fabric. (Claim 16) 1. A method for analyzing a subject containing a blood constituent, comprising: acquiring depth and color images of an object including a blood component; using the depth image to determine the presence of the object within a field of view; and evaluating a characteristic of a blood constituent using the color image. (Claim 17) generating an image mask using the depth image; The method of claim 16 , further comprising applying the image mask to the color image to remove background pixels. (Claim 18) The method of claim 17 , further comprising applying at least one of erosion and dilation image processing to the depth image. (Claim 19) 1. A method for assessing the amount of a fluid component in a surgical fabric, comprising: acquiring a first infrared image of a field of view at a first time; Detecting a shape or surface characteristic of a surgical fabric in the first infrared image; indexing a frame counter when the shape or surface feature is detected in the first infrared image; acquiring a second infrared image of the field of view at a second time subsequent to the first time; and Detecting a surface characteristic of the surgical fabric within the second infrared image; indexing the frame counter when the shape or surface feature is detected in the second infrared image; and indexing a non-detection counter upon failure to detect a shape or surface characteristic of the surgical fabric in the second infrared image. (Claim 20) 20. The method of claim 19, further comprising clearing the frame counter and the non-detection counter when a preset value is reached. (Claim 21) comparing the frame counter to a threshold; acquiring a color image of the field of view when the frame counter is greater than or equal to the threshold; converting an area of ​​said color image corresponding to the surgical fabric into an estimate of the amount of blood constituent; 20. The method of claim 19, further comprising clearing the frame counter. (Claim 22) generating an image mask using the second infrared image; and 22. The method of claim 21, further comprising: applying the image mask to the color image to identify the region of the color image that corresponds to the surgical fabric. (Claim 23) 1. A method for performing fluid monitoring, comprising: acquiring an infrared image of a field of view; acquiring a color image of the field of view; generating an image mask using the infrared image; and using the image mask on the color image to identify areas of the color image that correspond to surgical fabric. (Claim 24) The method of claim 23 , further comprising converting the infrared image into a geometric perspective view of the color image. (Claim 25) The method of claim 23 , further comprising down-resolving the infrared image prior to generating the image mask. (Claim 26) 26. The method of claim 25, further comprising high resolution processing of the image mask prior to using the image mask to identify the region of the color image.

Claims

1. 1. A method for camera analysis of a surgical fabric, comprising: acquiring a first depth image of the field of view of the camera at a first time; detecting a shape of the surgical fabric in the first depth image; incrementing a frame counter upon detecting the shape in the first depth image; acquiring a second depth image of the field of view of the camera at a second time subsequent to the first time; detecting whether the shape of the surgical fabric appears in the second depth image; incrementing the value of the frame counter when detecting that the shape of the surgical fabric appears in the second depth image, and incrementing the value of a non-detection counter when detecting that the shape of the surgical fabric does not appear in the second depth image; acquiring a color image of the field of view of the camera based on the frame counter being equal to or greater than a threshold value; analyzing an area of ​​the color image corresponding to the surgical fabric to assess an amount of blood constituents on the surgical fabric; The method comprising:

2. 1. A method for camera analysis of a surgical fabric, comprising: acquiring a first depth image of the field of view of the camera at a first time; detecting surface characteristics of the surgical fabric in the first depth image; incrementing a frame counter upon detecting the surface feature in the first depth image; acquiring a second depth image of the field of view of the camera at a second time subsequent to the first time; detecting whether a surface characteristic of the surgical fabric appears in the second depth image; incrementing a frame counter upon detecting that a surface characteristic of the surgical fabric appears in the second depth image or incrementing a non-detection counter upon detecting that a surface characteristic of the surgical fabric does not appear in the second depth image; acquiring a color image of the field of view of the camera based on the frame counter being equal to or greater than a threshold value; analyzing an area of ​​the color image corresponding to the surgical fabric to assess an amount of blood constituents on the surgical fabric; The method comprising:

3. The method of claim 1 or 2, further comprising clearing the frame counter when the frame counter reaches a predesignated value.

4. The method of claim 1 or 2, further comprising clearing the non-detection counter when the non-detection counter reaches a pre-specified value.

5. the first depth image is a first infrared image and the second depth image is a second infrared image; generating an image mask using the second infrared image; using the image mask together with the color image to identify an area of ​​the color image that corresponds to the surgical fabric; The method of claim 1 or 2, further comprising:

6. 1. A system for analyzing surgical fabric, comprising: a depth imaging system configured to acquire one or more depth images of the surgical fabric; a color imaging system configured to acquire one or more color images of the surgical fabric; receiving from the depth imaging system a first depth image of a field of view of the depth imaging system at a first time; Detecting the shape of the surgical fabric in the first depth image; incrementing a frame counter upon detecting the shape of the surgical fabric in the first depth image; receiving a second depth image of the field of view of the depth imaging system from the depth imaging system at a second time subsequent to the first time; Detecting whether the shape of the surgical fabric appears in the second depth image; incrementing the value of the frame counter when detecting that the shape of the surgical fabric appears in the second depth image, and incrementing the value of a non-detection counter when detecting that the shape of the surgical fabric does not appear in the second depth image; acquiring a color image of the field of view of the color imaging system based on the value of the frame counter being equal to or greater than a threshold; Analyzing an area of ​​the color image corresponding to the surgical fabric to assess the amount of blood constituents on the surgical fabric. A processor configured as The system comprising:

7. 1. A system for analyzing surgical fabric, comprising: a depth imaging system configured to acquire one or more depth images of the surgical fabric; a color imaging system configured to acquire one or more color images of the surgical fabric; receiving from the depth imaging system a first depth image of a field of view of the depth imaging system at a first time; Detecting surface characteristics of the surgical fabric in the first depth image; incrementing a frame counter upon detecting the surface feature in the first depth image; receiving a second depth image of the field of view of the depth imaging system from the depth imaging system at a second time subsequent to the first time; Detecting whether a surface characteristic of the surgical fabric appears in the second depth image; incrementing the frame counter when it is detected that the surface characteristic of the surgical fabric appears in the second depth image, and incrementing a non-detection counter when it is detected that the surface characteristic of the surgical fabric does not appear in the second depth image; acquiring a color image of the field of view of the color imaging system based on the value of the frame counter being equal to or greater than a threshold; Analyzing an area of ​​the color image corresponding to the surgical fabric to assess the amount of blood constituents on the surgical fabric. A processor configured as The system comprising:

8. The system of claim 6 or 7, wherein the processor is further configured to clear the frame counter when the frame counter reaches a predesignated value.

9. 8. The system according to claim 6, wherein the non-detection counter is cleared on condition that the non-detection counter reaches a pre-specified value.

10. the first depth image is a first infrared image and the second depth image is a second infrared image; The processor, generating an image mask using the second infrared image; Using the image mask in conjunction with the color image to identify an area of ​​the color image that corresponds to the surgical fabric. The system according to claim 6 or 7, further configured to:

Images