Estimation of blood loss within a waste container of a medical waste collection system

Abstract

A method of estimating blood loss within a waste container. An image or a video feed of the waste container is captured, including waste material disposed therein. A neural network may determining a probabilistic location of at least one comer of at least one reference marker. The reference marker(s) may be used to align the image. Locations of a fluid and foam menisci may be determined. A correction factor may be determined based on the location of the foam meniscus. A volume of the waste material within the waste container is determined based on the correction factor and the location of the fluid meniscus. The blood loss is determined based on the volume and a blood component as analyzed in the image(s). The blood loss may be displayed on a display in real-time, thereby facilitating improved monitoring of blood loss while the waste material is being drawn into the waste container.

Description

ESTIMATION OF BLOOD LOSS WITHIN A WASTE CONTAINER OF A MEDICAL WASTE COLLECTION SYSTEMCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and all the benefits of United States Provisional Patent Application No. 63 / 612,478, filed December 20, 2023, the entire contents of which are hereby incorporated by reference.BACKGROUND

[0002] A byproduct of surgical procedures is the generation of liquid, semisolid, and / or solid waste material. The medical waste may include blood, interstitial fluids, mucus, irrigating fluids, and the like. The medical waste may be removed from the surgical site through a suction tube under the influence of a vacuum from a vacuum source to be collected within a waste container.

[0003] Estimating blood loss during surgery may be used to monitor intraoperative patient health. Advances in imaging and computing have provided for estimating blood loss by capturing an image of the fluid-containing media, such as a freestanding container. One such system is sold under the tradename Triton by Gauss Surgical, Inc. (Menlo Park, Calif.) and disclosed in commonly-owned United States Patent No. 9,773,320, issued September 26, 2017, and United States Patent No. 9,824,441, issued November 21, 2017, and United States Patent No. 10,641,644, issued May 5, 2020, the entire contents of each being hereby incorporated by reference. The system utilizes a reference marker to identify a region of interest for image analysis. The system may require the user to enter the fluid volume of the fluid observed within the container.

[0004] To provide for improved estimation of blood loss, it would be desirable for the fluid volume to be determined based on the image analysis. Such an improvement would provide for superior accuracy and / or real-time, continuous monitoring of blood loss, among other advantages. Determining the fluid volume based on the image analysis, particularly of a video feed, is associated with several technical challenges not satisfactorily addressed by the current state of the art. One non-limiting example is a foam or froth layer that may develop based on the turbulent nature of the fluid being drawn into the waste container under vacuum. Therefore, thereis a need in the art for improved systems and methods for estimating a quantity of a blood component in a fluid canister.SUMMARY

[0005] The present disclosure is directed to methods of estimating blood loss within a waste container, for example, a medical waste collection system. One initiated, the methods may be performed in a continuous or near-continuous manner, or until otherwise deactivated. The methods may be a computer program product stored on non-transitory computer readable medium configured to be executed by one or more processors. The processors may be those of the imaging device used to capture images or video feed of the waste container. Therefore, the present disclosure provides for leveraging the imaging and computing capabilities of, for example, certain smartphones to provide for real-time monitoring of blood loss while waste material is being drawn into the waste container under influence of suction. The blood loss is updated and displayed on real-time on a display, for example, the touchscreen display of the smartphone. The methods disclosed herein address many of the technical challenges associated with doing so.

[0006] Therefore, according to a first aspect, a method of estimating blood loss based on waste material of a medical waste collection system includes capturing, with an imaging device, an image or a video feed of the waste container and the waste material disposed therein. For example, the image may be an image frame of the video feed. A location of a fluid meniscus of the waste material is detected. Further, a location of a foam meniscus of the waste material is detected. A correction factor may be determined based on the location of the foam meniscus. The estimated blood is determined based, in part, on the correction factor and the location of the fluid meniscus.

[0007] According to a second aspect, a method of estimating blood loss within a waste container of a medical waste collection system includes capturing, with the imaging device, the image or the video feed of the waste container and the waste material disposed therein. At least one reference marker affixed to the waste container is detected. The image is provided as an input to a neural network. A heatmap corresponding to a probabilistic location of at least one corner of the at least one reference marker may be generated. Based on the probabilistic location of the at least one corner of the at least one reference marker, a region of interest for processing the image for estimating the blood loss is identified.

[0008] According to a third aspect, a method of estimating blood loss within a waste container of a medical waste collection system includes capturing, with the imaging device, the image or the video feed of the waste container and the waste material disposed therein. At least one reference marker affixed to the waste container is detected. The image of the waste container is registered from an image coordinate system to a canonical coordinate system. The registration may include providing the image as an input to a neural network, which is processed as a heatmap corresponding to a probabilistic location of at least one comer of the at least one reference marker. The image is aligned based on the probabilistic location(s), and a reference point of the imaging coordinate system of the imaging device.

[0009] Any of the above aspects can be combined in part or in whole with any other aspect. Any of the above aspects, whether combined in part or in whole, can be further combined with any of the following implementations, in full or in part. The volume of the medical waste is determined, and the blood loss is determined based on the volume and an estimated blood component from analysis of the image(s). The blood loss may be displayed on a display.

[0010] In certain implementations, the method further includes segmenting the image by classifying pixel values of the image as at least one of blood, non-blood, fluid, fluid surface, fluid meniscus, foam, foam surface, foam meniscus, and empty. Class labels are assigned based on the classification of the pixel values. The step of segmenting the image may include providing the image as an input to a neural network. The neural network may be trained on an image dataset which includes images of a similar waste container filled with various amounts of fluid. The neural network may classify the pixel values and assign the class labels based on the classification of the pixel values. The location of the fluid meniscus may be determined based on which of the pixel values of the image were assigned the class label of fluid meniscus. Detecting the location of the fluid meniscus may include fitting a parabola to pixel values which were assigned the class label of fluid meniscus, and the location of the fluid meniscus may be determined as a vertex of the parabola fit to the pixel values which were assigned the class label of fluid meniscus. Similar to the location of the fluid meniscus, the location of the foam meniscus may be determined based on which of the pixel values of the image were assigned the class label of foam meniscus. Detecting the location of the foam meniscus may include fitting a parabola to the pixel values which were assigned the class label of foam meniscus, and the location of the foam meniscus may bedetermined as a vertex of the parabola fit to the pixel values which were assigned the class label of foam meniscus.

[0011] In certain implementations, the step of detecting the location of the fluid meniscus of the waste material includes detecting a pixel-based location of a fluid meniscus in the image frames. Similarly, the step of detecting the location of the foam meniscus of the waste material includes detecting, with the one or more processors, a pixel-based location of a foam meniscus in the image frames. A correction factor may be determined based on the location of the foam meniscus and the location of the fluid meniscus. Additionally or alternatively, a volume of foam may be determined based on the image. The correction factor may then be determined based on the location of the foam, and optionally, a density of the foam.

[0012] In certain implementations, the method further includes registering the image to a coordinate system. After registering the image, the method may include detecting at least one of the locations of the fluid meniscus and the foam meniscus by determining the location of the fluid meniscus and / or the foam meniscus relative to an axis of the coordinate system. Further, the correction factor may be determined based on a distance between the locations of the fluid meniscus and the foam meniscus along the axis of the coordinate system. A pre-correction blood loss may be estimated based on the location of the fluid meniscus, and estimating the blood loss by adding the correction factor to the pre-correction blood loss.

[0013] In certain implementations, the methods of improved corner detection may further include applying a non-maximum suppression algorithm to the heatmap to generate a filtered heatmap. The probabilistic location of at least one corner of the at least one reference marker may be identified according to points representing at least one local maxima within the filtered heatmap. A clustering algorithm may be applied to the filtered heatmap to generate a clustered heatmap. The probabilistic location of at least one corner of the at least one reference marker may be further identified according to a location of a representative point within the clustered heatmap. The clustered heatmap may include at least one point cluster and the representative point may be determined as a local maximum within the at least one point cluster. The clustered heatmap may include at least one point cluster and the representative point may be determined as a weighted average of each point within the at least one point cluster.

[0014] In certain implementations, an expected number of clusters within the clustered heatmap may be received or determined, and an actual number of clusters within the clusteredheatmap based on the at least one point cluster may be determined. The actual number may be compared to the expected number to determine whether the actual number of clusters is higher than the expected number of clusters. A confidence score may be assigned to each cluster of the at least one point cluster. At least one cluster of the at least one point cluster may be discarded based on the assigned confidence scores such that the actual number of clusters equals the expected number of clusters. The probabilistic location of at least one corner of the at least one reference marker based may be determined on a set of remaining point clusters.

[0015] Additional advantages and aspects of the present disclosure will be appreciated from the description that follows and accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG. 1 shows a medical waste collection system configured to suction medical waste through a suction tube to be collected in a waste container. A device cradle is coupled to the medical waste collection system and positioned to removably receive an imaging device configured to capture an image of the waste container.

[0017] FIG. 2 is a top view of the waste container with a schematic representation of a field of view of the imaging device.

[0018] FIG. 3 is a side view of the waste container with a schematic representation of a field of view of the imaging device.

[0019] FIG. 4 is a perspective view of the waste container including an insert disposed therein. An alignment frame and reference markers are coupled to the exterior of the waste container.

[0020] FIGS. 5 A and 5B depict steps of methods for estimating blood loss in which a video feed is captured to be analyzed by one or more processors.

[0021] FIG. 6 is a perspective view of the waste container with fluids disposed therein, and with machine vision output overlying the reference markers for image alignment.

[0022] FIG. 7 depicts substeps of the method of FIGS. 5 A and 5B for detecting reference markers.

[0023] FIGS. 8A-8D are visualizations of various steps used to detect corners of reference markers according to the substeps of FIG. 7.

[0024] FIGS. 9A-9C are representative of the processor(s) using the reference markers for image alignment.

[0025] FIGS. 10A-10F are representations of certain steps of the method applied to a first image frame being processed during collection of the waste material in the waste container.

[0026] FIGS. 11A-11F are representations of certain steps of the method applied to a second image frame being processed during collection of the waste material in the waste container.

[0027] FIGS. 12A-12F are representations of certain steps of the method applied to a third image frame being processed during collection of the waste material in the waste container.

[0028] FIGS. 13A-13F are representations of certain steps of the method applied to a fourth image frame being processed during collection of the waste material in the waste container.

[0029] FIG. 14 is exemplary volume calculation plane for use with steps of the method of FIGS. 5A or 5B.

[0030] FIG. 15 is a partial view of the waste container depicting the machine vision output overlying the reference marker, and machine vision output associated with a region of interest.DETAILED DESCRIPTION

[0031] FIG. 1 shows a medical waste collection system 20 for collecting waste material generated during medical procedures. The medical waste collection system 20 includes a chassis 22 and wheels 24 for moving the chassis 22 within a medical facility. At least one waste container 26 is supported on the chassis 22 and defines a waste volume for receiving and collecting the waste material. In implementations in which there is more than one waste container, an upper waste container 26 may be positioned above a lower waste container 26, and a valve (not shown) may facilitate transferring the waste material from the upper waste container 26 to the lower waste container 26. A vacuum source 30 is supported on the chassis 22 and configured to draw suction on the waste container(s) 26 through one or more internal lines. The vacuum source 30 may include a vacuum pump, and a vacuum regulator configured to regulate a level of the suction drawn on the waste container(s). Suitable construction and operation of several subsystems of the medical waste collection system 20 are disclosed in commonly-owned United States Patent No. 7,621,898, issued November 24, 2009, United States Patent No. 10,105,470, issued October 23, 2018, and United States Patent No. 11,160,909, issued November 2, 2021, the entire contents ofeach being hereby incorporated by reference. Subsequent discussion is with reference to the upper waste container, but it should be appreciated that the objects of the present disclosure may be alternatively or concurrently extended to the lower waste container. In alternative implementation, it is contemplated that the methods disclosed herein may be used on a freestanding canister in which medical waste is drawn therein by suction provided by a facility-integrated vacuum source.

[0032] The medical waste collection system 20 includes at least one receiver 28 supported on the chassis 22. The receiver 28 defines an opening sized to removably receive at least a portion of a manifold 34. A suction path may be established from a suction tube 36 to the waste container 26 through the manifold 34 removably inserted into the receiver 28. In other words, the vacuum generated by the vacuum source 30 is drawn on the suction tubes 36, and the waste material is drawn from the surgical site through the suction tube 36, the manifold 34, and the receiver 28 to be collected in the waste container 26. The manifold 34 may be a disposable component with exemplary implementations of the receiver 28 and the manifold 34 disclosed in commonly-owned United States Patent No. 10,471,188, issued November 12, 2019, the entire contents of which are hereby incorporated by reference.

[0033] The medical waste collection system 20 may include a fluid measuring subsystem 38 (see FIG. 4), a cleaning subsystem, and / or a container lamp or backlight. An exemplary implementation of the fluid measuring subsystem 38 is disclosed in the aforementioned United States Patent 7,621,898 in which a float element 79 is movably disposed along a sensor rod 80 (see also FIG. 4). Based on signals received from the fluid measuring subsystem 38 indicative of a fluid level, a processor in electronic communication with the fluid measuring system 38 is configured to determine a fluid volume of the waste material in the waste container 26. The fluid level determined by the controller of the fluid measuring subsystem 38 may be combined with the methods provided below. The cleaning subsystem 40 may include sprayers rotatably disposed within the waste container 26 and configured to direct pressurized liquid against an inner surface of the waste container 26, as disclosed in the aforementioned United States Patent No. 10,105,470. Lastly, the container backlight is configured to illuminate an interior of the waste container 26. The container backlight may be activated based on an input to a user interface 52, or another device in communication with the controller 42.

[0034] The chassis 22 includes a front casing 46 that defines at least one cutout or window 48 to expose a portion of the waste container 26. The waste container 26 may be formedwith transparent material through which a user may visually observe the waste material collected within the waste containers 26, and, if needed, visually approximate a volume of the waste material collected therein with volumetric markings disposed on an outer surface of the waste container 26 (see FIG. 4). The waste container 26 being optically clear also permits the waste material collected therein to be imaged by an imaging device 50, preferably a camera. The imaging device 50 may also optionally include a user interface (e.g., a touchscreen display), and a flash or light source. The video feed e.g., image frames thereof) from the imaging device 50 may be transmitted to and processed by one or more processors 44, hereinafter referred to in the singular', to determine a blood component within the waste material. The blood component may be a constituent concentration within the waste material (e.g., hemoglobin concentration). More particularly, for example, certain aspects of the optical properties of the waste material may be analyzed and processed in a manner similar to that disclosed in commonly-owned United States Patent No. 8,792,693, issued on July 29, 2014, the entire contents of which are hereby incorporated by reference, to determine the blood component. The blood volume within the waste material may be estimated from the determined blood component and the volume of the waste material. As mentioned, however, determining the volume of the waste material (and the blood component) from a video feed in real-time as the medical waste is being drawn into the waste container 26 is associated significant technical challenges overcome by the devices and methods to described.

[0035] With further reference to FIG. 1, a device cradle 54 is removably coupled or rigidly secured to the chassis 22. The device cradle 54 positions the imaging device 50 relative to the waste container 26 in a precise manner to provide for continuous image capture (e.g., the video feed) of at least a portion of the waste container 26. The video feed results in continuous data from which the fluid volume, and the blood component therein, may be determined in real-time. As a result, the estimated blood loss (eBL) may be continuously updated and streamed on a display (e.g., the touchscreen display of the imaging device 50, a user interface 52 of the medical waste collection system 20, and / or another display terminal). Such advantages are not readily feasible with a handheld imaging device that requires the user to manually support the imaging device during image capture, and to manually input an estimated volume based on a visual observation. Moreover, the image-based volumetric determinations obviate the need for the imaging device 50 be in data communication with the processor 44 of the medical waste collection system 20. Stillfurther, the precise positioning may be designed to reduce glare and eliminate variations or aberrations in lighting to improve accuracy of the algorithmic determinations.

[0036] Referring now to FIGS. 2 and 3, a representation of the device cradle 54 with the imaging device 50 supported therein. The imaging device 50 may be a smartphone with suitable imaging and processing capabilities (e.g., iPhone), a tablet, digital camera, or the like. The device cradle 54 may be dimensioned such that a field of view of an image sensor 64 the imaging device 50 spans at least a portion or an entirety of the width of the waste container 26, and optionally, to provide desired spacing between the flash of the imaging device 50 and the surface of the waste container 26. The designed spacing may be tuned to reduce glare while adequately illuminating the waste container 26 with the flash of the imaging device 50. Further, the field of view of the imaging device 50 may span from below a bottom of the waste container 26, to above a top of the waste container 26 or a predetermined fill level thereof. FIG. 3 shows the field of view of the imaging device 50 including up to approximately 4,000 mL of fluid within the waste container 26. The distance by which the imaging device 50 is spaced apart from the surface of the waste container 26 may be based on the aspect ratio of the imaging device 50 e.g., 4:3, 16:9, etc.). In one example, the precise positioning includes the imaging device 50 being spaced apart from the surface of the waste container 26 by a distance within the range of approximately 30 millimeters (mm) to 70 mm, and more particularly within the range of approximately 40 to 60 mm, and even more particularly at approximately 50 mm. In certain examples, a horizontal viewing angle may be within the range of approximately 80° to 90°, and a vertical viewing angle may be within the range of approximately 100° to 105° - corresponding lens viewing angles may be within the range of approximately 110° to 120°. Additional optical components such as a fish-eye lens and mirrors may be used on the device cradle 54 to expand the field of view of the imaging device 50. With the imaging device and the flash facing towards the waste container 26, the touchscreen display of the imaging device 50 remains visible and operable, including displaying the field of view or an augmented field of the imaging device. It is further appreciated that the arrangement facilitates the imaging device 50 being removable from and replaceable within the device cradle 54 in a quick and intuitive manner.

[0037] In another implementation, the imaging device 50 may be integrated on the chassis 22, and not necessarily disposed on a mobile device supported on the device cradle 54. In such an implementation, the imaging device 50 may be at least one digital camera coupled to thechassis 22 in any suitable location, such as within the front casing 46. Additionally or alternatively, the digital camera may be coupled to the waste container 26 to be positioned internal and / or external to the waste volume. For example, the digital camera may be coupled to the container lid 82 and oriented downwardly. Multiple cameras may be utilized in combination from which the images are analyzed with the analysis synthetized by the machine learning algorithm for improved accuracy and redundancy.

[0038] Referring now to FIG. 4, an insert 74 is optionally disposed within the waste container 26 to be within the field of view of the imaging device 50. The insert 74 includes several geometries, at least one of which is an imaging surface 75 to be spaced apart from an inner surface of the waste container 26 to define a gap of known and fixed distance. The gap permits a thin layer of fluid to be situated between the insert 74 and the inner surface of the waste container 26 that exhibits a region of at least substantially uniform color that is below a color intensity to cause signal saturation. The imaging surface 75 may include first and second imaging surfaces 75a, 75b positioned lateral to one another in a side-by-side arrangement (z.e., a multi-level insert). The first imaging surface 75a and the second imaging surface 75b may be separated by a ridge having a thickness equal to a difference between the first distance and the second distance. Alternatively, the imaging surface 75 may also include a continuous gradient imaging surface such as in the shape of a wedge to allow several levels of increasing fluid color intensity being measured. The insert 74 may be mounted to a container lid 82 of the waste container 26. An exemplary implementation the insert 74 is disclosed in the aforementioned United States Patent No. 9,773,320.

[0039] At least one reference marker 76 may be detected by the imaging device 50 for locating the region of the image associated with the imaging feature of the insert 74, and for colorcorrecting for variances in lighting, flash, or other optical aberrations. Herein, the reference marker(s) 76 are referred to in the singular’ as the reference marker 76, but any reference to a singular reference marker 76 should be understood as including either one reference marker 76 and / or a plurality of reference markers 76. One suitable implementation of the reference marker is disclosed in commonly-owned United States Patent No. 9,824,441, issued November 21, 2017, the entire contents of which are hereby incorporated by reference, in which a quick response (QR) code of a known red color component value (or RGB, HSV, or CMYK color schemes, etc.) is affixed with adhesive to an outer surface of the waste container 26 corresponding to a position of an upper aspect of the insert 74. Further, in a manner to be described, calibration data may beassociated with the unique code of the reference marker 76 to account for mechanical variances of the waste container 26. Still further, in a preferred implementation, at least two reference markers 76 may be affixed to the waste container 26 in a manner to facilitate image alignment. The locating of the region of interest is optional, and the image analysis may be performed on cropped sections of images to be described.

[0040] An alignment frame 90 may also be affixed and contoured to the waste container 26 to assist in detecting a tilt of the container 26 relative to the imaging device 50 (i.e., relative to an image coordinate system). The alignment frame 90 may include supplemental markers 92, which may be similar to the reference markers 76. Any of the methods described herein may alternatively be carried out with the supplemental markers 92 of the alignment frame 90 as opposed to reference marker(s) 76.

[0041] Referring now to FIGS. 5A and 5B, a method 100 of estimating the volume of blood within the waste container 26 is provided, also referred to herein as estimated or estimating blood loss. The method 100 and one or more of its steps disclosed herein are configured to be executed by the processor(s) 44 according to instructions stored on non-transitory computer readable medium. The data may be analyzed by the processor 44 on the imaging device 50, the processor 44 within the chassis 22, and / or the data may be transmitted for remote processing (e.g., cloud computing). The method 100 may be executed with a machine-learning (ML) model, or one or more trained neural networks may be implemented. The computer-executable instructions may be implemented using an application, applet, host, server, network, website, communication service, communication interface, hardware, firmware, software, or the like. The computer- readable medium can be stored on any suitable computer readable media such as RAM, ROM, flash memory, EEPROM, optical devices, hard drives, floppy drives, or the like.

[0042] The neural network(s) may execute algorithms trained using supervised, unsupervised, and / or semi-supervised learning methodology in order to perform the methods of the present disclosure. A diverse and representative set of training dataset is collected to train each of the neural networks on diverse scenarios both within and outside of expected uses of the system 20. For example, the neural network may be trained on images of a similar waste container filled with various amounts of fluid. Training methods may include data augmentation, pre-trained neural networks, and / or use semi- supervised learning methodology to reduce the requirement of labeled training dataset. For example, data augmentation may increase sample size such as varyingblood concentrations, semisolid, varying soiling of the sidewall of the waste container 26, lighting conditions (brightness, contrast, hue-saturation shift, Planckian Jitter, etc.), geometric transformations (rotation, homography, flips, etc.), noise and blur addition, custom augmentation, hemolysis, placement of imaging device 50, clotting, detergent, suction level, or the like. Further, tolerance limit testing may also occur to determine acceptable variations for parameters such as insert depth, canister tilt, and mechanical variations of the waste container 26.

[0043] The method 100 may include operating the system in an optional standby or low-power mode (step 102). As implied by its name, the low-power mode is configured to preserve power (e.g., battery life of the imaging device 50) for periods of time in which there is no change in fluid level within the waste container 26. For example, the software and the image sensor 64 of the imaging device 50 may be initiated prior to commencement of the surgical procedure; however, the medical waste collection system 20 is not drawing waste material into the waste container 26. The low-power mode 102 may include capturing, with the imaging device 50, the video feed (z'.e., a series of image frames) (step 104) with the flash off and at a relatively lower frame rate. In one example, the frame rate is approximately 15 frames per second (fps), but other frame rates are within the scope of the present disclosure. The image frames - also referred to herein as images - are preprocessed (step 106), for example, downsampled. The downsampled images are analyzed by an activity recognition algorithm (step 108). The activity recognition algorithm detects a pixel-based location of a fluid meniscus (z’.e., predominately liquid meniscus) in each of the images (as detected by the step(s) below). The activity recognition algorithm is configured to determine whether the pixel-based location of the fluid meniscus changes by a greater amount than a predetermined threshold in a predetermined period of time. In other words, activity recognition algorithm determines whether the waste material is being drawn into the waste container 26 at a predetermined rate. In one example, the activity recognition algorithm may compare the pixel-based location the fluid meniscus between successive image frames of the video feed and compare the changes against the predetermined threshold. If the changes remain below the predetermined threshold, the processor 44 maintains the system in the low-power mode, and foregoes executing the remainder of the method 100. If the change in the fluid meniscus is above the predetermined threshold, the processor 44 executes remaining steps of the method 100. Additionally or alternatively, the user may provide an input to the touchscreen display of theimaging device 50, a paired external device, or the like, to terminate the low-power mode and initiate an cBL mode.

[0044] The method 100 includes activating the light source (step 110), such as activating the flash of the imaging device 50. The step is optional, and alternatively the flash of the imaging device 50 may in continuously activated. In another variant, a level of the flash of the imaging device 50 may be increased with the eBL mode. The container backlight of the waste container 26 may already be activated. Using light from both the flash and the container backlight may provide optimal illumination with minimal glare. Owing to the precise position of the imaging device 50 supported by the device cradle 54, it is noted that the glare may be limited to a small dot positioned above the insert 74 and the reference marker 76, which is a region of the image of less concern for the image analysis. Moreover, the greater signal-to-noise ratio (z.e., blood color signal to background reflection) has less effect on the image analysis, and therefore may obviate the need for a separate algorithm to reduce or remove the influence of glare. In addition to activating the flash, the feed rate of the video feed may be increased for increased data resolution.

[0045] The method 100 may include analyzing at least one of the image frames of the video feed (step 112), and preferably multiple image frames in a series according to the algorithm. In other words, every image frame may be analyzed, every other image frame, every third image frame, or the like, based on desired data resolution in view of available computing resources. In an alternative implementation not utilizing a video feed, the method 100 may include the step of capturing one or more multi-exposure photos for the subsequent analysis.

[0046] The method 100 includes the step of detecting the reference marker 76 (step 114), for example, the QR code(s) 76, a barcode, the supplemental marker(s) 92, or another marker having optically-readable data. More specifically, the step of detecting the reference marker 76 may include determining the location of at least one comer 77 of the reference marker 76. With concurrent reference to FIGS. 5A-5B and 7-8D, the step of detection the reference marker 76 (step 114) is shown along with its substeps 114A-114E. FIG. 7 includes a flow diagram of step 114, and FIGS. 8A-8D depict visual representations of various heatmaps output by the neural network during step 114. The heatmaps of FIGS. 8B-8D are cropped to sections of the image which include the reference markers 76; however, the heatmaps may be uncropped as well.

[0047] In order to determine the location of the at least one corner 77, starting with substep 114A, step 114 includes providing the image captured during step 112 to a neural network.The neural network may be a trained neural network which was trained on images of reference markers, such as QR codes, and / or images of waste containers containing fluid which arc similar to the waste container 26. After the image is input to the neural network, substep 114B is carried out and the neural network outputs a heatmap corresponding to probabilistic locations of the at least one corner 77 of the reference marker 76 which is received by the processor 44.

[0048] In some implementations, the neural network outputs a heatmap 78 for each of the at least one comer 77, as shown in FIG. 8A. The neural network may process four heatmaps 78, each heatmap 78 corresponding to the probabilistic location of a specific one of the at least one comer 77 of each of (or one of) the reference markers 76. FIG. 8A depicts four exemplary heatmaps 78 which each provide the probabilistic locations of specific corners 77 of each detected reference marker 76. More specifically, in FIG. 8A, the heatmaps 78 correspond to upper left, upper right, bottom left, and bottom right comers of each detected reference marker 76, as determined by the neural network. For illustration, a phantom shape is also shown in FIGS. 8A- 8D corresponds to the actual location of the reference markers 76; however, the heatmaps 78 may include only the probabilistic locations of the corners 77. In these figures, each point P represents a location in the image, such as a coordinate point in an image coordinate system (e.g., the coordinate system associated with the images captured by the imaging device 50), where the neural network has classified as likely corresponding to the specific comer 77 (e.g., upper left corner) of the reference marker 76. For example, looking to FIG. 8A, the points P in one heatmap 78 are all classified as corresponding to one comer 77, while the points P in each of the other heatmaps 78 are classified as corresponding to each of the other comers 77, respectively. Further, each point P may have a confidence score assigned by the neural network based on the likelihood of the point P accurately representing the corner 77.

[0049] The method 100 may include further processing the heatmap(s) 78 during step 114. For example, substep 114C includes applying a non-maximum suppression (NMS) algorithm, such as the MaxPool2D algorithm, to each heatmap 78 output by the neural network. As shown in FIG. 8B, the NMS algorithm converts the heatmap 78 into a filtered heatmap 78s by suppressing all but local maxima M within the heatmap 78. The local maxima M are the points P shown in FIG. 8A which have the highest confidence score(s). In some implementations, the local maxima M arc points P which have a confidence score higher than a threshold confidence score.After the NMS algorithm is applied to the heatmap 78s, during substep 114D, a clustering algorithm, such as DBSCAN, may be applied to the filtered hcatmap 78s.

[0050] As shown in FIG. 8C, the clustering algorithm converts the filtered heatmaps 78s into clustered heatmaps 78c by grouping the points P into point clusters C. Each point cluster C includes a group of the points P which correspond to one of the corners 77 of a specific reference marker 76. One point P may be clustered with another point P based on their proximity to one another. For example, a first point cluster Ci and a second point cluster C2 are shown in FIG. 8C. The first point cluster Ci corresponds to the upper left comer of one reference marker 76, and the second point cluster C2 corresponds to the upper left comer of another reference marker 76. The points P included in the first point cluster Ci were classified by the neural network as likely being the upper left comer of one reference marker 76, while the points P included in the second point cluster C2 were classified by the neural network as likely being the upper left corner of another reference marker 76. Further, the points P of the first point cluster Ci are within a proximity of one another, and the points P of the second point cluster C2 are also within a proximity of one another. In another one of the clustered heatmaps 78c, a third cluster C3 and a fourth cluster C4 are shown. Both of the third and fourth clusters C3, C4 contain points classified as corresponding to the bottom left corner of one reference marker 76. However, the clustering algorithm has found two point clusters C3, C4 because some of the points P within the third cluster C3 are outside the proximity of the points P of the fourth cluster C4 (and vice versa).

[0051] Subsequently, step 114 proceeds to substep 114E at which point the probabilistic locations of the corner(s) 77 are determined based on the clustered heatmap 78c, for example, by identifying a representative point RP within each point cluster C as shown in FIG. 8D. The processor 44 may identify the representative point RP within each point cluster C included in the clustered heatmap 78c in multiple ways. In one example, the processor 44 identifies the representative point(s) RP as the point P within the respective point cluster C which has the highest confidence score. In another example, the processor 44 computes a weighted average of all points P within the point cluster C and identifies the representative point RP as the weighted average. The weighted average can be found according to a product of the location of each point P and their respective confidence scores. The weighted average approach may also be applied to scenarios where two nearby clusters C are identified, such as the third and fourth point clusters C3, C4. In this case, the location of all of the points P in both clusters C3, C4 may be averaged to find therepresentative point RP. Once the representative points RP are found, the processor 44 identifies the probabilistic location of each of the corners 77 as the corresponding representative point RP.

[0052] The method 100 may determine whether the number of reference markers 76 detected during step 114 matches an expected number of reference markers 76. For example, the method 100 may expect there to be two reference markers 76 in the image based on an input from the user, calibration data, etc. In such an example, step 114 of the method 100 may include determining an expected number of point clusters C based on the expected number of reference markers 76, determining an actual number of point clusters C within the image based on the heatmap 78 and / or clustered heatmap 78c, and comparing the expected number to the actual number. The expected number of reference markers 76 and / or the expected number of point clusters C may be predetermined (e.g., during calibration) or otherwise known to the processor 44, If the actual number of point clusters C is higher than the expected number of point clusters C, the method 100 may include discarding at least one point cluster C such that the actual number of point clusters C matches the expected number of point clusters C. The method 100 may determine which point cluster(s) C to discard based on which point cluster(s) have the lowest confidence score(s).

[0053] The method 100 includes the step of aligning the image (step 116). As the assessment of the fluid meniscus is on the vertical, y-axis, it is desirable for the image of the canister to be oriented vertically with sufficient precision. In other words, the step of aligning the image may account for mechanical variances in the rotational positioning of the waste container 26 within the chassis 22, and / or variances in the positioning of the imaging device 50 within the device cradle 54. The alignment may be based on features external to the waste container 26 (i.e., external alignment), features associated with or inherent to the waste container (i.e., internal alignment), or a combination thereof. Additionally or alternatively, referring to FIGS. 9A-9C, the step of aligning the image may be based on the step of detecting the reference markers 76 (step 114 of the method 100). For example, the reference markers 76 may be affixed to be waste container 26 with a jig or guide to be vertically arranged with sufficient vertical precision. The processor 44 is configured to detect the reference markers 76, as represented by the bounding boxes and / or comers 77 detected during step 114 and shown in FIGS. 9A-9C, and rotate the image accordingly. Alignment techniques to orient and position the image within the frame in three dimensions are within the scope of the present disclosure, including homography, perspectivetransformation, 3D-to-2D mapping, and the like. FIGS. 10B, 1 1B, 12B and 13B show the effect of image rotation relative to FIGS. 10A, 11A, 12A and 13A, respectively.

[0054] Still referring to FIGS. 9A-9C, the step of aligning the image (step 116) may be based on the probabilistic location of the comers 77 of each reference marker 76. After the corners 77 of the reference markers 76 have been detected during the method 100 (step 114), the images may be aligned by aligning the comers 77 of the reference markers 76. In the illustrated embodiment, the reference markers 76 are shown as QR codes; however, other reference markers such as AruCo markers may be used instead. The figures also show that all four corners 77 of each reference marker 76 have been detected and used for image alignment; however, it is contemplated to only use one corner 77 (or two, three, etc.), such as the upper left corner, of each reference marker 76 for image alignment. In some implementations, two corners 77 of each detected reference marker 76 are used for image alignment . These two corners 77 may be any pair of corners 77 of the reference marker 76, but using two opposite corners 77 may be preferred. For example, the two opposite comers 77 may be the upper right comer and the lower left corner of the reference marker 76 such that the position of the reference marker 76 may be more effectively determined. FIG. 9A depicts the image captured by the imaging device 50. FIG. 9B shows the waste container 26 as interpreted during step 114 compared to an ideal waste container 26i oriented in a predefined alignment, for example, an alignment in which the reference markers 76 are aligned vertically. The waste container 26 of the image captured by the imaging device 50 may be aligned to the ideal waste container 26i by rotating and / or translating the image such that the comers 77 of the reference marker 76 on the waste container 26 are at similar canonical coordinates compared to the comers 77 of the reference marker 76 on the ideal waste container 26i. FIG. 9C shows the waste container 26 aligned in the predefined alignment e.g. , aligned with the ideal waste container 26i) according to step 116 as described above.

[0055] In some implementations, registering the image of the waste container 26 from the image coordinate system to the canonical coordinate system ( / .<?., aligning the image to the predefined alignment) includes aligning at least one of the detected corners 77 to a reference point(s) of the image coordinate system. The reference points may be the comers 77 of the reference marker 76 on the ideal waste container 26 or predefined coordinates of the image coordinate system.

[0056] Referring again to FIGS. 5A-5B and with further reference to FIGS. 10D, 11D, 12D and 13D, the method 100 includes the step of segmenting the image of the waste container 26, including the fluid contained therein. The neural network(s), hereinafter referred to in the singular, may characterize whether a pixel is of blood (B), non-blood (NB), fluid (FL), fluid surface, fluid meniscus (FLM), foam (FO), foam surface, foam meniscus (FOM), and empty. In other words, each pixel is assigned a value from the neural network, and then assigned a class label based on the value. The neural network may be a deep-learning neural network such as U-Net, Mask-RCNN, DeepLab etc., or any imaged-based segmentation algorithm such as grabcut, clustering, region-growth, etc. For further examples, the neural network may use object localization, segmentation (e.g., edge detection, background subtraction, grab-cut-based algorithms, etc.), gauging, clustering, pattern recognition, template matching, feature extraction, descriptor extraction (e.g., extraction of texton maps, color histograms, HOG, SIFT, MSER (maximally stable extremal regions for removing blob-features from the selected area, etc.), feature dimensionality reduction (e.g., PCA, K-Means, linear discriminant analysis, etc.), feature selection, thresholding, positioning, color analysis, parametric regression, non-parametric regression, un supervised or semi-supervised parametric or non-parametric regression, neural network and deep learning based methods or any other type of machine learning or machine vision. FIGS. 10D, 11D, 12D and 13D illustrate representations of a segmentation mask represented by different coloring for the non-blood, meniscus, and blood class labels. FIGS. 11D and 13D also show the reference markers 76 being excluded from the segmentation mask.

[0057] The method 100 includes determining the location of the fluid meniscus as well as a foam meniscus (step 132). The step 132 is based on the segmentation mask and may include using a full width of the image to post-process segments. The aforementioned machine learning techniques may be implemented to train the segmentation network on video feeds to identify the meniscus in varying conditions, including blood concentration levels, thickening agents, lighting from the flash, lighting from the container backlight, vacuum levels of the vacuum source 30, frothiness of the waste material, cloudiness, or opaqueness of the waste container 26, and the like. Other means by which the meniscus may be detected is disclosed in commonly-owned United States Patent No. 8,983,167, issued March 17, 2015, the entire contents of which are hereby incorporated by reference.

[0058] The step 132 may include fitting a first parabola to pixels having the class label of fluid meniscus FLM and a second parabola to pixels having the class label of foam meniscus FOM. FIGS. 10E, 1 IE, 12E and 13E illustrate representations of the fluid meniscus and the foam meniscus in which the non-blood and blood class labels are removed. In instances where the fluid meniscus and / or foam meniscus is below the imaging device 50, the parabolas include a lower vertex and open upwardly. Conversely, if the fluid meniscus and / or the foam meniscus is above the imaging device 50, the parabolas include a higher vertex and open downwardly. The locations of the fluid meniscus and the foam meniscus is the y-axis value of the vertex of the respective parabolic curve. Therefore, with the lower vertex, the location of the fluid and foam menisci is the minimum of the corresponding parabolic curve; and with the higher vertex, the location of the fluid and foam menisci is the maximum of the corresponding parabolic curve. FIGS. 10B, 11B, 12B and 13B illustrate representations of machine vision output 94 indicative of the y-axis values of the vertexes overlayed on the adjusted images of the waste container 26.

[0059] The method 100 further includes the step of mapping the images from the image coordinate space to the canonical coordinate space (step 124). The step may utilize the corners 77 of the reference marker 76 to generate homography and map the fluid / foam meniscus from the image coordinate system to the canonical coordinate system. In other words, one or more of the comers 77 of the QR codes 76 (e.g., top left corner) may be used for the transformation to two- dimensional canonical coordinates, then to three-dimensional canonical coordinates. FIGS. 10C, 11C, 12C and 13C illustrate representations of machine vision output 94’ in the canonical coordinates indicative of the y-axis values of the vertex overlayed on the adjusted images of the waste container 26, and FIGS. 10F, 1 IF, 12F and 13F illustrate representations of cropped adjusted images of the waste container 26 with the machine vision line 94’ overlayed. The cropped image frames in the canonical coordinate space may include the at least two reference markers 76, and a portion of the waste container disposed therebetween. A width of the cropped and mapped image frames is approximately equal to a width of the at least two reference markers 76. The cropping of the images limits the range (e.g. , the width of the image) in which the fluid and foam menisci may be located. The cropping of the adjusted images is optional.

[0060] The method 100 includes the step of pixel-to-volume mapping to determine the fluid volume within the waste container (step 126). The step may include retrieving or receiving the calibration data from the memory 88, for example, container- specific coefficients to accountfor container-specific variances, as described. The image-based volumetric determination may be based on (1) the y-axis value of the fluid meniscus in the canonical coordinates and (2) the y-axis value of the foam meniscus in the canonical coordinates. In other words, the lowest point of the meniscus, along a y-axis as defined in the image by the processor 44, may be mapped relative to a datum, and the y-axis position of the meniscus is converted to a volume in milliliters. The datum may be an uppermost aspect of the image. Again, the aforementioned machine learning techniques may be implemented to train the segmentation network to interpret the y-axis positions of the fluid meniscus and the foam meniscus as corresponding to a specific fluid volume. As one example and with reference to FIG. 11C, the fluid meniscus has a y-axis value of approximately “2500” in the canonical coordinates (left axis of image). Further, the foam meniscus has a y-axis value of approximately “2300” in the canonical coordinates. Based on the fluid meniscus and the foam meniscus y-axis values of 2500 and 2300, the fluid volume may be determined to equal approximately 1 ,300mL.

[0061] The contribution of the foam meniscus to the fluid volume calculation may be based on various aspects of the image and the step of pixel-to-volume mapping (step 126) may include an analysis of the pixels classified as foam during the step of image segmentation (step 130). For example, the pixel-to-volume mapping may include determining a volume of the foam, a thickness of the foam, a density of the foam, and / or other characteristics of the foam within the waste container 26. In one implementation, after the image has been segmented (step 130), the pixel-to-volume mapping includes determining the y-axis value of the fluid meniscus in the canonical coordinates and the y-axis value of the foam meniscus in the canonical coordinates. The determination could be relative to a different coordinate system, such as the image coordinate system or global coordinate system. After the two y-axis values have been determined, an initial volume estimation may be determined based on the y-axis value corresponding to the fluid meniscus. A correction factor may then be determined based on the y-axis value of the foam meniscus - the difference between the y-axis values of the fluid meniscus and the foam meniscus may also be included in the determination of the correction factor. The correction factor provides a volumetric correction such that the volume calculation includes not only the fluid, but the foam as well. More specifically, the correction factor may be equal to an amount of fluid equal to the amount of fluid within the foam.

[0062] In order to determine how much fluid is represented by the foam in the image, the volume, thickness, and / or density of the foam may be determined. The volume, thickness, and density of the foam may be determined based on the pixels classified as foam in the segmented image. For example, referring back to FIG. 11C and the example calculation equating the fluid volume to approximately 1,300 mL, it was determined that the fluid meniscus and the foam meniscus y-axis values were approximately 2500 and 2300, respectively. The step of pixel-to- volume mapping may include a calculation of the correction factor based on the foam pixels present between y-axis values of 2300 and 2500. To that end, the foam pixels of the segmented image, such as the one represented by FIG. 11D, may include labels according to the color in addition to their classification as foam. Based on the color of the foam pixels, the density of the foam may be calculated. A lighter colored foam pixel may be interpreted as foam of relatively lower density, while a darker (e.g., more red) foam pixel may be interpreted as foam of relatively higher density. The color of the foam may also be used to determine the concentration of blood within the foam (e.g., higher red value of a pixel may represent a higher blood concentration). The volume and / or thickness may be determined based on the difference between the fluid meniscus and the foam meniscus (i.e., the area / volume defined between the fluid and foam menisci). After the volume and density of the foam has been determined, the correction factor may be calculated.

[0063] Using FIG. HD as an example, the height of the foam is approximately equal to 200 in the canonical coordinate system (spanning from 2300 to 2500). The height of the foam may be directly related to the volume of the foam if a relationship between the height and volume of the foam is known. Otherwise, a width of the foam may also be calculated based on a width of the foam. In FIG. HD, the width of the foam is approximately 2000 in the canonical coordinate system (spanning from 500 to 2500). The height and width of the foam may then be multiplied to determine the volume of the foam in the canonical coordinate system. In either case, the volume of the foam may be determined to be approximately 400,000 square pixels. The density of the foam may be determined as described above, and the correction factor may be determined by a formula including the volume and density of the foam. As the volume of fluid (as represented by the fluid pixels and the foam pixels) was determined to be approximately 1,300 mL, and the fluid pixels represent approximately 1,200 mL of fluid, the foam pixels represent approximately 100 mL of fluid. Thus, in this case, the foam has a density of around 1 mL of fluid per 4000 square pixels of foam.

[0064] The color of the foam pixels (or an average color of the foam pixels) may also be used to determine the concentration of the blood component within the foam. The relationship between the color of the foam pixels and the concentration of blood may be known to the trained neural network. For example, where the color of the pixel is stored as an RBG value with a red value, a blue value, and a green value, there may be a linear (or other) relationship between blood concentration and red value (and / or the blue and / or green values).

[0065] It will be appreciated that the volumetric determinations could be performed relative to a two-dimensional coordinate system or a three-dimensional coordinate system. For ease of illustration, the above refers to two-dimensional calculations but it is also contemplated to carry out the calculations relative to a three-dimensional coordinate system.

[0066] FIG. 14 shows an example of a volume calculation plane VCP that may be used to during the step of pixel-to- volume mapping (step 126) of the method 100. As noted above, the relationship between the fluid and foam menisci and the volume of waste material within the container 26 may be known, and this relationship may be used during the image-based volumetric determination (such as during step 126). FIG. 14 depicts this relationship as a plane of points in a three-dimensional coordinate space. The particular' three-dimensional coordinate space shown in FIG. 14 has orthogonal fluid and foam axes which are both normal to a volume axis. Thus, to determine the volume of waste material within the container 26, the volume of fluid and foam are plotted onto the volume calculation plane VCP. Although the relationship between the fluid and foam menisci and the volume of waste material is shown as a plane in 3D space, this is merely for illustrative purposes. The relationship may instead be stored as a set of entries within a database or as an equation which equates fluid and foam volumes to the volume of waste material.

[0067] The method 100 may optionally include extracting a region of interest (ROI) from the image (step 134). With concurrent reference to FIG. 15, the processor 44 may be configured to identify the reference marker 76, determine a pose 96 of the marker 76, and process the information contained therein to determine the region of interest 96’ corresponding to the identified marker 76. In other words, for example, the data within the QR code may cause the processor 44 to analyze the region of interest 96’ of the image frame at a predetermined position relative to the QR code. Further, in implementations in which the insert 74 includes the first and second imaging surfaces 75a, 75b, the reference marker 76 may contain data to cause the processor 44 to analyze first and second regions of interest 98a, 98b, respectively. In some implementations,the processor 44 may analyze the entire image, or a cropped section of the image, such as the cropped image of the waste container 26 shown in FIG. 15. The image may be cropped based on the location of the reference markers 76.

[0068] In certain implementations, further optional steps 136 may include extracting palette colors from the reference marker 76, noise removal, extracting features, and performing light normalization. The palette extraction, noise removal, and feature extraction may be performed in manner at least similar’ to the aforementioned United States Patent No. 9,824,441. Variances in lighting may be accounted for by training the segmentation network with images in which lighting from only the flash of the imaging device 50 is used, light from only the container backlight is used, or combinations of both at varying levels. Among other advantages, this may permit the blood component to be determined at greater ranges of blood concentration without signal saturation. The levels of light intensity provided by the flash of the imaging device 50 may be of known brightness, color temperature, spectrum, and the like. Additionally or alternatively, the container backlight may provide light of a known brightness, color temperature, spectrum, and the like.

[0069] The method 100 may further include analyzing the image to quantify a concentration the blood component in the waste material at step 138. The analysis may be carried out in a manner disclosed in the aforementioned United States Patent No. 8,792,693 in which a parametric model or a template matching algorithm is executed to determine the concentration of the blood component associated with fluid within the waste container 26. In particular, the processor 44 may be configured to extract a color component value (e.g., a redness value) from the image, and execute the trained algorithm to determine the concentration of the blood component on a pixel-by-pixel or other suitable basis.

[0070] The hemoglobin (Hb) or blood loss (eBL) may be estimated with a hemoglobin estimation algorithm (step 140) and based on the image-based volumetric determination (as performed during step 124) and the concentration of the blood component (as determined during step 138). In one implementation, the hemoglobin estimation algorithm determines the blood loss according to the amount of blood within the fluid inside the waste container 26 and the amount of blood within the foam inside the waste container 26. For example, the hemoglobin estimation algorithm may receive the volume of the fluid based on the fluid meniscus and the volume of the foam based on the foam meniscus, both volumes determined as described above. In such anexample, the hemoglobin estimation algorithm may also receive the concentration of the blood component within the fluid from the processor 44 as determined during step 138, and / or the concentration of the blood component within the foam as determined during step 126. Thus, the hemoglobin estimation algorithm may estimate the blood loss based on the volume of fluid, the concentration of the blood component within the fluid, the volume of foam, and the concentration of the blood component within the foam. In some implementations, the hemoglobin estimation algorithm may eschew the concentration of the blood component within the foam when estimating the blood loss. For example, the blood loss may be calculated based on the volume of fluid, the concentration of the blood component within the fluid, and the volume of foam. In such an example, a pre-correction blood loss may be calculated based on the volume of fluid and the concentration of the blood component within the fluid. To estimate the total blood loss, the hemoglobin estimation algorithm may then add the correction factor to the pre-correction blood loss, and the correction factor may be determined according to the location of the foam meniscus as described above.

[0071] The estimated blood loss may be displayed on one or more displays (step 142), for example, the touchscreen display of the imaging device 50. Additionally or alternatively, the eBL may be wirelessly transmitted to the user interface 52 of the medical waste collection system 20 or another display terminal within the surgical suite. As should be readily appreciated from the foregoing disclosure, the eBL may be displayed and updated at desired intervals or in real-time. In particular, in the video mode the imaging device 50 may take the video feed in which each of the blood component and the fluid volume is repeatedly determined in a near-instantaneous manner. Moreover, the touchscreen display of the imaging device 50 shows the field of view of the camera, and therefore the ability to visualize the internal volume the waste container 26 is generally unimpeded, and further may be augmented with information of use to the user.

[0072] In another implementation, some of the steps of the method 100 depicted in FIG. 5A may be incorporated into other steps of the method 100 to provide a more streamlined process as shown in FIG. 5B. More specifically, compared to the method 100 shown in FIG. 5A, the method 100 of FIG. 5B has incorporated the ROI extraction step 134 into the marker detection step 114 and the image alignment step 116 into canonical coordinate mapping step(s) 124. Combining the ROI extraction step 134 into the marker detection step 114 can be accomplished by providing the image to the neural network which detects the locations of the markers 76 andoutputs the region of interest based on the detected markers 76 similar to the process described with reference to FIG. 15. Further, the image alignment step 116 can be combined into the canonical coordinate mapping step 124 by using the image alignment information, such as the comers 77 detected during the marker detection step 114, during the canonical coordinate mapping step 124. For example, the image alignment step 116 involves rotating the image such that the markers 76 are aligned substantially vertically. This rotation may be considered (e.g., mathematically) and carried out during the canonical coordinate mapping step 124. The optional steps 136 of the method 100 shown in FIG. 5A are also omitted in the method shown in FIG. 5B, and the step of retrieving or receiving the calibration data from the memory 88 has been incorporated into the pixel-to- volume mapping step 126.

[0073] Several implementations have been discussed in the foregoing description. However, the implementations discussed herein are not intended to be exhaustive or limit the invention to any particular form. Modifications and variations are possible in light of the above teachings and may be practiced otherwise than as specifically described. In one example, the methods disclosed herein may be performed on a waste container that is not disposed on a mobile chassis. In another example, the methods may be used to register images of other subjects (i.e., images which do not include a waste container) from the image coordinate system to a canonical coordinate system. In yet another example, the methods may be used to estimate a concentration and an amount of a non-blood component within the waste container 26, such as saline, ascites, bile, irrigating fluids, saliva, gastric fluid, mucus, pleural fluid, interstitial fluid, urine, fecal matter, or the like. In yet another example, the medical waste collection system 20 may communicate with other systems to form a fluid management ecosystem for generating a substantially comprehensive estimate of extracorporeal blood volume, total blood loss, patient euvolemia status, or the like.

Claims

CLAIMS1. A method of estimating blood loss within a waste container of a medical waste collection system, the method comprising the steps of: capturing, with an imaging device, an image of the waste container and waste material disposed therein; detecting a location of a fluid meniscus of the waste material based on the image; detecting a location of a foam meniscus of the waste material based on the image; determining a correction factor based on the location of the foam meniscus; estimating the blood loss based on the correction factor and the location of the fluid meniscus; and displaying, on a display, the estimated blood loss.

2. The method of claim 1, wherein the step of segmenting the image includes providing the image as an input to a neural network.

3. The method of claim 2, wherein the neural network is trained on an image dataset which includes images of a similar waste container filled with various amounts of fluid.

4. The method of claim 2, further comprising: classifying pixel values of the image as at least one of blood, non-blood, fluid, fluid surface, fluid meniscus, foam, foam surface, foam meniscus, and empty; and assigning class labels based on the classification of the pixel values.

5. The method of claim 4, wherein the neural network classifies the pixel values and assigns the class labels based on the classification of the pixel values.

6. The method of claim 4 or 5, wherein the location of the fluid meniscus is determined based on which of the pixel values of the image were assigned the fluid meniscus class label.

7. The method of any one of claims 4-6, wherein detecting the location of the fluid meniscus includes fitting a parabola to pixel values which were assigned the fluid meniscus class label.

8. The method of claim 7, wherein the location of the fluid meniscus is determined as a vertex of the parabola fit to the pixel values which were assigned the fluid meniscus class label.

9. The method of any one of claims 2-8, wherein the location of the foam meniscus is determined based on which of the pixel values of the image were assigned the foam meniscus class label.

10. The method of any one of claims 4-9, wherein the step of detecting the location of the foam meniscus comprises fitting a parabola to the pixel values which were assigned the foam meniscus class label.

11. The method of claim 10, wherein the location of the foam meniscus is determined as a vertex of the parabola fit to the pixel values which were assigned the class label of foam meniscus.

12. The method of any one of claims 1-11, wherein the step of detecting the location of the fluid meniscus of the waste material comprises detecting a pixel-based location of a fluid meniscus in the image frames.

13. The method of any one of claims 1-12, wherein the step of detecting the location of the foam meniscus of the waste material comprises detecting a pixel-based location of a foam meniscus in the image frames.

14. The method of any one of claims 1-13, wherein the correction factor is determined based on the location of the foam meniscus and the location of the fluid meniscus.

15. The method of any one of claims 1-14, further comprising determining a volume of foam based on the image, wherein the correction factor is determined based on the volume of the foam.

16. The method of claim 15, further comprising: determining a density of foam based on the image; and wherein the correction factor is determined based on the volume of the foam and the density of the foam.

17. The method of any one of claims 1-16, further comprising registering the image to a coordinate system.

18. The method of claim 17, wherein detecting at least one of the locations of the fluid meniscus and the foam meniscus includes determining the location of the fluid meniscus and / or the foam meniscus relative to an axis of the coordinate system.

19. The method of claim 18, wherein the correction factor is determined based on a distance between the locations of the fluid meniscus and the foam meniscus along the axis of the coordinate system.

20. The method of any one of claims 1-19, wherein the step of estimating the blood loss comprises: estimating a pre-correction blood loss based on the location of the fluid meniscus, and estimating the blood loss by adding the correction factor to the pre-correction blood loss.

21. A method of estimating blood loss within a waste container of a medical waste collection system, the method comprising the steps of: capturing, with an imaging device, an image of the waste container and waste material disposed therein; detecting at least one reference marker affixed to the waste container; providing the image as an input to a neural network;generating, from the neural network, a heatmap corresponding to a probabilistic location of at least one corner of the at least one reference marker, wherein a region of interest for processing the image is identified based on the probabilistic location of the at least one corner of the at least one reference marker; estimating blood loss by processing the region of interest of the image; and displaying, on a display, the estimated blood loss.

22. The method of claim 21, further comprising applying a non-maximum suppression algorithm to the heatmap to generate a filtered heatmap.

23. The method of claim 22, further comprising identifying the probabilistic location of at least one corner of the at least one reference marker according to points representing at least one local maxima within the filtered heatmap.

24. The method of claim 23, further comprising applying a clustering algorithm to the filtered heatmap to generate a clustered heatmap.

25. The method of claim 24, further comprising identifying the probabilistic location of at least one corner of the at least one reference marker according to a location of a representative point within the clustered heatmap.

26. The method of claim 24 or 25, wherein the clustered heatmap includes at least one point cluster and the representative point is determined as a local maximum within the at least one point cluster.

27. The method of any one of claims 24-26, wherein the clustered heatmap includes at least one point cluster and the representative point is determined as a weighted average of each point within the at least one point cluster.

28. The method of any one of claims 24-27, further comprising: determining an actual number of point clusters within the clustered heatmap;comparing the actual number of point clusters against an expected number of point clusters; assigning a confidence score to each point cluster of the at least one point cluster; if the actual number of point clusters is greater than the expected number of point clusters, discarding at least one point cluster based on the assigned confidence scores such that the actual number of point clusters equals the expected number of point clusters; and determining the probabilistic location of at least one comer of the at least one reference marker based on a set of remaining point clusters.

29. The method of any one of claims 24-28, wherein the step of discarding at least one point cluster of the at least one point cluster based on the assigned confidence scores includes discarding at least one point cluster which was assigned a confidence score below a threshold confidence score.

30. The method of claim 28, wherein the region of interest is identified as a cropped section of the image which includes the at least one reference marker, and, optionally, two reference markers.

31. The method of any one of claims 21-30, wherein the at least one reference marker includes at least one QR code, and / or at least one AruCo marker.

32. A method of estimating blood loss within a waste container of a medical waste collection system, the method comprising the steps of: capturing, with an imaging device, an image of the waste container and waste material disposed therein; detecting at least one reference marker affixed to the waste container; and transforming the image of the waste container from an image coordinate system to a canonical coordinate system by: providing the image as an input to a neural network;generating, with the neural network, a processed image containing a heatmap corresponding to a probabilistic location of at least one comer of the at least one reference marker; aligning the image of the waste container from the image coordinate system to the canonical coordinate system based on the probabilistic location of at least one comer of the at least one reference marker, and a reference point of the image coordinate system of the imaging device; and estimating the blood loss by analyzing the aligned image; and displaying, on a display, the estimated blood loss.

Images