Estimating blood loss in waste containers of medical waste collection systems.

Abstract

A method for estimating blood loss in a waste container of a medical waste collection system, in which an imaging device is supported using a device cradle, includes analyzing an image, such as an image frame of a video feed, to determine fluid volume and blood components within the waste material. The image can be registered based on fiducial markers, and the method includes segmenting the image to determine blood, meniscus, and non-blood classes. A fluid meniscus of the waste material can be identified and mapped to a canonical coordinate space for conversion to pixel volume. Calibration data associated with the fiducial markers can be used to do so. At least one region of interest associated with an insert placed within the waste container is analyzed. Estimated blood loss based on the determined fluid volume and blood components can be displayed in real time.

Description

[Technical Field]

[0001] [Priority Claim] This application claims priority to and the full benefit of U.S. Provisional Patent Application No. 63 / 353,208, filed June 17, 2022, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] A by-product of surgical procedures is the generation of liquid, semi-solid, and / or solid waste materials. Medical waste may include blood, interstitial fluid, mucus, irrigation fluid, etc. Medical waste may be removed from the surgical site through suction tubing under the influence of vacuum from a vacuum source and collected in a waste container.

[0003] Estimating blood loss during surgery can be used to monitor a patient's health during surgery. Advances in imaging and computing technologies have provided for estimating blood loss by capturing images of a fluid-containing medium, such as a freestanding container. One such system is sold under the trade name Triton by Gauss Surgical, Inc. (Menlo Park, California) and is disclosed in commonly owned U.S. Patent No. 9,773,320, issued September 26, 2017, the entire contents of which are incorporated herein by reference. The freestanding container is placed in-line between a suction tube and a vacuum source. When it is desired to know the amount of blood in the freestanding container, a user instructs a handheld imaging device to capture an image of the area of the container where the insert is located and manually enters the volume of fluid in the container. While the Triton system has achieved significant gains in estimating blood loss, it would be beneficial if the system could estimate and display blood loss in real time, requiring less user interaction.

[0004] The vacuum source can be a facility-integrated vacuum source or a medical waste collection system vacuum source. One exemplary medical waste collection system is sold under the trade name Neptune by Stryker Corporation (Kalamazoo, Michigan), in which at least one waste container is disposed on a mobile housing. Because the Neptune system includes an integrated waste container, it is further desirable to eliminate the need for a freestanding container, which may otherwise be redundant. However, doing so may be related to the technical challenges addressed herein. For example, while a freestanding container may be disposable after a single use, the waste container(s) of a medical waste collection system are capital components that are reused over multiple procedures. Contamination of the transparent sidewall of the waste container(s) can affect the consistency of image quality and, therefore, the accuracy of estimated blood loss. As another example, the Neptune system includes a subsystem that determines the volume of fluid in the waste container(s), but in certain instances, the imager may be unable to access the volume data (e.g., if the imager and the Neptune system are not in electronic communication). In another example, the fluid volume determined by the Neptune system may not quantify the pre-fill volume, i.e., the volume of fluid in the canister before the start of treatment. To be sufficiently accurate, image-based determination of fluid volume may need to account for camera positioning, fluid foaming, and mechanical dispersion of medical waste collection system subcomponents, among other considerations, such as lighting or flash variations and glare. Summary of the Invention

[0005] The present disclosure relates to a method for estimating blood constituents in a waste container of a medical waste collection system, preferably in real time as the waste material is drawn into the waste container under the influence of suction. The blood constituents can be hemoglobin concentrations to estimate blood loss (eBL) and indicate the patient's estimated blood loss (and / or hemoglobin). The present disclosure also relates to an apparatus for performing the methods disclosed herein. The method can be instructions stored on a non-transitory computer-readable medium configured to be executed by one or more processors. The method can be implemented by a machine learning (ML) model, a trained neural network, or a combination thereof. The neural network can be trained on an image dataset or combination of any number of factors, including, but not limited to, the position of the camera relative to the waste container, the position of the flash reflected on the surface of the waste container, color component values of one or more regions of interest (i.e., the imaged region(s) of the waste material contained between the insert and the inner surface of the waste container), color component values of at least two fiducial markers affixed to the outer surface of the waste container of known color profiles, or phase characteristics of the waste material (e.g., blood, non-blood, fluid meniscus, bubbles, hemolysis level, etc.). The image dataset can be broad and diverse to train the neural network for at least substantially the entire range of scenarios expected within the expected parameters of the medical waste collection system. The image dataset can also deviate from the expected parameters, such as by one, two, or three or more standard deviations therefrom. Thus, during operation, images or image frames of a video feed being analyzed by the neural network on one or more processors can provide a real-time estimate of blood loss in the waste material, particularly as the waste material is drawn into the waste container under the influence of a vacuum, often in a turbulent and unpredictable manner.

[0006] An imaging device including a camera is configured to capture images of the waste container. The captured images may be analyzed by one or more processors, or image frames from a video feed being captured by the camera may be analyzed by the processor(s). As a result, the eBL is continuously updated and streamed onto the display. The device cradle may be removably coupled to the housing of the medical waste collection system or may be rigidly secured to the housing. The device cradle may include a shield and at least one arm extending from the shield. One of the arms may include a hinge. A latch or other suitable locking mechanism may be coupled to the shield. An opening provides communication between the rear recess and the front recess. The opening is defined in the rear recess to precisely position the camera and light source relative to the waste container. The front recess is sized to receive and support the imaging device.

[0007] A method for estimating the volume of blood in a waste container may include operating in a standby mode or low power mode. The low power mode may include using a camera to capture image frames of a video feed at a low frame rate with the flash turned off or at a low level. The image frames may be pre-processed, and an activity recognition algorithm may analyze the image frames to detect a pixel-based position of the fluid meniscus in each of the images. The activity recognition algorithm may determine whether the pixel-based position of the fluid meniscus has changed by an amount greater than a predetermined threshold within a predetermined time period. The activity recognition algorithm may compare the pixel-based position of the fluid meniscus between successive image frames of the video feed and compare the change to a predetermined threshold. If the change remains below the predetermined threshold, the system remains in the low power mode. If the change in the fluid meniscus exceeds the predetermined threshold, the method may include activating a light source or increasing the light source level of the imaging device and increasing the feed rate of the video feed.

[0008] In a first variation, the method may include capturing one or more single or multiple exposure photographs for subsequent processing and analysis. In another variation, the method may include analyzing at least one of the image frames of the video feed, preferably a series of multiple image frames, according to an algorithm. This step may be performed while the waste material is being drawn into the waste container (e.g., while the vacuum source is on), or while the vacuum is off, or a combination thereof.

[0009] The method includes detecting at least one fiducial marker(s), such as a quick response (QR) code(s). The fiducial marker(s) can be affixed to an exterior surface of the waste container. The QR code(s) can be associated in memory with calibration-based data associated with the waste container. The camera can be activated in a calibration mode, and at least one calibration operation is performed, in which at least one known volume of fluid is aspirated into the waste container. The processor performs image-based volume determination, as described below, including ascertaining a vertical y-axis value of the fluid meniscus, and associates each of the y-axis value and the known volume(s) of fluid with a unique code in the QR code(s). The calibration data is stored in memory.

[0010] The method includes aligning the image. The aligning the image can be based on detecting fiducial markers. The processor is configured to detect a QR code and rotate the image accordingly. Additionally or alternatively, the camera can be configured to detect landmarks or fiducials on the waste container or enclosure and align the image accordingly. The image frame can be segmented to determine the center of mass of the waste container. The camera can be calibrated to determine the center of the optical sensor. The camera calibration can be performed by determining the orientation of the imaging device. The image coordinates can be mapped to canonical coordinates. One or more of the QR alignment blocks can be used for transformation to two-dimensional canonical coordinates and then to three-dimensional canonical coordinates. A cropped image frame in canonical coordinate space can include at least two fiducial markers and a portion of the waste container between them. The width of the cropped, mapped image frame is approximately equal to the width of the at least two fiducial markers. Cropping of the adjusted image is optional. The center of mass pixel point and the center pixel point of the optical sensor can each be transformed into a three-dimensional point in the canonical space of the waste container. A mathematical correction is determined from the orientation of the waste container and the imaging device. A corrected volume is determined based on the image-based volume determination and the mathematical correction.

[0011] The captured image or image frame of the video feed, including the waste material contained therein, can be segmented using a neural network. The neural network can characterize pixels as blood, fluid meniscus, non-blood, and optionally, foam, foam, and / or fluid surface. Each pixel is assigned a value from the neural network and then assigned a class label based on that value. The location of the fluid meniscus is determined based on the segmentation mask. This step can include fitting a parabola to pixels with a class label of meniscus. In an example where the fluid meniscus is below the camera, the parabola will have a lower apex and open upward. If the fluid meniscus is above the camera, the parabola will have an upper apex and open downward. The location of the fluid meniscus is the minimum or maximum value of the y-axis of the apex of the parabolic curve. Any other polynomial function may be used to approximate the meniscus line, and the minimum or maximum point may be found accordingly to determine the location of the fluid.

[0012] The method then includes performing pixel-to-volume mapping to determine the volume of fluid in the waste container, which may include obtaining or receiving calibration data from memory, the image-based volume determination being the y-axis value of the fluid meniscus in canonical coordinates adjusted by the calibration data.

[0013] In certain embodiments, the method includes extracting a region of interest (ROI) from the image. The region of interest may be a first imaging surface and a second imaging surface of the insert. Regions of the image associated with imaging features of the insert are analyzed to quantify the concentration of blood components in the waste material. Further optional steps may include extracting palette colors from the fiducial marker(s), noise removal, extracting features (e.g., red-green-blue (RGB) values from the insert), and performing light normalization to account for variations in light intensity. The light normalization may include training a neural network to predict the color profile of a lower one of the fiducial markers based on the color profile of one or more upper ones of the fiducial markers, and applying color correction based on the predicted color profile of the lower fiducial marker. Alternatively, certain aspects of the method are integrated for analysis by a neural network trained on the image dataset of such aspects.

[0014] The estimated blood loss can be displayed on one or more displays, for example, on a touchscreen display of the imaging device. The eBL can be displayed and updated at desired intervals or in real time. In particular, in video mode, the camera can capture a video feed and the blood constituent and fluid volumes can be repeatedly determined in a near-instantaneous manner. The camera's touchscreen display can display the camera's field of view and can be further augmented with information useful to the user.

[0015] Therefore, according to one aspect of the present disclosure, a method for estimating blood loss in waste material within a waste container of a medical waste collection system is provided. An imaging device captures a video feed of the waste container and waste material placed therein as a vacuum source of the medical waste collection system draws the waste material into the waste container. One or more processors analyze the image frames of the video feed to determine a volume of the waste material within the waste container. The one or more processors analyze the image frames to determine concentrations of blood components within the waste material. The one or more processors estimate blood loss based on the determined volume and the determined concentrations of the blood components. The estimated blood loss is then displayed on a display.

[0016] According to another aspect of the present disclosure, a method for estimating blood loss in waste material within a waste container of a medical waste collection system is provided. An imaging device captures a video feed of the waste container and the waste material placed therein as a vacuum source of the medical waste collection system draws the waste material into the waste container. Image frames from the video feed are analyzed by one or more processors to determine the concentration of blood components within the waste material. Analysis of the image frames is facilitated by a neural network trained on at least one of image data sets: relative positioning of the imaging device's camera with respect to the waste container, location of a flash reflected on a surface of the waste container, color component values associated with at least two fiducial markers affixed to the surface of the waste container, and color component values associated with the imaging surface of an insert placed within the waste container. Blood loss is estimated based on the volume and determined concentration of the blood components and displayed on a display.

[0017] In certain embodiments, the neural network is further trained on an image dataset of a region of interest associated with at least two imaging surfaces of the insert, each of which provides a different color component value for the waste material between the insert and the waste container. Alternatively, the neural network is further trained on an image dataset of a region of interest associated with an imaging surface of the insert, which provides a color gradient for the waste material between the insert and the waste container. The neural network can be further trained on image datasets of at least one of various volumes of waste material and various phase characteristics of the waste material. Image frames can be mapped from the image coordinate space to a canonical coordinate space and provided to the neural network.

[0018] According to yet another aspect of the present disclosure, a method for estimating blood loss in waste material within a waste container of a medical waste collection system is provided. An image of the waste container and the waste material contained therein is captured by an imaging device. One or more processors are configured to process the image, and the processing includes processing the image using the one or more processors, where processing the image further includes determining a position of a fluid meniscus, registering the image, mapping the image from an image coordinate space to a canonical coordinate space, and converting a y-axis value of the image mapped in the canonical coordinates to a determined volume. The method includes detecting at least two fiducial markers affixed to the waste container using the one or more processors, the fiducial markers including position data. Based on the position data, a region of interest is extracted from the raw image. The image is analyzed to determine concentrations of blood components within the waste material. Based on the determined volume and the determined concentrations of the blood components using the one or more processors, blood loss is estimated. The estimated blood loss is displayed on a display.

[0019] In certain embodiments, the imaging device captures the video feed at a first frame rate. One or more processors determine the rate at which the volume of waste material is increasing within the waste container. The one or more processors can detect a pixel-based position of the fluid meniscus in the image frames and determine whether the pixel-based position of the fluid meniscus changes by an amount greater than a predetermined threshold within a predetermined time period. The image frames can be downsampled prior to detecting the pixel-based position of the fluid meniscus. The imaging device is configured to capture the video feed at a second frame rate greater than the first frame rate if the rate is greater than the predetermined threshold. If the rate is greater than the predetermined threshold, a flash can be activated or its level can be increased. This approach helps reduce power requirements for continuously operating the device.

[0020] In certain embodiments, the one or more processors can detect at least two fiducial markers affixed to the waste container, and the one or more processors can align the image or image frame based on the relative positioning of the at least two fiducial markers, where the at least two fiducial markers are QR codes arranged in a generally vertical configuration, and aligning the image frame further includes aligning the QR codes so that they are precisely vertical.

[0021] The present disclosure also provides a method for correcting tilt of a waste container in a medical waste collection system and / or tilt of an imaging device supported in a device cradle. Images or video feeds of a waste container and waste material placed therein are captured by the imaging device. One or more processors receive the image frames. The one or more processors detect the positions of at least two fiducial markers in the image frames and determine an orientation of the waste container based on the positions of the at least two fiducial markers. The orientation of the imaging device is determined, for example, based on an internal gyroscope, and the one or more processors determine a mathematical correction based on the orientation of the waste container and the orientation of the imaging device. The mathematical correction is applied to determine a corrected volume of waste material placed within the waste container.

[0022] The at least two fiducial markers may be ArUco codes, and the image registration may compensate for tilt of the waste container by determining the orientation of the ArUco codes. The ArUco codes may be located on a registration frame affixed to the waste container. In certain embodiments, the image frames are analyzed to determine the concentrations of blood components within the waste material. One or more processors estimate blood loss based on the corrected volume and the determined concentrations of the blood components. The estimated blood loss may be displayed on a display.

[0023] In certain embodiments, the one or more processors can segment the image or image frame by characterizing each pixel value in the image frame as blood, meniscus, non-blood, bubbles on the fluid surface, or other phase characteristic of the waste material. The class label is an assigned class label based on the pixel value. The position of the fluid meniscus can be determined based on the meniscus class label. The one or more processors can fit a parabola to pixels having the meniscus class label, and determining the position of the fluid meniscus further includes determining a y-axis value of a maximum or minimum of the parabola. The fluid meniscus can be below the level of the imaging device, such that the parabola opens upward, and the position of the fluid meniscus can be the y-axis value of the parabola's minimum. The fluid meniscus can be above the level of the imaging device, such that the parabola opens downward, and the position of the fluid meniscus can be the y-axis value of the parabola's maximum. The one or more processors can receive container-specific calibration data and map the y-axis values to the data. The one or more processors may convert the y-axis values to volume values.

[0024] In certain embodiments, an image or image frame can be mapped from an image coordinate space to a canonical coordinate space. A homography of the orientation markers (e.g., four orientation markers or fiducial markers) can be generated, and the image coordinates can be transformed into 2D canonical coordinates. The orientation markers and fiducial markers can be the same or different. The 2D canonical coordinates can be transformed into 3D canonical coordinates.

[0025] According to another aspect of the present disclosure, there is provided a method for calibrating mechanical dispersion in a waste container of a medical waste collection system, the method including: supporting an imaging device in a device cradle; capturing images of the waste container and waste material placed therein in a calibration mode in which a vacuum source of the medical waste collection system draws at least one known volume of waste material into the waste container; analyzing image frames of the video feed by one or more processors to determine a y-axis value of a fluid meniscus of the waste material within the waste container; storing calibration data in a memory, the data relating at least one known value to the y-axis value of the fluid meniscus;

[0026] In certain embodiments, a fiducial marker affixed to the waste container can be detected by one or more processors. The fiducial marker includes optically readable data or a code. The one or more processors associate calibration data with the optically readable data of the fiducial marker. Image frames of the video feed can be analyzed by the one or more processors to determine a volume of the waste material within the waste container. The volume is displayed on a display, and a user interface is configured to receive confirmation of the volume of the waste material.

[0027] According to yet another aspect of the present disclosure, a method for estimating blood loss in waste material within a waste container of a medical waste collection system is provided, the method including: supporting an imaging device in a device cradle; capturing a video feed of the waste container and the waste material contained therein while a vacuum source of the medical waste collection system draws the waste material into the waste container; one or more processors segment image frames of the video feed by characterizing each pixel value of the image frame as blood, a meniscus, non-blood, and optionally, a bubble or fluid surface; assigning class labels to the pixels based on the pixel values and determining a position of the fluid meniscus based on the meniscus class label; mapping the image frames from an image coordinate space to a canonical coordinate space by the one or more processors; and determining a volume of the waste material within the waste container based on the position of the fluid meniscus.

[0028] In certain embodiments, the one or more processors detect at least two fiducial markers affixed to the waste container. The mapped image frame is cropped by the one or more processors. The cropped mapped image frame can be in a canonical coordinate space and includes the at least two fiducial markers and a portion of the waste container between them. The width of the cropped mapped image frame can be approximately equal to the width of the at least two fiducial markers. The one or more processors can align the image frame based on the relative positioning of the at least two fiducial markers.

[0029] According to yet another aspect of the present disclosure, there is provided a device cradle for supporting an imaging device for capturing images of waste containers in a medical waste collection system. The device cradle includes a shield coupled to a front casing. The shield defines a rear recess configured to cover a window, a front recess, and an opening providing fluid communication between the rear recess and the front recess. The front recess is sized to receive the imaging device, and the opening is positioned relative to the front recess to be aligned with a camera and flash of the imaging device.

[0030] In certain embodiments, at least one arm couples the shield to the front casing. The device cradle includes a hinge coupling the at least one arm to the shield and configured to allow the shield to be pivoted to a configuration in which the window is visible with the imaging device placed therein. The front housing can be movably coupled to the shield. The waste container is configured so that at least substantially the entire waste container is visible within the field of view of the camera of the imaging device. [Brief explanation of the drawings]

[0031] [Figure 1] 1 illustrates a medical waste collection system configured to aspirate medical waste through a suction tube and a manifold for collection in a waste container, wherein a device cradle is coupled to a housing of the medical waste collection system and positioned to removably receive an imaging device configured to capture images of the waste container. [Figure 2] FIG. 2 is a perspective view of the device cradle in a first configuration. [Figure 3] FIG. 10 is a perspective view of the device cradle in a second configuration. [Figure 4] FIG. 10 is an elevation view of another embodiment of the device cradle. [Figure 5] FIG. 1 is a plan view of a waste container with a schematic representation of the field of view of an imaging device. [Figure 6] 1 is an elevational view of a waste container with a schematic representation of the field of view of an imaging device. [Figure 7] FIG. 10 shows a representation of an imaging device capturing a video feed of a waste container while supported in a device cradle. [Figure 8] 1 is a perspective view of a waste container with an insert placed therein, with an alignment frame and fiducial markers coupled to the exterior of the waste container. [Figure 9A] FIG. 1 illustrates how one or more images are captured and analyzed by one or more processors to estimate blood loss. [Figure 9B] FIG. 1 illustrates how image frames from a video feed are analyzed by one or more processors to estimate blood loss. [Figure 10] FIG. 1 shows a waste container with fluid therein with machine video output overlaid on fiducial markers for alignment of the image(s). [Figure 11] FIG. 10 illustrates a waste container with a processor identifying markers and external structures of the housing of the medical waste collection system for alignment of the image(s). [Figure 12A] FIG. 10 shows a representation of a process that uses internal features of a waste container for alignment of an image or images. [Figure 12B] FIG. 10 shows a representation of a process that uses internal features of a waste container for alignment of an image or images. [Figure 12C] FIG. 10 shows a representation of a process that uses internal features of a waste container for alignment of an image or images. [Figure 13] FIG. 10 illustrates a method for compensating for tilt of a waste container. [Figure 14A] 10A-10C show alternative embodiments of fiducial markers for compensating for tilt of the waste container. [Figure 14B] 10A-10C show alternative embodiments of fiducial markers for compensating for tilt of the waste container. [Figure 15A]1A-1C show representations of certain steps of the method applied to a first image frame processed during collection of waste material in a waste container. [Figure 15B] 1A-1C show representations of certain steps of the method applied to a first image frame processed during collection of waste material in a waste container. [Figure 15C] 1A-1C show representations of certain steps of the method applied to a first image frame processed during collection of waste material in a waste container. [Figure 15D] 1A-1C show representations of certain steps of the method applied to a first image frame processed during collection of waste material in a waste container. [Figure 15E] 1A-1C show representations of certain steps of the method applied to a first image frame processed during collection of waste material in a waste container. [Figure 15F] 1A-1C show representations of certain steps of the method applied to a first image frame processed during collection of waste material in a waste container. [Figure 16A] 10A-10C show representations of certain steps of the method applied to a second image frame processed during collection of waste material in a waste container. [Figure 16B] 10A-10C show representations of certain steps of the method applied to a second image frame processed during collection of waste material in a waste container. [Figure 16C] 10A-10C show representations of certain steps of the method applied to a second image frame processed during collection of waste material in a waste container. [Figure 16D] 10A-10C show representations of certain steps of the method applied to a second image frame processed during collection of waste material in a waste container. [Figure 16E] 10A-10C show representations of certain steps of the method applied to a second image frame processed during collection of waste material in a waste container. [Figure 16F]10A-10C show representations of certain steps of the method applied to a second image frame processed during collection of waste material in a waste container. [Figure 17A] 10A and 10B show representations of certain steps of the method applied to a third image frame processed during collection of waste material in a waste container. [Figure 17B] 10A and 10B show representations of certain steps of the method applied to a third image frame processed during collection of waste material in a waste container. [Figure 17C] 10A and 10B show representations of certain steps of the method applied to a third image frame processed during collection of waste material in a waste container. [Figure 17D] 10A and 10B show representations of certain steps of the method applied to a third image frame processed during collection of waste material in a waste container. [Figure 17E] 10A and 10B show representations of certain steps of the method applied to a third image frame processed during collection of waste material in a waste container. [Figure 17F] 10A and 10B show representations of certain steps of the method applied to a third image frame processed during collection of waste material in a waste container. [Figure 18A] 10 shows a representation of certain steps of the method applied to a fourth image frame processed during collection of waste material in a waste container. [Figure 18B] 10 shows a representation of certain steps of the method applied to a fourth image frame processed during collection of waste material in a waste container. [Figure 18C] 10 shows a representation of certain steps of the method applied to a fourth image frame processed during collection of waste material in a waste container. [Figure 18D] 10 shows a representation of certain steps of the method applied to a fourth image frame processed during collection of waste material in a waste container. [Figure 18E]10 shows a representation of certain steps of the method applied to a fourth image frame processed during collection of waste material in a waste container. [Figure 18F] 10 shows a representation of certain steps of the method applied to a fourth image frame processed during collection of waste material in a waste container. [Figure 19] 1 is a partial view of a waste container showing machine video output overlaid on a fiducial marker and associated with a region of interest; [Figure 20] FIG. 1 illustrates another embodiment of a method for estimating blood loss, where the method can be implemented in a machine learning environment. DETAILED DESCRIPTION OF THE INVENTION

[0032] FIG. 1 illustrates a medical waste collection system 20 for collecting waste materials generated during medical procedures. The medical waste collection system 20 includes a housing 22 and wheels 24 for moving the housing 22 within a medical facility. At least one waste container 26 is supported on the housing 22 and defines a waste space for receiving and collecting waste materials. In embodiments with more than one waste container, an upper waste container 26 can be positioned above a lower waste container 26, and a valve (not shown) can facilitate transferring waste materials from the upper waste container 26 to the lower waste container 26. A vacuum source 30 is supported on the housing 22 and configured to draw suction on the waste container(s) 26 through one or more internal lines. The vacuum source 30 can include a vacuum pump and a vacuum regulator configured to adjust the level of suction drawn on the waste container(s). Suitable structures and operations of several subsystems of medical waste collection system 20 are disclosed in commonly owned U.S. Patent No. 7,621,898, issued November 24, 2009, U.S. Patent No. 10,105,470, issued October 23, 2018, and U.S. Patent No. 11,160,909, issued November 2, 2021, the entire contents of which are incorporated herein by reference. While the ensuing discussion is made with reference to an upper waste container, it should be understood that the scope of this disclosure can be extended to a lower waste container as an alternative or concurrently.

[0033] The medical waste collection system 20 includes at least one receptacle 28 supported on the housing 22. The receptacle 28 defines an opening sized to removably receive at least a portion of a manifold 34. A suction path can be established from a suction tube 36, through the manifold 34, which is removably inserted into the receptacle 28, to the waste container 26. In other words, a vacuum generated by the vacuum source 30 is drawn into the suction tube 36, and waste material is drawn from the surgical site through the suction tube 36, the manifold 34, and the receptacle 28 to be collected in the waste container 26. The manifold 34 can be a disposable component; an exemplary embodiment of the receptacle 28 and manifold 34 is disclosed in commonly owned U.S. Patent No. 10,471,188, issued November 12, 2019, the entire contents of which are incorporated herein by reference.

[0034] The medical waste collection system 20 includes a fluid measurement subsystem 38, a cleaning subsystem 40, and a container lamp or backlight. An exemplary embodiment of the fluid measurement subsystem 38 is disclosed in the aforementioned U.S. Patent No. 7,621,898, in which a float element 79 is movably disposed along a sensor rod 80 (see also FIG. 8 ). A controller 42, in electronic communication with the fluid measurement subsystem 38, is configured to determine the volume of the waste fluid in the waste container 26 based on signals received from the fluid measurement subsystem 38 indicating the fluid level. The cleaning subsystem 40 may include a sprayer, as disclosed in the aforementioned U.S. Patent No. 10,105,470, rotatably disposed within the waste container 26 and configured to direct pressurized liquid against the interior surface of the waste container 26. Finally, a container backlight is configured to illuminate the interior of the waste container 26. The container backlight may be activated based on input to a user interface 52 or other device in communication with the controller 42.

[0035] The housing 22 includes a front casing 46 defining at least one cutout or window 48 for exposing a portion of the waste container 26. The waste container 26 may be formed of a transparent material through which a user can visually observe the waste material collected within the waste container 26 and, if necessary, visually estimate the volume of the waste material collected therein using volumetric markings disposed on the exterior surface of the waste container 26 (see FIGS. 11 and 14). The optically transparent waste container 26 also allows the waste material collected therein to be captured by a camera 50. The camera 50 is also referred to herein as an imaging device, which includes the camera 50, a user interface (e.g., a touchscreen display), and optionally, a flash or light source. Images from the camera 50 may be transmitted to and processed by one or more processors 44 (hereinafter referred to in the singular) to determine blood constituents within the waste material. The blood constituents may be blood concentrations (e.g., hemoglobin concentrations) within the waste material. More specifically, the optical properties of the waste material can be analyzed and processed to determine blood constituents, as disclosed in commonly owned U.S. Patent No. 8,792,693, issued July 29, 2014, the entire contents of which are incorporated herein by reference. The volume of blood within the waste material (i.e., blood loss in this patent) can be estimated from the determined blood constituents and the volume of the waste material.

[0036] 2 and 3 , a device cradle 54 is removably coupled to or rigidly secured to the housing 22. The device cradle 54 precisely positions the camera 50 relative to the waste container 26 to provide continuous image capture (e.g., a video feed) of at least substantially the entire waste container 26. The video feed results in continuous data from which the volume of fluid and blood components therein can be determined in real time. As a result, the estimated blood loss (eBL) can be continuously updated and streamed onto a display (e.g., a touchscreen display of the imaging device 50, the user interface 52 of the medical waste collection system 20, and / or another display terminal). Such benefits are not easily achieved with handheld imaging devices, which require the user to manually hold the camera during image capture and manually input the estimated volume based on visual observation. Furthermore, image-based volume determination eliminates the need for the imaging device 50 to communicate data with the controller 42 of the medical waste collection system 20. Additionally, precise positioning can be designed to reduce glare and eliminate illumination variations or aberrations to improve the accuracy of algorithmic decisions.

[0037] The device cradle 54 can be removably coupled or rigidly secured to the front casing 46 or other suitable surface or internal structure of the housing 22. For example, a complementary coupling (e.g., a flange-in-slot, detents, etc.) can facilitate removably coupling the device cradle 54 to the housing 22. The device cradle 54 can include a shield 56 configured to be disposed over the window 48 defined by the front casing 46. At least one arm 58 can extend from the shield 56 and be coupled to the housing 22. In certain embodiments, one of the arms 58 can include a hinge 60 configured to allow the device cradle 54 to be moved between a first configuration in which the shield 56 covers the window 48 and a second configuration in which the shield 56 does not cover the window 48. For example, in certain procedures, it may be desirable to forgo the use of an eBL and / or otherwise view the contents of the waste container 26. A latch or other suitable locking mechanism coupled to shield 56 can be actuated, and shield 56 can be pivoted about hinge 60 to a second configuration, as shown in Figure 3. For a subsequent procedure, or as otherwise desired, the user can pivot shield 56 about hinge 60 to return device cradle 54 to the first configuration.

[0038] The device cradle 54 may include a rear recess 62, a front recess 66, and an opening 64 providing communication between the rear recess 62 and the front recess 66. The rear recess 62 may be sized and shaped to mate with a rim or lip disposed within the window 48 of the front casing 46 to limit or prevent the intrusion of ambient light. Alternatively, a resilient flange may extend from the rear of the shield 56 and engage the front casing 46 to achieve this. The opening 64 is defined at a location within the rear recess 62 to precisely position the camera 50 and light source (e.g., the imager's flash) relative to the waste receptacle 26, in a manner to be further described. The front recess 68 is sized to receive and support the imager 50. The dimensions of the front recess 68 may vary based on the type of imager 50 itself. In other words, the imaging device 50 can be a smartphone, tablet, or custom-designed device, and variations of the device cradle 54 can be made to fit specific models of devices (e.g., Apple iPhone®, Samsung Galaxy®, Google Pixel®, etc.). The illustrated embodiment of the device cradle 54 can be for use with an Apple iPhone® 13 Pro Max running the iOS® operating system. In such a configuration, the device cradle 54 is dimensioned so that the camera and flash (located in the top corners of the Apple iPhone® 13 Pro Max) are correctly located relative to the waste receptacle 26. In such a configuration, the front recess 68 is positioned lower than the rear recess 62. Furthermore, the front recess 68 can be sized to support the imaging device 50 in a protective case. In certain embodiments, the front housing 70 can be movably coupled to the shield 56 using a second hinge and latch mechanism (not shown) that pivotally couples the front housing 70 to the shield 56.

[0039] FIG. 4 shows another embodiment of the device cradle 54 in which a geometric shape associated with the window 48 in the front casing 46 supports the device cradle 54. For example, there may be a small gap or slot between the front casing 46 and the outer surface of the waste container 26 around the perimeter of the window 48. Certain components of the device cradle 54 may be integrated with the waste container 26 such that the device cradle 54 is located inside the front casing 46 and outside the waste container 26. The opening 64 is sized and positioned to align with the camera 50 and flash. The lip 72 of the device cradle 54 may be sized and shaped to receive the base of the camera 50 and position the camera 50 against the shield 56 with the camera and flash aligned with the opening 64. Other means of supporting the camera 50 are also contemplated, such as magnets, clips, and straps. The device cradle 54 may be positioned on the housing 22 to avoid obscuring regulatory labels or other components of the medical waste collection system 20 requiring accessibility.

[0040] As previously mentioned, the device cradle 54 supports the camera 50 so that the waste container 26 is within the camera's field of view. In certain embodiments, the entire waste container 26 is within the field of view. Referring now to FIGS. 5 and 6, a representation of the device cradle 54 is shown having an opening 64 located to approximate the location of the camera 50. The device cradle 54 can be sized so that the camera's field of view spans the entire width of the waste container 26 and, optionally, to provide a desired spacing between the flash of the imaging device 50 and the surface of the waste container 26. The designed spacing can be adjusted to reduce glare while still properly illuminating the waste container 26 with the flash of the imaging device 50. Furthermore, the camera's field of view can extend from below the bottom of the waste container 26 to above the top of the waste container 26 or its predetermined fill level. FIG. 6 shows the field of view of the camera 50 with up to approximately 4000 mL of fluid in the waste container 26. The distance the camera 50 is spaced from the surface of the waste container 26 can be based on the aspect ratio of the camera 50 (e.g., 4:3, 16:9, etc.). In one example, accurate positioning includes the camera 50 being spaced from the surface of the waste container 26 by a distance in the range of approximately 30 millimeters (mm) to 70 mm, more particularly by a distance in the range of approximately 40 mm to 60 mm, and even more particularly by a distance of approximately 50 mm. In one particular example, the horizontal field of view angle can be in the range of approximately 80 degrees to 90 degrees, the vertical field of view angle can be in the range of approximately 100 degrees to 105 degrees, and the corresponding lens field of view angle can be in the range of approximately 110 degrees to 120 degrees. Additional optical components, such as a fisheye lens and mirrors, can be used on the device cradle 54 to expand the field of view of the camera 50. With the camera and flash facing the waste container 26, the touchscreen display of the imaging device 50 remains visible and operable, including displaying the camera's field of view or an expanded field of view, as shown in Figure 7. It will be further appreciated that this configuration facilitates the camera 50 being removable from and replaceable within the device cradle 54 in a quick and intuitive manner.

[0041] Referring now to FIG. 8 , an insert 74 is placed within the waste container 26 so as to be within the field of view of the camera 50. The insert 74 includes several geometric shapes, at least one of which is an imaging surface 75 that is spaced from the interior surface of the waste container 26 to define a gap of a known, fixed distance. This gap allows a thin layer of fluid exhibiting at least a substantially uniform color region below the color intensity that would cause signal saturation to be positioned between the insert 74 and the interior surface of the waste container 26. Exemplary embodiments of the insert 74 are disclosed in the aforementioned U.S. Pat. No. 9,773,320 and commonly owned International Patent Application No. PCT / US2013 / 015437, filed March 17, 2023, the entire contents of which are incorporated herein by reference. The insert 74 can be attached to a container lid 82 of the waste container 26.

[0042] The imaging surface 75 can include a first imaging surface 75a and a second imaging surface 75b positioned laterally relative to one another in a side-by-side configuration (i.e., a multi-level insert). The insert 74 is spaced from but not directly contacts the interior surface of the waste container 26, such that the first imaging surface 75a is spaced a first distance from the interior surface and the second imaging surface 75b is spaced a second distance from the interior surface that is greater than the first distance. In one example, the first distance is 1.7 millimeters and the second distance is 2.2 millimeters. It is contemplated that the first distance can be approximately between 0.7 millimeters and 5.7 millimeters, more particularly between 1.2 millimeters and 3.7 millimeters, and the second distance can be approximately between 1.2 millimeters and 6.2 millimeters, more particularly between 1.7 millimeters and 4.2 millimeters. The first and second imaging surfaces 75a and 75b can be separated by a ridge having a thickness equal to the difference between the first and second distances. For example, the ridge can be approximately 0.5 millimeters. It should be understood that the insert 74 can include two, three, four, five, or more imaging surfaces, and the illustrated embodiment is a non-limiting example. Alternatively, the imaging surface 75 can also include a continuous gradient imaging surface, such as a wedge shape, to allow several levels of increasing fluid color intensity to be measured.

[0043] The side-by-side configuration provides, among other advantages, improved cleanability of the insert 74. The cleaning subsystem's sprayer is rotatably coupled to the container lid 82 of the waste container 26, and the sprayer directs liquid downward and radially outward (schematically represented by arrows) toward the inner surface. The cleaning subsystem's rotatable sprayer directs pressurized fluid toward the insert 74. The relative spacing of the first and second imaging surfaces 75a, 75b can be based on the direction of rotation of the cleaning subsystem's sprayer. More specifically, the first and second imaging surfaces 75a, 75b can be configured such that the pressurized liquid contacts the first imaging surface 75a (i.e., closer to the inner surface) before contacting the second imaging surface 75b. If there is any semi-solid or solid debris between the insert 74 and the inner surface, the aforementioned flow direction increases the likelihood of removing the debris. Therefore, the thin layer of medical waste material is better washed away from the gap between the imaging surface 75 and the interior surface, and improved cleaning can limit contamination of the insert 74 or otherwise preserve the optical properties of the insert 74.

[0044] At least one fiducial marker 76 can be detected by the camera 50 to localize the region of the image associated with the imaging features of the insert 74 and to color correct for variations in lighting, flash, or other optical aberrations. A preferred embodiment of a fiducial marker is disclosed in commonly owned U.S. Patent No. 9,824,441, issued November 21, 2017, the entire contents of which are incorporated herein by reference, in which a quick response (QR) code of known red color component values (or RGB, HSV, or CMYK color schemes, etc.) is adhesively affixed to the exterior surface of the waste container 26 corresponding to the location of the upper surface of the insert 74. Additionally, in a manner described below, calibration data can be associated with the unique code of the fiducial marker(s) 76 to account for mechanical variations of the waste container 26. Additionally, in a preferred embodiment, at least two fiducial markers 76 can be affixed to the waste container 26 in a manner that facilitates image registration.

[0045] 9A and 9B, a method 100 for estimating the volume of blood in a waste container 26, also referred to herein as estimated blood loss or estimated blood loss, is provided. The method 100 disclosed herein and one or more of its steps are configured to be executed by a processor(s) 44 according to instructions stored on a non-transitory computer-readable medium. The data can be analyzed by the processor 44 on the imaging device 50 and / or the data can be transmitted for remote processing (e.g., cloud computing). The method 100 can also be performed using a machine learning (ML) model or can implement one or more trained neural networks. The computer-executable instructions can be implemented using an application, applet, host, server, network, website, communication service, communication interface, hardware, firmware, software, or the like. The computer-readable medium may be stored on any suitable computer-readable medium, such as RAM, ROM, flash memory, EEPROM, optical elements, hard drive, or floppy drive.

[0046] The neural network(s) can execute algorithms trained using supervised, unsupervised, and / or semi-supervised learning methodologies to perform the methods of the present disclosure. A diverse and representative set of training datasets is collected to train each of the neural networks for a variety of scenarios both within and outside of the expected use of the system 20. Training methods can include the use of data augmentation, pre-trained neural networks, and / or semi-supervised learning methodologies to reduce the requirement for labeled training datasets. For example, data augmentation can include various blood concentrations, semi-solids, various soiling of the sidewalls of the waste container 26, lighting conditions (brightness, contrast, hue-saturation shift, Planckian jitter, etc.), geometric transformations (rotation, homography, inversion, etc.), addition of noise and blur, custom enhancements, hemolysis, placement of the imager 50, coagulation, detergent, or suction level to increase sample size, etc. Additionally, tolerance testing can be performed to determine acceptable dispersion for parameters such as insertion depth, canister tilt, and mechanical dispersion of waste container 26.

[0047] The method 100 may include operating the system in an optional standby or low-power mode (step 102). As the name implies, the low-power mode is configured to conserve power (e.g., the battery life of the imaging device 50) during periods when there is no change in the fluid level in the waste container 26. For example, prior to the start of a surgical procedure, the imaging device 50 software may be started, but the medical waste collection system 20 does not draw waste materials into the waste container 26. The low-power mode 102 may include capturing a video feed (i.e., a series of image frames) using the camera 50 with the flash turned off and at a relatively low frame rate (step 104). In one example, the frame rate is approximately 15 frames per second (fps), although other frame rates are within the scope of this disclosure. The image frames (also referred to herein as images) are preprocessed (step 106), e.g., downsampled. The downsampled images are analyzed by an activity recognition algorithm (step 108). The activity recognition algorithm detects the pixel-based position of the fluid meniscus in each of the images. The activity recognition algorithm is configured to determine whether the pixel-based position of the fluid meniscus has changed by an amount greater than a predetermined threshold within a predetermined time period. In other words, the activity recognition algorithm determines whether waste material is being drawn into the waste container 26 at a predetermined rate. In one example, the activity recognition algorithm can compare the pixel-based position of the fluid meniscus between successive image frames of the video feed and compare the change to a predetermined threshold. If the change remains below the predetermined threshold, the processor 44 maintains the system in low power mode and forgoes performing the remaining steps of method 100. If the change in the fluid meniscus exceeds the predetermined threshold, the processor 44 performs the remaining steps of method 100. Additionally or alternatively, a user can provide input, such as to a touchscreen display of the imaging device 50 or a paired external device, to exit low power mode and enter eBL mode.

[0048] The method 100 includes activating a light source (step 110), such as activating the flash of the imaging device 50. This step is optional; alternatively, the flash of the imaging device 50 can be continuously activated. In another variation, the level of the flash of the imaging device 50 can be increased in eBL mode. The container backlight of the waste container 26 can also be already activated. Using light from both the flash and the container backlight can provide optimal illumination with minimal glare. Note that the precise location of the camera 50 supported by the device cradle 54 can limit glare to small dots located above the insert 74 and fiducial marker 76 (see FIGS. 19 and 20), areas of the image that are less important for image analysis. Furthermore, a greater signal-to-noise ratio (i.e., blood color signal relative to background reflections) reduces the impact on image analysis, thereby eliminating the need for a separate algorithm to reduce or eliminate the impact of glare. In addition to activating the flash, the feedrate of the video feed can also be increased for improved data resolution.

[0049] Method 100 may include analyzing at least one of the image frames of the video feed (step 112), preferably analyzing a series of multiple image frames according to an algorithm. In other words, every image frame may be analyzed, every other image frame may be analyzed, or every third image frame may be analyzed, etc., based on a desired data resolution taking into account available computing resources. In alternative embodiments that do not utilize a video feed, method 100 may also include capturing one or more multiple exposure photographs for subsequent analysis.

[0050] Method 100 includes detecting fiducial marker(s) 76 (step 114), such as, for example, QR code(s), barcode, or another marker having optically readable data. Among other aspects described below, a QR code can be used to account for mechanical variations in the volume of waste container 26. In other words, manufacturing tolerances can result in slight variations in the interior volume of waste container 26 and the volume of insert 74 and insert mount placed therein, leading to inaccuracies in image-based volume determination. Furthermore, slight variations in the interior volume can result from vacuum-induced variations, where pressure differences due to a vacuum being drawn inside waste container 26 cause the sidewalls of waste container 26 to “bow” inward, displacing the waste material contained therein. Furthermore, further slight variations in the interior volume can result from thermal expansion of waste container 26 due to the higher temperature of the patient-derived body temperature fluid within waste container 26 relative to the surroundings.

[0051] The present disclosure addresses such concerns by associating, in memory, QR codes with calibration-based data associated with the waste container 26. Calibration can be performed by a technician during deployment (e.g., assembly or retrofit) of the eBL subsystem with the medical waste collection system 20. The technician can affix at least two QR codes, preferably in a vertical configuration as shown in FIG. 10 . The device cradle 54 is installed in the manner previously described. The camera 50 is activated in a calibration mode, and at least one calibration operation is performed in which at least one known volume of fluid is aspirated into the waste container 26. The processor 44 performs image-based volume determination, described below, including ascertaining the vertical y-axis value of the fluid meniscus, and associates each of the y-axis value and known fluid volume(s) with a unique code in the QR code 76. The calibration data 88 is stored in memory, such as the memory of the imaging device 50 or a cloud-based memory. During operation, camera 50 detects QR code 76 and processor 44 accesses calibration data 88 from memory and performs an image-based volume determination that is adjusted by or takes into account the calibration data.

[0052] Alternative methods for container-specific calibration are also contemplated. In a first example, the displayed calibration mode of the imaging device 50 can perform a field-of-view calibration, guiding the user to aspirate a target volume of fluid. After doing so, input is provided to the user interface, and the processor 44 performs the image-based volume determination described below, including confirming the y-axis value of the vertical direction of the fluid meniscus and associating each of the y-axis values. The user interface displays the predicted volume, and the user can provide a confirmation input to the user interface to indicate that the predicted volume determined by the processor 44 is equal to the target volume of fluid. In a second example, a fiducial marker 76 can be affixed to a precise location and verified, for example, via laboratory testing. The camera 50 is operated in calibration mode, and the y-axis position of the fiducial marker 76 is determined and associated with the fiducial marker's 76 unique code. In a third example, the container-specific calibration data can be stored on a memory chip, such as a near-field communication (NFC) tag. The NFC tag can be detected by a complementary NFC reader in the imaging device 50, and calibration data is transmitted to the imaging device 50. The calibration data is stored in memory along with the unique code of the fiducial marker 76. Such an alternative method may require precise placement of the fiducial marker 76. The step of compensating for volume variance in the waste container 26 is optional, and the image-based volume determination may be sufficiently accurate without it.

[0053] Method 100 includes aligning the image (step 116). Because evaluation of the fluid meniscus is on the vertical y-axis, it is desirable that the image of the canister be oriented vertically with sufficient precision. In other words, aligning the image can account for mechanical variations in the rotational positioning of waste container 26 within housing 22. Alignment can be based on features external to waste container 26 (i.e., external alignment), features associated with or inherent to the waste container (i.e., internal alignment), or a combination thereof. With continued reference to FIG. 10 , aligning the image can be based on detecting fiducial markers 76. For example, a QR code can be affixed to waste container 26 using a jig or guide so that it is positioned vertically with sufficient vertical precision. Processor 44 is configured to detect the QR code, as represented by the bounding box in FIG. 10 , and rotate the image accordingly. Registration techniques for three-dimensionally orienting and positioning an image within a frame, including homography, perspective transformation, 3D-to-2D mapping, etc., are within the scope of this disclosure. Figures 15B, 16B, 17B, and 18B show the effect of image rotation relative to Figures 15A, 16A, 17A, and 18A, respectively.

[0054] Additionally or alternatively, camera 50 may be configured to detect landmarks or fiducials on waste container 26 or housing 22 and align the image accordingly. Figure 11 depicts a machine vision output with the interior structure 78 of housing 22 superimposed on the image. Additionally or alternatively, external landmarks may be located at fixed location(s) on waste container 26, e.g., a printed fiducial such as an additional QR code or April tag affixed to a known location on waste container 26. Other reference structures are also contemplated, such as a sensor rod 80 of the fluid measurement subsystem, a mount for insert 74, a subcomponent of container lid 82, etc.

[0055] The internal registration may include the processor 44 detecting, determining, or identifying container landmarks. Referring to FIGS. 12A-12C, the container landmarks may include a bottom edge 86′, at least one side edge 84′, and / or other identifiable landmarks on the waste container 26. These edges may be identified by comparing adjacent pixels, clusters of pixels, or portions of the image. Sufficient contrast in the comparison may indicate a transition, i.e., an edge, from the interior structure of the housing 22 to the waste container 26. The bottom edge 86′ may be an arc-shaped boundary associated with a surface near the base of the waste container 26. This step may implement any suitable machine vision and / or machine learning techniques to identify edges and / or estimate features or dimensions of the waste container 26. The processor 44 is configured to register in the image to a predefined registration. For example, the processor 44 may be configured to orient the side edge represented by reference numeral 84 to be substantially vertical. As another example, processor 44 may be configured to align the lowest surface of bottom end 86' with the lower boundary of the image frame represented by reference numeral 86. It will be appreciated that external alignment may be used alone or in combination with internal alignment.

[0056] Image registration step 116 can also include compensating for tilt (e.g., front-to-back misalignment) of the waste container 26. Tilt may be due to an uneven floor surface, manufacturing variations, or the like. A tilt correction method 118 is shown in FIG. 13. Method 118 includes processor 44 receiving an image frame (step 120) and detecting fiducial markers 92 (step 122). Referring again to FIG. 8, fiducial markers 92 are configured to facilitate determining tilt relative to the plane of the image. For example, markers 92 can be ArUco markers affixed and contoured to the alignment frame 90 to the curvature of the waste container 26, with markers 92 located on four sides of the frame 90. ArUco markers provide better angular orientation than is possible with a QR code. In one variation, ArUco markers are not placed on the frame 90, but separate ArUcos are affixed to the waste container 26 at predetermined locations. For example, Figures 14A and 14B show a variation in which the fiducial markers 92 are printed with a color reference palette and human-readable code and positioned in a manner that limits unnecessary obstruction of the waste container 26 and volumetric markings.

[0057] The image frames are segmented to determine the center of mass of the waste container 26. Method 100 may include performing camera calibration and determining the center of the optical sensor. Camera calibration may be performed by determining the orientation of the imaging device 50. In an exemplary embodiment, where the imaging device 50 is an iPhone® 13 Pro Max, this orientation is determined based on an internal gyroscope and, optionally, from a point associated with the detection of the fiducial marker 92. Method 100 further includes mapping image coordinates to canonical coordinates (step 124). The center of mass pixel point and the optical sensor center pixel point may each be transformed to a three-dimensional point in the canonical space of the waste container 26. From the orientation of the waste container 26 and the imaging device 50, a mathematical correction is determined. For example, the mathematical correction may be a coefficient associated with the pose of the waste container 26 that minimizes a least-squares equation. Based on the image-based volume determination (step 126) and the mathematical correction, a corrected volume is determined (step 128).

[0058] 9A and 9B, and with further reference to FIG. 10, method 100 includes segmenting an image of waste container 26 with fluid contained therein. Neural network(s) (hereinafter referred to in the singular) can characterize whether a pixel is blood (B), fluid meniscus (M), or non-blood (NB). In an optional variation, the neural network can further characterize whether a pixel is foam or froth (F) and / or fluid surface (FS). In other words, each pixel is assigned a value from the neural network and then assigned a class label based on that value. The neural network can be a deep learning neural network such as U-Net, Mask-RCNN, DeepLab, or any image-based segmentation algorithm such as grabcut, clustering, region growing, etc. As further examples, neural networks can be used for object localization, segmentation (e.g., edge detection, background subtraction, grab-cut-based algorithms, etc.), gauging, clustering, pattern recognition, template matching, feature extraction, descriptor extraction (e.g., extraction of texton maps, color histograms, HOG, SIFT, MSER (Most Stable Extremal Regions for removing blob features from selected areas), etc.), feature dimensionality reduction (e.g., PCA, K-means, linear discriminant analysis, etc.), feature selection, thresholding, localization, color analysis, parametric regression, non-parametric regression, unsupervised or semi-supervised parametric or non-parametric regression, neural networks and deep learning-based methods, or any other type of machine learning or machine vision. Figures 15D, 16D, 17D, and 18D show representations of segmentation masks with different colorings for non-blood, meniscus, and blood class labels. Figures 16D and 18D also show fiducial markers 76 excluded from the segmentation masks.

[0059] Method 100 includes determining the location of the fluid meniscus (step 132). Step 132 is based on a segmentation mask and may include using the full width of the image to post-process the segments. The machine learning techniques described above can be implemented to train a segmentation network on the video feed to identify the meniscus under various conditions, including blood concentration levels, thickening agents, illumination from a flash, illumination from a container backlight, vacuum level of the vacuum source 30, foaming of the waste material, and cloudiness or opacity of the waste container 26. Other means by which the meniscus can be detected are disclosed in commonly owned U.S. Patent No. 8,983,167, issued March 17, 2015, the entire contents of which are incorporated herein by reference.

[0060] Step 132 may include fitting a parabola to pixels having the meniscus class label. FIGS. 15E, 16E, 17E, and 18E show representations of the fluid meniscus with the non-blood and blood class labels removed. In instances where the fluid meniscus is below camera 50, the parabola will include a lower apex and open upward. Conversely, if the fluid meniscus is above camera 50, the parabola will include an upper apex and open downward. The position of the fluid meniscus is the y-axis value of the apex of the parabolic curve. Therefore, for a lower apex, the position of the fluid meniscus is the minimum of the parabolic curve, and for an upper apex, the position of the fluid meniscus is the maximum of the parabolic curve. FIGS. 15B, 16B, 17B, and 18B show representations of machine video output 94 showing the y-axis values of the apexes overlaid on the adjusted image of waste container 26.

[0061] Method 100 further includes mapping the image from image coordinate space to canonical coordinate space (step 124), as introduced above. Step 124 may require four orientation markers (e.g., two QR codes) to generate a homography and map the fluid meniscus from image coordinates to canonical coordinates. One or more of the alignment blocks of the QR code (e.g., the top left alignment block) can be used for transformation to two-dimensional canonical coordinates and then to three-dimensional canonical coordinates. FIGS. 15C, 16C, 17C, and 18C show representations of machine image output 94′ in canonical coordinates showing the y-axis values of the vertices overlaid on an adjusted image of waste container 26, and FIGS. 15F, 16F, 17F, and 18F show representations of a cropped adjusted image of waste container 26 overlaid with machine image output 94′. The cropped image frame in canonical coordinate space can include at least two fiducial markers and the portion of the waste container between them. The width of the cropped and mapped image frame is approximately equal to the width of the at least two fiducial markers. Cropping the image limits the range (e.g., image width) over which the meniscus can be located. Cropping the adjusted image is optional.

[0062] Method 100 includes a pixel-to-volume mapping step (step 126) to determine the volume of fluid in the waste container. This step may include retrieving or receiving calibration data 88 from memory, e.g., container-specific coefficients to account for container-specific variance, as described above. The image-based volume determination is the y-axis value of the fluid meniscus in canonical coordinates, adjusted by the calibration data. In other words, the lowest point of the meniscus along the y-axis as defined in the image by processor 44 can be mapped to the data, and the y-axis location of the meniscus is converted to a volume in milliliters. Again, the machine learning techniques described above can be implemented to train a segmentation network to convert the y-axis location of the meniscus to a volume of fluid. As an example, referring to FIG. 16C, a y-axis value of “2000” in canonical coordinates (left axis of the image) may result in a volume determination of approximately 1000 mL.

[0063] Method 100 may include extracting a region of interest (ROI) from the image (step 134). Referring simultaneously to FIG. 19 , processor 44 is configured to identify fiducial marker 76 and process information contained therein to determine the corresponding location of region of interest 96′. In other words, data in, for example, a QR code may cause processor 44 to analyze region of interest 96′ in the image frame at a predetermined location relative to the QR code. Furthermore, in embodiments in which insert 74 includes first and second imaging surfaces 75 a and 75 b, fiducial marker 76 may include data that causes processor 44 to analyze first and second regions of interest 98 a and 98 b, respectively.

[0064] In certain embodiments, a further optional step 136 may include extracting a palette color from the fiducial marker(s) 76, performing noise reduction, feature extraction, and light normalization. Palette extraction, noise reduction, and feature extraction may be performed in a manner at least similar to that described in the aforementioned U.S. Pat. No. 9,824,441. Illumination variance may be accounted for by training the segmentation network with images using illumination from the camera 50 flash alone, illumination from the container backlight alone, or a combination of both at various levels. Among other benefits, this may allow blood constituents to be determined over a wider range of blood concentrations without signal saturation. The light intensity level provided by the camera 50 flash may be of known brightness, color temperature, spectrum, etc. Additionally or alternatively, the container backlight may provide light of known brightness, color temperature, spectrum, etc.

[0065] Light normalization according to known solutions can be primarily used to account for the variance of ambient lighting. When the imaging device's camera 50 and flash are supported in the device cradle 54 and configured for detection in the insert 74 near the bottom of the waste container 26, the effects of directional light become relatively significant. Variance in the relative positioning between the camera 50 and the waste container 26 and the gradation of light from the imaging device's 50 flash result in an uncertain, nonlinear relationship such that empirical determinations and machine learning models may result in insufficiently accurate determinations of the blood component concentrations in the waste material. For example, slight variations in the imaging device's 50 orientation may result in different positions and intensities of the flash reflected from the surface of the waste container 26 and different qualities (e.g., color component values) of the waste material images or image frames. In other words, the light intensity detected by the camera 50 may depend on the position of the flash on the surface of the waste container 26. As a result, in certain embodiments, analysis of images or image frames to determine blood constituent concentrations can be facilitated by a neural network trained on image datasets of various positions of the camera 50 relative to the waste container 26 and various positions of the flash of the imaging device 50 reflected off the surface of the waste container 26.

[0066] Therefore, in certain embodiments, light normalization may require at least two fiducial markers, each with a known color profile. In one example, a neural network may be trained to predict the color profile of a lower one of the fiducial markers 76 based on the color profile of one or more of the upper ones of the fiducial markers 76. Light normalization may include applying a color correction based on the predicted color profile of the lower fiducial marker 76. In another example, an image or image frame for determining blood constituent concentrations may be facilitated by a neural network trained on an image dataset of imaged color component values of at least two, at least three, or four or more fiducial markers 76.

[0067] The images captured by camera 50 can be captured under lighting of a constant intensity and other characteristics. Alternatively, step 106 is optional and can include capturing images using camera 50 at various levels of light intensity provided by the camera's 50 flash, with processor 44 utilizing an algorithm to compensate for the levels of light intensity. For example, a first image can be captured under high flash intensity, a second image under medium flash intensity, a third image under low flash intensity, etc., allowing for an increased dynamic range of the captured images and an increased associated range of blood constituents measurable by the algorithm. Another light normalization technique includes intensity gradient normalization, in which the distance and / or angle of the flash are taken into account.

[0068] Method 100 may further include step 138, which involves analyzing regions of the image associated with the imaging features of insert 74 to quantify the concentration of blood constituents in the waste. The analysis may be performed in the manner disclosed in the aforementioned U.S. Patent No. 8,792,693, where a parametric model or template matching algorithm is implemented to determine the concentration of blood constituents associated with the fluid in waste container 26. In particular, processor 44 may be configured to extract color component values (e.g., redness values) from the image and execute a trained algorithm to determine the concentration of blood constituents on a pixel-by-pixel or other suitable basis. Hemoglobin (Hb) or blood loss (eBL) is estimated based on the image-based volume determination and the concentration of blood constituents using a hemoglobin estimation algorithm (step 140). Step 140 may be the product of the two or based on other suitable calculations, such as Equation 1. δM=VC(δL / L)+VδC algo +CδV algo (1)

[0069] 20 illustrates a variation of method 100 in a machine learning environment in which certain features are extracted using the aligned images and the positions of fiducial markers 76, and light normalization is performed in color space. The variation of FIG. 9B includes certain steps integrated to be performed by a neural network analyzing image frames from a video feed. In particular, by training the neural network on a diverse and representative set of training data sets, the steps of region of interest extraction, color palette extraction, noise removal, feature extraction, and light normalization can be performed by the neural network rather than as separate steps performed by method 100 without compromising the accuracy of the hemoglobin estimation.

[0070] In another variation, the method 100 shown in FIG. 9A can be implemented in a boundary case, as described below, and the method 100 shown in FIG. 9B can be implemented in other ways by default. The boundary case can be an instance where the volume of waste material is below a predetermined volume. The predetermined volume can be based on the waste material fluid level not being above the insert 74 placed in the waste container 26. For example, the predetermined volume can be less than approximately 1000 mL, which can include 800 mL of pre-fill liquid and an additional 400 mL of waste collected during the procedure. Below this predetermined volume, the method 100 of FIG. 9A (and FIG. 20) implemented by an ML model may be more suitable for these threshold cases. Otherwise, for fluid volumes above the predetermined volume, the method 100 of FIG. 9B implemented by a trained neural network may be more accurate over a wider range of blood concentrations. Processor 44 may be configured to implement any variation of method 100 based on the volume of fluid or other considerations or decisions.

[0071] The estimated blood loss can be displayed on one or more displays (step 142), such as a touchscreen display of the imaging device 50. Additionally or alternatively, the eBL can be wirelessly transmitted to the user interface 52 of the medical waste collection system 20 or to another display terminal in the operating room. As will be readily appreciated from the above disclosure, the eBL can be displayed and updated at desired intervals or in real time. In particular, in video mode, the camera 50 can capture a video feed in which the volumes of blood components and fluids, respectively, are repeatedly determined in a near-instantaneous manner. Furthermore, the touchscreen display of the camera 50 shows the camera's field of view, so the ability to visualize the interior space of the waste container 26 is generally unobstructed and can be further augmented with information useful to the user.

[0072] In certain embodiments, it is contemplated that the camera 50 may be paired with the medical waste collection system 20, and more particularly, the controller 42, via two-way wireless communication. In such a configuration, a fluid measurement subsystem may be utilized as an alternative to or in combination with image-based volume determination. Data from the fluid measurement subsystem may also be utilized to improve machine learning. If a patient's blood loss exceeds a predetermined or selected limit, a warning may be provided via the user interface 52. Blood loss data associated with the medical procedure may be transmitted to an electronic medical record.

[0073] Additional warnings or guardrails are contemplated by this disclosure. In certain embodiments, the camera 50 and processor 44 can be configured to determine and provide event detectors regarding the status of the waste container 26. In other words, the machine learning algorithms can be extended to determine whether suction is on or off, whether waste material is flowing into the waste container 26, whether the waste container 26 is static or emptying, etc. Similarly, the machine learning algorithms can be extended to determine whether a clot or other debris is lodged near or in front of the imaging feature of the insert 74 and / or whether the insert 74 is missing, dislodged, or otherwise improperly positioned beyond a predetermined threshold. While a decrease in blood component concentration is acceptable, an error warning can also be provided if the processor 44 determines that the eBL has decreased.

[0074] Additionally or alternatively, a baseline image of the insert 74, e.g., an initial image prior to treatment with no blood in the container, can be used to determine blood constituents. The baseline image can be compared to some number of all subsequent images. Utilizing a baseline image provides compensation for color changes and light intensity variations of the imaged features of the insert 74 over multiple uses, for example, inside the waste container 26, subject to repeated soiling and cleaning cycles. Note that image-based blood concentration ratios relative to the baseline image may not be directly dependent on light intensity.

[0075] In certain embodiments, the machine learning algorithms may be extended to determine the degree of cloudiness of the interior or exterior surfaces of the waste container 26, damage (e.g., scratches) to the exterior surface of the waste container 26, or dirt on the lens of the camera 50. For example, the soiling rate of the waste container 26 or the insert 74 can be determined by capturing images, such as baseline images, prior to and during each procedure. For example, it can be assumed that in the absence of soiling, analysis of the baseline images should fall within a predetermined similarity threshold. If analysis of the baseline images exceeds a predetermined similarity threshold, the waste container 26 can be deemed too soiled beyond a confidence interval for reliable analysis. Corresponding predictive warnings or corrective actions can also be provided. In particular, predictive maintenance can be requested, in which a technician cleans the interior and exterior surfaces of the waste container 26 and replaces the insert 74, based on the analysis.

[0076] Furthermore, the machine learning algorithm can be extended to determine the heterogeneity of waste material within the waste container 26, for example, by analyzing the fluid-containing imaged region determined by the segmentation network. Based on calculating features of heterogeneity, such as edge scores, standard deviation of pixel values, HOG, or other similar features, the user may be alerted to scenarios in which the fluid content is not uniform, as such a situation may lead to an inaccurate analysis of the blood content within the fluid. In such a situation, the user may be instructed to agitate the fluid, for example, by mechanical movement or by drawing water at high suction pressure, thereby adequately mixing the fluid contents within the imaging container for accurate analysis of the blood content. Once adequate mixing is achieved and the heterogeneity score is below a certain threshold, the analysis of the blood content may proceed normally.

[0077] In another embodiment, camera 50 can be integrated into housing 22 and not necessarily a mobile device supported on device cradle 54. In such an embodiment, camera 50 can be at least one digital camera coupled to housing 22 in any suitable location, such as within front casing 46. Additionally or alternatively, a digital camera can be coupled to waste container 26 so as to be positioned inside and / or outside the waste volume. For example, a digital camera can be coupled to container lid 82 and pointing downward. For improved accuracy and redundancy, multiple cameras can be utilized in combination, with images analyzed therefrom using combined analysis by machine learning algorithms.

[0078] Several embodiments have been discussed in the preceding description. However, the embodiments discussed herein are not intended to be exhaustive or to limit the present invention to any particular form. Modifications and variations are possible in light of the above teachings, and may be practiced in manners other than those specifically described. For example, the methods disclosed herein may be performed on a waste container that is not disposed on a mobile housing. In other words, a device cradle may be provided in which a freestanding canister is placed, such as that disclosed in commonly owned U.S. Patent No. 10,641,644, issued May 5, 2020, the entire contents of which are incorporated herein by reference. For example, the blood component may be hemoglobin or one or more of whole blood, red blood cells, platelets, plasma, white blood cells, and analytes, etc. The method may also be used to estimate the concentration and amount of non-blood components in waste container 26, such as saline, ascites, bile, perfusate, saliva, gastric juice, mucus, pleural fluid, interstitial fluid, urine, or feces. The medical waste collection system 20 can communicate with other systems to form a fluid management ecosystem that generates fairly comprehensive estimates of extracorporeal blood volume, total blood loss, or the patient's euvolemic status, etc. Additionally, certain inventive aspects of the present disclosure are made with reference to the following exemplary sections.

[0079] Item 1 - A method for estimating blood loss in waste material within a waste container of a medical waste collection system, wherein an imaging device is supported in a device cradle, the method including the steps of: capturing images of the waste container and waste material placed therein using the imaging device; analyzing the images using one or more processors to determine a volume of the waste material within the waste container; analyzing the images using one or more processors to determine a concentration of blood components within the waste material; estimating blood loss using the one or more processors based on the determined volume and the determined concentrations of blood components; and displaying the estimated blood loss on a display.

[0080] Item 2 - A method for estimating blood loss in waste material within a waste container of a medical waste collection system, wherein an imaging device is supported in a device cradle, the method comprising: using the imaging device to capture a video feed of the waste container and waste material contained therein while a vacuum source of the medical waste collection system draws the waste material into the waste container; using one or more processors, segmenting image frames of the video feed by characterizing each pixel value of the image frame as blood, a meniscus, non-blood, and optionally a bubble or a fluid surface; using one or more processors, assigning a class label based on the pixel values; using one or more processors, determining a position of the fluid meniscus based on the meniscus class label; mapping the image frames from the image coordinate space to a canonical coordinate space; and using one or more processors, determining a volume of the waste material within the waste container based on the position of the fluid meniscus.

[0081] Item 3 - The method described in Item 2, further comprising using one or more processors to detect at least two reference markers affixed to the waste container, and cropping the image frame mapped in the canonical coordinate space to include the at least two reference markers and a portion of the waste container between them.

[0082] Item 4 - The method of item 3, further comprising using one or more processors to register the images based on the relative positioning of at least two fiducial markers.

[0083] Item 5 - An apparatus cradle for supporting an imaging device for capturing images of waste containers of a medical waste collection system, wherein a front casing of the medical waste collection system defines a window, the apparatus cradle includes a shield coupled to the front casing, the shield defining a rear recess configured to cover the window, a front recess, and an opening providing fluid communication between the rear recess and the front recess, the front recess being sized to receive the imaging device, and the opening being positioned relative to the front recess to be aligned with a camera and flash of the imaging device.

[0084] Item 6 - The device cradle of item 5, further comprising at least one arm connecting the shield to the front casing, and a hinge connecting the at least one arm to the shield and configured to enable the shield to be pivoted into a configuration in which the window is visible with the imaging device disposed therein.

[0085] Item 7 - The device cradle of item 6, further comprising a front housing movably coupled to the shield.

[0086] Item 8 - The device cradle of any one of items 5-7, configured such that at least substantially the entire waste container is visible within the field of view of the camera of the imaging device.

Claims

1. A method for estimating blood loss of waste material in a waste container of a medical waste collection system, wherein the imaging device is supported in a device cradle, the method is: The steps include capturing a video feed of the waste container and the waste material contained therein using the imaging device while the vacuum source of the medical waste collection system is drawing the waste material into the waste container, A step of analyzing one or more image frames of the video feed using one or more processors to determine the volume of the waste material in the waste container, The steps include analyzing the image frame using one or more processors to determine the concentration of blood components in the waste material, A step of estimating the blood loss based on the determined volume and the determined concentration of the blood components using one or more processors, The steps include displaying the estimated blood loss on a display, Methods that include...

2. A method for estimating blood loss of waste material in a waste container of a medical waste collection system, wherein the imaging device is supported in a device cradle, the method is: The steps include capturing a video feed of the waste container and the waste material contained therein using the imaging device while the vacuum source of the medical waste collection system is drawing the waste material into the waste container, A step of analyzing image frames of a video feed to determine the concentration of blood components in the waste material using one or more processors, wherein the step of analyzing the image frames is facilitated by a neural network trained on at least one image dataset of the relative positioning of the camera of the imaging device with respect to the waste container, the location of the flash reflected on the surface of the waste container, the color component values ​​associated with at least two reference markers attached to the surface of the waste container, and the color component values ​​associated with the imaging surface of an insert placed inside the waste container. A step of estimating the blood loss based on the volume of the blood component and the determined concentration using one or more processors, The steps include displaying the estimated blood loss on a display, Methods that include...

3. The method according to claim 2, wherein the step of analyzing the image frame is facilitated by the neural network further trained on the image dataset of regions of interest related to at least two imaging surfaces of the insertion unit, and each of the at least two imaging surfaces provides different color component values ​​for the waste material between the insertion unit and the waste container.

4. The method according to claim 2, wherein the step of analyzing the image frame is facilitated by the neural network further trained on the image dataset of the region of interest related to the imaging surface of the insertion unit, thereby providing a color gradient to the waste material between the insertion unit and the waste container.

5. The method according to any one of claims 2 to 4, wherein the step of analyzing the image frames is facilitated by the neural network further trained on the image dataset of at least one of various volumes of the waste material and various phase characteristics of the waste material.

6. Mapping the image frame from the image coordinate space to the canonical coordinate space, Providing the image frame in the canonical coordinate space to the neural network, The method according to any one of claims 2 to 4, further comprising:

7. The method according to any one of claims 2 to 4, further comprising using one or more processors to analyze the image frames of the video feed to determine the volume, wherein the step of analyzing the image frames is facilitated by the neural network further trained on the image dataset.

8. The steps include capturing the video feed at a first frame rate using the imaging device, A step of using one or more processors to determine the rate at which the volume of the waste material is increasing in the waste container, If the rate is greater than a predetermined threshold, the imaging device is used to capture the video feed at a second frame rate greater than the first frame rate. The method according to claim 1 or 2, further comprising:

9. The method according to claim 8, further comprising activating the flash of the imaging device or increasing the level of the flash when the rate is greater than the predetermined threshold.

10. Using one or more of the aforementioned processors, the pixel-based position of the fluid meniscus in the image frame is detected. Using one or more of the aforementioned processors, determine whether the position of the pixel base of the fluid meniscus has changed by an amount greater than a predetermined threshold within a predetermined period of time. The method according to claim 9, further comprising:

11. The method according to claim 10, further comprising downsampling the image frame prior to the step of detecting the pixel base position of the fluid meniscus.

12. Using one or more processors, detect at least two reference markers attached to the waste container, Using one or more of the aforementioned processors, the image frame is aligned based on the relative positioning of at least two reference markers. The method according to claim 1 or 2, further comprising:

13. The method according to claim 12, wherein the at least two reference markers are quick response codes (QR codes®) arranged in a vertical configuration, and the step of aligning the image frame further includes aligning the QR codes® so that they are precisely vertical.

14. The image frame is segmented by using one or more processors to characterize each pixel value of the image frame as blood, a fluid meniscus, or non-blood, and as foam or a fluid surface. Assigning a class label based on the aforementioned pixel values, The method according to claim 1 or 2, further comprising:

15. The method according to claim 14, further comprising determining the position of the fluid meniscus based on the class label of the meniscus.

16. The method according to claim 1 or 2, further comprising mapping the image frame from the image coordinate space to the canonical coordinate space.

17. The step of mapping the aforementioned image frame is: To generate homography of localization markers, The aforementioned image coordinates are transformed into two-dimensional canonical coordinates, The transformation of the aforementioned two-dimensional canonical coordinates into three-dimensional canonical coordinates, The method according to claim 16, further comprising:

18. The method according to claim 16, referencing claim 2, further comprising cropping the mapped image frame in the canonical coordinate space to include at least two reference markers and a portion of the waste container between them.

19. The step of analyzing the image frame to determine the volume of the waste material is: Receiving container-specific calibration data, Converting the y-axis value of the mapped image frame in the canonical coordinate system into a volume value, The method according to claim 16, which further includes the following:

20. A method for estimating blood loss of waste material in a waste container of a medical waste collection system, wherein the imaging device is supported in a device cradle, the method is: The steps include capturing images of the waste container and the waste material contained therein using the imaging device, A step of processing the image using one or more processors, wherein the step of processing the image is: Determining the position of the fluid meniscus, Aligning the aforementioned images, Mapping the image from the image coordinate space to the canonical coordinate space, Convert the y-axis value of the mapped image in the canonical coordinate system to the determined volume, Further steps include, A step of detecting at least two reference markers attached to the waste container using one or more processors, wherein the reference markers include position data, Using one or more of the aforementioned processors, the step of extracting a region of interest in the raw image based on the position data, Using one or more processors, the step of analyzing the image in order to determine the concentration of blood components in the waste, A step of estimating the blood loss based on the determined volume and the determined concentration of the blood components using one or more processors, The steps include displaying the estimated blood loss on a display, Methods that include...

21. The step of determining the position of the fluid meniscus is: The image frame is segmented by using one or more processors to characterize each pixel value of the image frame as blood, meniscus, or non-blood, and as foam or fluid surface. Using one or more processors, assign a class label based on the pixel value, wherein the position of the fluid meniscus is based on the class label of the meniscus. The method according to claim 20, further comprising:

22. The method according to claim 20, wherein the step of aligning the image further includes detecting at least two reference markers and aligning the at least two reference markers so that they are precisely vertical.

23. The step of mapping the aforementioned image is: Using one or more of the above processors, generate a homography of the localization marker, Using one or more of the aforementioned processors, the image coordinates are transformed into two-dimensional canonical coordinates, The transformation of the aforementioned two-dimensional canonical coordinates into three-dimensional canonical coordinates, The method according to claim 20, further comprising:

24. The method according to claim 20, further comprising cropping the mapped image in the canonical coordinate space to include the at least two reference markers and a portion of the waste container between them.

25. The method according to any one of claims 20 to 24, wherein the step of converting the y-axis value of the mapped image in canonical coordinates to a determined volume further comprises applying container-specific calibration data.

26. A non-temporary computer-readable medium for storing instructions configured to be executed by one or more processors to perform the method according to any one of claims 1 to 2 and 20.

Images