Assessment of blood loss in waste container of medical waste collection system

Abstract

A method for estimating an amount of blood loss within a waste container of a medical waste collection system, wherein an imaging device is supported by a device holder. The method includes analyzing an image, such as image frames of a video stream, to determine a fluid volume and a blood composition within a waste substance. These images may be adjusted based on a reference marker, and the method includes segmenting the images to determine blood, meniscus, and non-blood categories. A fluid meniscus of the waste material may be identified and mapped to a canonical coordinate space for pixel-to-volume conversion. Calibration data associated with the reference marker may be used to achieve this. At least one region of interest associated with an insert disposed within a waste container is analyzed. The estimated amount of blood loss may be displayed in real time based on the determined fluid volume and blood composition.

Description

[0001] Priority claim

[0002] This application claims priority to and all benefits of U.S. Provisional Patent Application No. 63 / 353,208, filed on June 17, 2022, the entire contents of which are incorporated herein by reference. Background Art

[0003] A byproduct of surgical procedures is the generation of liquid, semi-solid and / or solid waste materials. Medical waste may include blood, tissue fluid, mucus, irrigation fluids, etc. Medical waste can be removed from the surgical site through a suction tube under the action of a vacuum source to be collected in a waste container.

[0004] Estimating blood loss during surgery can be used to monitor the health of patients during surgery. Advances in imaging and computing technology provide for estimating blood loss by capturing images of a medium containing a fluid, such as an independent container. One such system is sold by Gauss Surgical, Inc. (Menlo Park, California) under the trademark Triton and disclosed in co-owned U.S. Patent No. 9,773,320, issued on September 26, 2017, the entire contents of which are hereby incorporated by reference herein. The independent container is arranged in series between the suction tube and the vacuum source. Once the amount of blood in the independent container is needed, the user can command the handheld imaging device to capture an image of the area in the container where the insert is placed and manually enter the volume of fluid in the container. Although the Triton system has achieved significant advantages in estimating blood loss, it would be more beneficial if the system required less user involvement and was able to estimate and display blood loss in real time.

[0005] The vacuum source may be a vacuum source integrated with the facility, or a vacuum source of a medical waste collection system. An exemplary medical waste collection system is sold by Stryker Corporation (Kalamazoo, Michigan) under the Neptune trademark, wherein at least one waste container is arranged on a mobile chassis. Since the Neptune system includes an integrated waste container, it is also desirable to eliminate the need for a separate container, which otherwise may become redundant. However, doing so may involve technical challenges addressed herein. For example, although a separate container can be discarded after a single use, the waste container of a medical waste collection system is a major component to be reused in multiple procedures. Contamination of the transparent sidewall of the waste container may affect the consistency of image quality, thereby affecting the accuracy of the estimated blood loss. For another example, although the Neptune system includes a subsystem for determining the volume of fluid in the waste container, in some cases, the imaging device may not be able to access the volume data (e.g., the imaging device and the Neptune system do not have electronic communication). In another case, the liquid volume determined by the Neptune system may not be able to quantify the pre-filled volume, i.e., the volume of fluid in the tank before the procedure begins. To be sufficiently accurate, image-based fluid volume determination may need to account for camera positioning, fluid bubbling, mechanical differences in medical waste collection system subcomponents, and other considerations such as lighting or flash variations, glare, etc. Summary of the invention

[0006] The present disclosure relates to a method for estimating blood composition within a waste container of a medical waste collection system, preferably in real time as waste material is drawn into the waste container under the influence of suction. The blood composition may be a hemoglobin concentration for estimating blood loss (eBL) and displaying the patient's estimated blood loss (and / or hemoglobin). The present disclosure also relates to an apparatus for performing the method disclosed herein. The method may be instructions stored on a non-transitory computer-readable medium configured to be executed by one or more processors. The method may be implemented by a machine learning (ML) model, a trained neural network, or a combination thereof. The neural network may be trained on an image data set of any number of factors or combinations thereof, including but not limited to the position of the camera relative to the waste container, the position of flashes reflected on the surface of the waste container, the color component values ​​of one or more regions of interest (i.e., the imaged region of the waste material placed between the insert and the inner surface of the waste container), the color component values ​​of at least two reference markers having a known color configuration and attached to the outer surface of the waste container, the phase characteristics of the waste material (e.g., blood, non-blood, fluid meniscus, bubbles, hemolysis level, etc.), etc. The image data set can be extensive and diverse for training the neural network within expected parameters for a medical waste collection system for at least substantially all expected scenarios. The image data set can also exceed the expected parameters, e.g., by one, two, three, or more standard deviations. Thus, in operation, images or image frames of a video stream analyzed by a neural network on one or more processors can estimate in real time the amount of blood loss within the waste material, particularly as the waste material is drawn into the waste container (often in a turbulent and unpredictable manner) under the influence of a vacuum.

[0007] An imaging device including a camera is configured to capture an image of a waste container. The captured image can be analyzed by one or more processors, or image frames of a video stream captured by the camera can be analyzed by a processor. Thus, the eBL can be continuously updated and streamed on a display. The device support can be removably connected or rigidly fixed to a chassis of the medical waste collection system. The device support can include a shield and at least one arm extending from the shield. One of the arms can include a hinge. A latch or other suitable locking mechanism can be connected to the shield. An aperture provides communication between the rear groove and the front groove. The aperture is defined at a position within the rear groove so as to position the camera and light source at a precise position relative to the waste container. The front groove is sized to receive and support the imaging device.

[0008] A method for estimating the volume of blood in a waste container may include operating in a standby or low power mode. The low power mode may include capturing image frames of a video stream at a lower frame rate using a camera with a flash turned off or at a lower level. The image frames may be pre-processed, and then an activity recognition algorithm may analyze the image frames to detect the pixel-based position of a fluid meniscus in each image. The activity recognition algorithm determines whether the amount (value) of change in the pixel-based position of the liquid meniscus within a predetermined time period is greater than a predetermined threshold. The activity recognition algorithm may compare the pixel-based position of the fluid meniscus between consecutive image frames of the video stream and compare its change with a predetermined threshold. If the change remains below the predetermined threshold, the system will maintain the low power mode. If the change in the fluid meniscus is above the predetermined threshold, the method may include activating a light source of an imaging device or increasing the intensity of a light source of an imaging device, and increasing the feed rate of the video stream.

[0009] In a first variation, the method may include capturing one or more single or multiple exposure photographs for subsequent processing and analysis. In another variation, the method may include analyzing at least one image frame of the video stream according to the algorithm, and preferably analyzing multiple image frames continuously. This step may be performed while waste material is being drawn into the waste container (e.g., the vacuum source is activated), or with the vacuum deactivated, or a combination of both.

[0010] The method includes detecting at least one reference marker, such as a quick response (QR) code. The reference marker can be attached to the outer surface of the waste container. In a memory, the QR code can be associated with calibration-based data associated with the waste container. The camera can be activated in a calibration mode and perform at least one calibration run in which at least one known fluid volume is drawn into the waste container. The processor performs the image-based volume determination to be described, including determining the vertical y-axis value of the fluid meniscus, and the processor associates each y-axis value and known fluid volume with a unique code of the QR code. The calibration data is stored in the memory.

[0011] The method includes a step of aligning the image. The step of aligning the image may be based on the step of detecting the reference marker. The processor is configured to detect the QR code and rotate the image accordingly. Additionally or alternatively, the camera may be configured to detect a mark or a fiducial mark of a waste container or chassis and adjust the image accordingly. The image frame is segmented to determine the centroid of the waste container. The camera may be calibrated and the center of the optical sensor may be determined. The camera calibration may be performed by determining the orientation of the imaging device. The image coordinates may be mapped to standard coordinates. One or more of the adjustment blocks of the QR may be used for transformation to two-dimensional standard coordinates and then to three-dimensional standard coordinates. The image frame cropped in the standard coordinate space may include the at least two reference markers and a portion of the waste container located therebetween. The width of the cropped and mapped image frame is approximately equal to the width of the at least two reference markers. Cropping the adjusted image is optional. The pixel points of the centroid and the pixel points of the center of the optical sensor may be transformed into three-dimensional points in the standard space of the waste container, respectively. The orientation of the waste container and the imaging device determines the mathematical correction. The corrected volume is determined based on the image-based volume determination and the mathematical correction.

[0012] A neural network can be used to segment images or image frames of a captured video stream, including waste materials contained therein. The neural network can characterize whether a pixel belongs to blood, a liquid meniscus, non-blood, and optionally a bubble, foam and / or fluid surface. Each pixel is assigned a value from the neural network and then a class label is assigned based on the value. The location of the fluid meniscus is determined based on the segmentation mask. This step can include fitting a parabola to pixels with a meniscus class label. In an instance where the fluid meniscus is below the camera, the parabola includes a lower vertex and opens upward. If the fluid meniscus is above the camera, the parabola includes a higher vertex and opens downward. The location of the fluid meniscus is the y-axis minimum or maximum of the parabolic curve vertex. Any other polynomial function can be used to approximate the meniscus, and the minimum or maximum point is found accordingly to determine the location of the fluid.

[0013] The method then includes a step of pixel to volume mapping to determine the volume of fluid within the waste container. This step may include retrieving or receiving calibration data from a memory and determining a y-axis value of the fluid meniscus in canonical coordinate form based on the volume of the image, as adjusted by the calibration data.

[0014] In some embodiments, the method includes extracting a region of interest (ROI) from the image. The region of interest can be the first and second imaging surfaces of the insert. The image region associated with the imaging features of the insert is used to quantify the concentration of blood components in the waste material. Other optional steps can include extracting palette colors from the reference markers, eliminating noise, extracting features (e.g., red, green, blue (RGB) values ​​from the insert), and performing light normalization to account for light intensity variations. Light normalization can include training a neural network to predict the color configuration of a reference marker below based on the color configuration of one or more reference markers above, and applying color correction based on the predicted color configuration of the reference marker below. Alternatively, certain aspects of the method are combined and analyzed by a neural network trained on an image data set in these aspects.

[0015] The estimated blood loss can be displayed on one or more displays, such as a touch screen display of the imaging device. The eBL can be displayed and updated at desired intervals or in real time. Specifically, in video mode, the camera can capture a video stream in which each of the blood composition and fluid volume is repeatedly determined in a near-real-time manner. The camera's touch screen display can display the camera's field of view and can also add usage information to the user.

[0016] Therefore, according to one aspect of the present disclosure, a method for estimating blood loss of waste material within a waste container of a medical waste collection system is provided. When a vacuum source of the medical waste collection system draws waste into the waste container, an imaging device captures a video stream of the waste container and the waste material disposed therein. One or more processors analyze image frames of the video stream to determine the volume of the waste material within the waste container. One or more processors analyze the image frames to determine the concentration of blood components within the waste material. The one or more processors estimate the blood loss based on the determined volume and the determined concentration of the blood components. And the estimated blood loss is displayed on a display.

[0017] According to another aspect of the present disclosure, a method for estimating blood loss of waste material within a waste container of a medical waste collection system is provided. When a vacuum source of the medical waste collection system draws waste into the waste container, an imaging device captures a video stream of the waste container and the waste material disposed therein. One or more processors analyze image frames of the video stream to determine the concentration of blood components within the waste material. Analysis of the image frames is facilitated by a neural network trained on an image data set of at least one of the following: the relative positioning of the imaging device's camera relative to the waste container, the position of flashes reflected on the surface of the waste container, color component values ​​associated with at least two reference markers attached to the surface of the waste container, and color component values ​​associated with an imaging surface of an insert disposed within the waste container. Blood loss is estimated based on the volume and the determined concentration of the blood component and displayed on a display.

[0018] In certain embodiments, the neural network is further trained on an image dataset of an area of ​​interest associated with at least two imaging surfaces of the insert, wherein each of the at least two imaging surfaces provides a different color component value for waste material disposed between the insert and the waste container. Alternatively, the neural network is further trained on an image dataset of an area of ​​interest associated with an imaging surface of the insert, thereby providing a color gradient for waste material disposed between the insert and the waste container. The neural network is further trained on an image dataset of at least one of various volumes of waste material and different phase characteristics of the waste material. The image frame can be mapped from the image coordinate space to the canonical coordinate space and provided to the neural network.

[0019] According to yet another aspect of the present disclosure, a method for estimating blood loss of waste material within a waste container of a medical waste collection system is provided. An imaging device captures an image of the waste container and the waste material placed therein. One or more processors are configured to process the image, including: processing the image using the one or more processors, wherein the step of processing the image further includes: determining the position of a fluid meniscus; adjusting the image; mapping the image from an image coordinate space to a canonical coordinate space; and converting a y-axis value of the mapped image in canonical coordinate form to a determined volume. The method includes detecting at least two reference markers attached to the waste container using the one or more processors, wherein the reference markers include position data. A region of interest in the original image is extracted based on the position data. The image is analyzed to determine the concentration of a blood component in the waste material. The blood loss is estimated based on the determined volume and the concentration of the blood component using the one or more processors. The estimated blood loss is displayed on a display.

[0020] In some embodiments, the imaging device captures a video stream at a first frame rate. One or more processors determine a rate at which the volume of waste material within the waste container increases. The one or more processors can detect a pixel-based position of a fluid meniscus in each image frame and determine whether an amount of change in the pixel-based position of the fluid meniscus within a predetermined time period is greater than a predetermined threshold. Prior to the step of detecting the pixel-based position of the fluid meniscus, the image frames can be downsampled. If the rate is greater than the predetermined threshold, the imaging device is configured to capture a video stream at a second frame rate greater than the first frame rate. If the rate is greater than the predetermined threshold, a flash can be activated or its intensity increased. This approach helps reduce power requirements for continued operation of the device.

[0021] In certain implementations, the one or more processors may detect at least two reference markers attached to the waste container. The one or more processors may adjust the image or image frame based on the relative positioning of the at least two reference markers. The at least two reference markers are QR codes arranged in a substantially vertical configuration, wherein the step of adjusting the image frame further comprises adjusting the QR code to be completely vertical.

[0022] The present disclosure also provides a method for correcting the tilt of a waste container of a medical waste collection system and / or the tilt of an imaging device supported in an equipment stand. The imaging device captures an image or video stream of the waste container and the waste material placed therein. One or more processors receive an image frame. The one or more processors detect the position of at least two reference markers within the image frame and determine the orientation of the waste container based on the position of the at least two reference markers. For example, based on an internal gyroscope, the orientation of the imaging device is determined, and the one or more processors determine a mathematical correction based on the orientation of the waste container and the orientation of the imaging device. The mathematical correction is applied to determine a corrected volume of the waste material placed in the waste container.

[0023] The at least two reference markers may be AruCo codes, and the image adjustment may compensate for the tilt of the waste container by determining the orientation of the AruCo code. The AruCo code may be provided on an adjustment frame attached to the waste container. In certain embodiments, the image frame is analyzed to determine the concentration of blood components within the waste material. One or more processors estimate the amount of blood loss based on the correction volume and the determined concentration of the blood components. The estimated amount of blood loss may be displayed on a display.

[0024] In certain embodiments, one or more processors may segment an image or image frame by characterizing whether the pixel values ​​of each image frame are blood, meniscus, non-blood, bubbles on the surface of the fluid, or other phase features of waste material. Class labels are assigned based on pixel values. The location of the fluid meniscus can be determined based on the class label of the meniscus. One or more processors may fit a parabola to pixels with a meniscus class label, wherein the step of determining the location of the fluid meniscus also includes determining the y-axis value of the maximum or minimum value of the parabola. The fluid meniscus may be located below the level of the imaging device so that the parabola opens upward, and the location of the fluid meniscus may be the y-axis value of the minimum value of the parabola. The fluid meniscus may be located above the level of the imaging device so that the parabola opens downward, and the location of the fluid meniscus may be the y-axis value of the maximum value of the parabola. One or more processors may receive container-specific calibration data and map the y-axis value to a reference point. One or more processors may convert the y-axis value into a volume value.

[0025] In some embodiments, an image or image frame can be mapped from an image coordinate space to a canonical coordinate space. A homography of a positioning marker (e.g., four positioning or reference markers) can be generated, and the image coordinates can be transformed into two-dimensional canonical coordinates. The positioning marker and the reference marker can be the same or different. The two-dimensional canonical coordinates can be transformed into three-dimensional canonical coordinates.

[0026] According to another aspect of the present disclosure, a method for calibrating mechanical differences in a waste container of a medical waste collection system includes supporting an imaging device in an apparatus holder. The imaging device captures an image of the waste container and waste material disposed therein in a calibration mode, wherein a vacuum source of the medical waste collection system draws at least one known volume of waste material into the waste container. One or more processors analyze image frames of a video stream to determine a y-axis value of a fluid meniscus of the waste material within the waste container. Calibration data is stored in a memory along with data relating the at least one known value to the y-axis value of the fluid meniscus.

[0027] In certain embodiments, the one or more processors may detect a reference marker attached to the waste container. The reference marker includes optically readable data or code. The one or more processors associate the calibration data with the optically readable data of the reference marker. The one or more processors may analyze image frames of the video stream to determine the volume of waste material within the waste container. The volume is displayed on a display, and the user interface is configured to receive a confirmation input of the volume of the waste material.

[0028] According to yet another aspect of the present disclosure, a method for estimating the amount of blood loss of waste material in a waste container of a medical waste collection system includes supporting an imaging device in an equipment stand. When a vacuum source of the medical waste collection system draws waste material into the waste container, the imaging device captures a video stream of the waste container and the waste material placed therein. One or more processors segment image frames of the video stream by characterizing whether the pixel values ​​of each image frame are blood, meniscus, or non-blood, and optionally whether they are bubbles or fluid surfaces. One or more processors assign category labels to these pixels based on the pixel values, and determine the location of the fluid meniscus based on the category label of the meniscus. One or more processors map the image frames from the image coordinate space to the canonical coordinate space. One or more processors determine the volume of the waste material in the waste container based on the location of the fluid meniscus.

[0029] In some embodiments, the one or more processors detect at least two reference markers attached to the waste container. The mapped image frame is cropped by the one or more processors. The cropped mapped image frame can be located in a canonical coordinate space and include the at least two reference markers and a portion of the waste container located therebetween. The width of the cropped and mapped image frame can be approximately equal to the width of the at least two reference markers. The one or more processors can adjust the image frame based on the relative positioning of the at least two reference markers.

[0030] According to yet another aspect of the present disclosure, there is provided an equipment support for supporting an imaging device to capture images of a waste container of a medical waste collection system. The equipment support includes a shield coupled to a front shell. The shield defines a rear recess configured to cover a window, a front recess, and an aperture providing fluid communication between the rear recess and the front recess. The front recess is sized to receive an imaging device, and the aperture is positioned relative to the front recess to align with a camera and a flash of the imaging device.

[0031] In some embodiments, at least one arm couples the shield to the front housing. The device support includes a hinge that couples the at least one arm and the shield and is configured to allow the shield to be pivoted to a configuration in which the window is visible when the imaging device is disposed therein. The front housing can be movably coupled to the shield. At least substantially the entirety of the waste container is configured to be visible within the field of view of a camera of the imaging device. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A medical waste collection system configured to draw medical waste through an aspiration tube and a manifold for collection into a waste container is shown. An equipment bracket is coupled to a chassis of the medical waste collection system and positioned to removably receive an imaging device configured to capture an image of the waste container.

[0033] Figure 2 is a perspective view of an equipment rack in a first configuration.

[0034] Figure 3 is a perspective view of an equipment rack in a second configuration.

[0035] Figure 4 is an elevation view of another embodiment of an equipment support.

[0036] Figure 5 is a plan view of a waste container with a schematic illustration of the field of view of an imaging device.

[0037] Figure 6 is an elevation view of a waste container with a schematic illustration of the field of view of an imaging device.

[0038] Figure 7 is an illustration of an imaging device capturing a video stream of a waste container while supported within an equipment stand.

[0039] Figure 8 is a perspective view of a waste container including an insert disposed therein. An adjustment frame and a reference marker are coupled to the exterior of the waste container.

[0040] Fig.9A is a method in which one or more images are captured and analyzed by one or more processors to estimate blood loss.

[0041] Fig. 9B is a method in which image frames of a video stream are analyzed by one or more processors to estimate blood loss.

[0042] Fig.10 A waste container with a fluid disposed therein is shown with machine vision output superimposed on reference markers for image adjustment.

[0043] Fig.11 A waste container is shown with a processor identification marker and the external structure of a medical waste collection system chassis for image adjustment.

[0044] Figures 12A-12C Represents the process of adjusting an image using the internal features of a waste container.

[0045] Fig.13 is a method of compensating for the tilt of the waste container.

[0046] Fig.14A and 14B An alternative embodiment of a reference marker for compensating for tilt of a waste container is shown.

[0047] Figures 15A-15FCertain steps of the method are shown as applied to a first image frame being processed during waste material collection in a waste container.

[0048] Figures 16A-16F Certain steps of the method are shown as applied to a second image frame being processed during waste material collection in a waste container.

[0049] Figures 17A-17F Certain steps of the method are shown as applied to the third image frame being processed during waste material collection in a waste container.

[0050] Figures 18A-18F Certain steps of the method are shown as applied to the fourth image frame being processed during waste material collection in a waste container.

[0051] Fig.19 is a partial view of a waste container depicting machine vision output overlaid with a reference marker and associated with the region of interest.

[0052] Fig. 20 Another implementation of a method for estimating blood loss, wherein the method may be performed in a machine learning environment. DETAILED DESCRIPTION

[0053] Figure 1 A medical waste collection system 20 is shown for collecting waste material generated during medical procedures. The medical waste collection system 20 includes a chassis 22 and wheels 24 for moving the chassis 22 within a medical facility. At least one waste container 26 is supported on the chassis 22 and defines a waste volume (or space) for receiving and collecting waste material. In embodiments where there is more than one waste container, an upper waste container 26 can be positioned above a lower waste container 26, and a valve (not shown) can facilitate transfer of waste material from the upper waste container 26 to the lower waste container 26. A vacuum source 30 is supported on the chassis 22 and is configured to draw suction on the waste container 26 through one or more internal lines. The vacuum source 30 can include a vacuum pump and a vacuum regulator, the vacuum regulator being configured to adjust the level of suction drawn on the waste container. The appropriate construction and operation of several subsystems of the medical waste collection system 20 are disclosed in co-owned U.S. Patent No. 7,621,989 (issued on November 24, 2009), U.S. Patent No. 10,105,470 (issued on October 23, 2018), and U.S. Patent No. 11,160,909 (issued on November 2, 2021), the entire contents of which are hereby incorporated by reference herein. The subsequent discussion is conducted with reference to the upper waste container, but it should be understood that the objectives of the present disclosure can be extended to the lower waste container alternatively or simultaneously.

[0054] The medical waste collection system 20 includes at least one receiver 28 supported on the base frame 22. The receiver 28 defines an opening sized to removably receive at least a portion of a manifold 34. A suction path from a suction tube 36 to a waste container 26 can be established by the manifold 34 removably inserted into the receiver 28. In other words, the vacuum generated by the vacuum source 30 draws suction on the suction tube 36, and waste material is drawn from the surgical site through the suction tube 36, the manifold 34, and the receiver 28 to be collected in the waste container 26. The manifold 32 can be a disposable component, wherein exemplary embodiments of the receiver 28 and the manifold 32 are disclosed in commonly owned U.S. Patent No. 10,471,188 issued on November 12, 2019, the entire contents of which are incorporated herein by reference.

[0055] The medical waste collection system 20 includes a fluid measurement subsystem 38, a cleaning subsystem 40, and a container light or backlight. An exemplary embodiment of the fluid measurement subsystem 38 is disclosed in the aforementioned U.S. Patent No. 7,621,898, wherein a float element 79 is movably disposed along a sensor rod 80 (see also Fig.13 ). Based on the signal indicative of the liquid level received from the fluid measurement subsystem 38, the controller 42 in electronic communication with the fluid measurement subsystem 38 is configured to determine the fluid volume of the waste material in the waste container 26. The cleaning subsystem 40 may include an ejector rotatably disposed within the waste container 26 and configured to direct pressurized liquid onto the inner surface of the waste container 26, as disclosed in the aforementioned U.S. Patent No. 10,105,470. Finally, the container backlight is configured to illuminate the interior of the waste container 26. The container backlight can be activated based on input to the user interface 52 or another device in communication with the controller 42.

[0056] The chassis 22 includes a front housing 46 that defines at least one cutout or window 48 to expose a portion of the waste container 26. The waste container 26 may be made of a transparent material through which a user may visually observe the waste material collected within the waste container 26 and, if desired, visually estimate the volume of the waste material collected therein by volume markings disposed on an outer surface of the waste container 26 (see FIG. Fig.13). The waste container 26 is optically transparent and also allows the waste material collected therein to be imaged by the camera 50. The camera 50 is also referred to herein as an imaging device, wherein the imaging device includes the camera 50, a user interface (e.g., a touch screen display), and an optional flash or light source. The images from the camera 50 can be transmitted to and processed by one or more processors 44 (mentioned below without emphasis on quantity) to determine the blood components within the waste material. The blood components can be the blood concentration (e.g., hemoglobin concentration) within the waste material. More specifically, the optical properties of the waste material can be analyzed and processed to determine the blood components, as disclosed in co-owned U.S. Patent No. 8,792,693, issued on July 29, 2014, the entire contents of which are incorporated herein by reference. The blood volume within the waste material (i.e., the amount of blood loss in this patent) can be estimated based on the determined blood components and the volume of the waste material.

[0057] Further references Figure 2 and Figure 3 , the device support 54 is removably coupled or rigidly fixed to the base frame 22. The device support 54 positions the camera 50 relative to the waste container 26 in a precise manner to provide continuous image capture (e.g., video stream) of at least substantially the entire waste container 26. The video stream (video feed) generates continuous data, from which the continuous data can determine the fluid volume and blood components therein in real time. Therefore, the estimated blood loss (eBL) can be continuously updated and streamed on a display (e.g., a touch screen display of the imaging device 50, a user interface 52 of the medical waste collection system 20, and / or another display terminal). These advantages are not easily achieved for handheld imaging devices because handheld imaging devices require the user to manually support the camera during image capture and manually input the estimated volume based on visual observation. In addition, image-based volume determination eliminates the need for the imaging device 50 to communicate data with the controller 42 of the medical waste collection system 20. Further, the precise positioning can be designed to reduce glare and eliminate changes or deviations in lighting to improve the accuracy of the algorithm determination.

[0058] The device support 54 can be removably coupled or rigidly secured to the front shell 46 or another suitable surface or internal structure of the chassis 22. For example, a complementary coupler (e.g., a structure in which a flange engages in a groove, a detent positioning device, etc.) can facilitate the removable coupling of the device support 54 to the chassis 22. The device support 54 can include a shield 56 that is configured to be disposed above the window 48 defined by the front shell 46. At least one arm 58 can extend from the shield 56 and be coupled to the chassis 22. In some embodiments, one of the arms 58 can include a hinge 60 that is configured to allow the device support 54 to move between first and second configurations in which the shield 56 covers the window 48 and does not cover the window 48, respectively. For example, in certain procedures, it may be necessary to forego the use of the eBL, and / or otherwise view the contents of the waste container 26. A latch or other suitable locking mechanism coupled to the shield 56 can be actuated, and the shield 56 can be pivoted about the hinge 60 to a second configuration, such as Figure 3 For subsequent procedures or as needed, the user can pivot the shield 56 about the hinge 60 to return the device support 54 to the first configuration.

[0059] The device holder 54 can include a rear recess 62, a front recess 66, and an aperture 64 that provides communication between the rear recess 62 and the front recess 66. The size and shape of the rear recess 62 can be designed to cooperate with an edge or lip disposed within the window 48 of the front shell 46 to limit or prevent ambient light from entering. Alternatively, a resilient flange can extend from the rear of the shield 56 to engage the front shell 46 to achieve the above purpose. The aperture 64 is defined at a position within the rear recess 62 to position the camera 50 and the light source (e.g., a flash of the imaging device) in a precise position relative to the waste container 26, in a manner that will be further described. The size of the front recess 66 is designed to receive and support the imaging device 50. The size of the front recess 66 may vary depending on the type of imaging device 50 itself. In other words, the imaging device 50 can be a smartphone, a tablet computer, or a custom-designed device, and variations of the device holder 54 can be compatible with specific models of the device (e.g., Apple Samsung Google The embodiment of the equipment support 54 shown in the figure can be used with the operation Apple iPhone 13 operating system In such an arrangement, the device holder 54 is sized so that the device holder 54 is placed on the Apple iPhone 13 The camera and flash at the upper corner are in precise position relative to the waste container 26. In this arrangement, the front recess 66 is positioned lower than the rear recess 62. In addition, the front recess 66 can be sized to support the imaging device 50 in the protective shell. In some embodiments, the front housing 70 can be movably coupled to the shield 56 by a second hinge and latch mechanism (not shown) that pivotally couples the front housing 70 to the shield 56.

[0060] Figure 4 Another embodiment of the device holder 54 is shown in which a geometric structure associated with the window 48 of the front shell 46 supports the device holder 54. For example, there may be a small gap or slot around the perimeter of the window 48 between the front shell 46 and the outer surface of the waste container 26. Certain components of the device holder 54 can be integrated with the waste container 26 so that the device holder 54 is placed inside the front shell 46 and outside the waste container 26. The size and position of the aperture 64 are designed to align with the camera 50 and the flash. The size and shape of the lip 72 of the device holder 54 can be designed to receive the base of the camera 50 and position the camera 50 against the shield 56 so that the camera and flash are aligned with the aperture 64. Other ways of supporting the camera 50 are also contemplated, such as magnets, clips, straps, etc. The device holder 54 can be positioned on the chassis 22 to avoid obstructing regulatory labels or other components of the medical waste collection system 20 that require accessibility.

[0061] As described above, the equipment support 54 supports the camera 50 so that the waste container 26 is within the field of view of the camera. In some embodiments, the entire waste container 26 is within the field of view. Figure 5 and Figure 6 , a representation of the device holder 54 is shown with an aperture 64 identified as being proximate to the location of the camera 50. The dimensions of the device holder 54 can be designed so that the field of view of the camera 50 spans the entire width of the waste container 26 and, optionally, provides a desired spacing between the flash of the imaging device 50 and the surface of the waste container 26. The spacing of this design can be adjusted to reduce glare while adequately illuminating the waste container 26 with the flash of the imaging device 50. In addition, the field of view of the camera 50 can span from below the bottom of the waste container 26 to the top of the waste container 26 or above its predefined fill level. Figure 6The field of view of the camera 50 is shown to include up to approximately 4,000 mL of fluid within the waste container 26. The distance that the camera 50 is spaced from the surface of the waste container 26 can be based on the aspect ratio of the camera 50 (e.g., 4:3, 16:9, etc.). In one example, precise positioning includes spacing the camera 50 from the surface of the waste container 26 a distance in the range of approximately 30 millimeters (mm) to 70 mm, more specifically, in the range of approximately 40 to 60 mm, and even more specifically approximately 50 mm. In some examples, the horizontal viewing angle can be in the range of approximately 80° to 90°, and the vertical viewing angle can be in the range of approximately 100° to 105° - the corresponding lens viewing angle can be in the range of approximately 110° to 120°. Additional optical components such as fisheye lenses and mirrors can be used on the device support 54 to expand the field of view of the camera 50. When the camera and flash are directed toward the waste container 26, the touch screen display of the imaging device 50 remains visible and operable, including displaying the camera's field of view or an increased field of view, such as Figure 7 It will also be appreciated that this arrangement facilitates removal of the camera 50 from the device holder 54 and replacement within the device holder 54 in a quick and intuitive manner.

[0062] Reference now Figure 8 , the insert 74 is arranged in the waste container 26 so as to be within the field of view of the camera 50. The insert 74 includes several geometric structures, at least one of which is an imaging surface 75, which is spaced apart from the inner surface of the waste container 26 to define a gap having a known and fixed (unchanging) distance. The gap allows a thin layer of fluid to be located between the insert 74 and the inner surface of the waste container 26, which presents an area of ​​at least substantially uniform color, wherein the color intensity is lower than the color intensity that causes signal saturation. Exemplary embodiments of the insert 74 are disclosed in the aforementioned U.S. Patent No. 9,773,320 and the co-owned International Patent Application No. PCT / US2013 / 015437 (filed on March 17, 2023), the entire contents of which are incorporated herein by reference. The insert 74 can be mounted to the container cover 82 of the waste container 26.

[0063] The imaging surface 75 may include first and second imaging surfaces 75a, 75b positioned laterally relative to each other in a side-by-side arrangement (i.e., a multi-level insert). When the support of the insert 74 directly contacts the inner surface of the waste tank 26, the first imaging surface 75a is spaced apart from the inner surface by a first distance, and the second imaging surface 75b is spaced apart from the inner surface by a second distance greater than the first distance. In one example, the first distance is 1.7 mm and the second distance is 2.2 mm. More broadly, it is contemplated that the first distance may be approximately in the range of 0.7 to 5.7 mm, more specifically, in the range of 1.2 to 3.7 mm, and the second distance may be approximately in the range of 1.2 to 6.2 mm, more specifically, in the range of 1.7 to 4.2 mm. The first imaging surface 75a and the second imaging surface 75b may be separated by a ridge having a thickness equal to the difference between the first distance and the second distance. For example, the ridge may be approximately 0.5 mm. It should be understood that the insert 74 may include two, three, four, or five or more imaging surfaces, wherein the illustrated embodiments are non-limiting examples. Alternatively, the imaging surface 75 may also comprise a continuously gradient imaging surface, for example having a wedge shape, to allow for the measurement of several levels of increasing fluid color intensity.

[0064] Among other advantages, the side-by-side arrangement also provides improved lockability of the insert 74. The ejector of the cleaning subsystem is rotatably coupled to the container cap 82 of the waste tank 26, and the ejector directs the liquid downward and radially outward (schematically indicated by arrows) toward the inner surface. The rotatable ejector of the cleaning subsystem directs the pressurized fluid toward the insert 74. The relative spacing of the first and second imaging surfaces 75a, 75b can be based on the rotation direction of the ejector of the cleaning subsystem. More specifically, the first and second imaging surfaces 75a, 75b can be arranged so that the pressurized liquid first contacts the first imaging surface 75a (i.e., closer to the inner surface) and then contacts the second imaging surface 75b. If there are any semi-solid or solid debris between the insert 74 and the inner surface, the aforementioned flow direction will increase the possibility of debris removal. Therefore, a thin layer of medical waste is better washed away from the gap between the imaging feature 75 and the inner surface, and the improved cleaning can limit (or reduce) staining or otherwise maintain the optical characteristics of the insert 74.

[0065] At least one reference marker 76 may be detected by the camera 50 for locating an image region associated with an imaging feature of the insert 74 and for color correction for changes in lighting, flash, or other optical aberrations. One suitable embodiment of a reference marker is disclosed in commonly owned U.S. Patent No. 9,824,441, issued November 21, 2017, the entire contents of which are incorporated herein by reference, wherein a quick response (QR) code of a known red component value (or RGB, HSV, or CMYK color scheme, etc.) is attached to the outer surface of the waste container 26 with an adhesive, corresponding to the location of the upper portion of the insert 74. In addition, in a manner to be described, calibration data may be associated with the unique code of the reference marker 76 to account for mechanical differences in the waste container 26. Furthermore, in a preferred embodiment, at least two reference markers 76 may be attached to the waste container 26 in a manner that facilitates image adjustment.

[0066] Reference now Fig.9A and 9B , a method 100 for estimating the volume of blood within a waste container 26, also referred to herein as estimated or estimated blood loss, is provided. The method 100 disclosed herein and one or more steps thereof are configured to be executed by a processor 44 in accordance with instructions stored on a non-transitory computer-readable medium. The data may be analyzed by the processor 44 on the imaging device 50, and / or the data may be transmitted for remote processing (e.g., cloud computing). The method 100 may be executed using a machine learning (ML) model, or may implement one or more trained neural networks. The computer executable instructions may be implemented using an application, an applet, a host, a server, a network, a website, a communication service, a communication interface, hardware, firmware, software, etc. The computer readable medium may be stored on any suitable computer readable medium, such as RAM, ROM, flash memory, EEPROM, an optical device, a hard drive, a floppy drive, etc.

[0067] The neural network can execute an algorithm trained using supervised, unsupervised, and / or semi-supervised learning methods to perform the methods of the present disclosure. A diverse and representative set of training data sets is collected to train each neural network in a variety of scenarios within and outside the intended use of the system 20. The training methods may include data enhancement, pre-trained neural networks, and / or the use of semi-supervised learning methods to reduce the requirements for labeled training data sets. For example, data enhancement can increase the sample size, such as different blood concentrations, semi-solids, different degrees of contamination on the side walls of the waste container 26, lighting conditions (brightness, contrast, hue saturation shift, PlanckianJitter, etc.), geometric transformations (rotation, homography, flipping, etc.), noise and blur addition, custom enhancements, hemolysis, placement of the imaging device 50, coagulation, detergents, suction levels, etc. In addition, tolerance limit testing can also be performed to determine the permissible acceptable changes in parameters such as insertion depth, tank tilt, and mechanical differences of the waste container 26.

[0068] The method 100 may include operating the system in an optional standby or low power mode (step 102). As the name implies, the low power mode is configured to maintain power (e.g., the battery life of the imaging device 50) during a period of time when the liquid level in the waste container 26 does not change. For example, the software of the imaging device 50 can be started before the start of a surgical procedure, but the medical waste collection system 20 will not suck waste material into the waste container 26. The low power mode 102 can include using the camera 50 to capture a video stream (i.e., a series of image frames) at a relatively low frame rate with the flash turned off (step 104). In one example, the frame rate is about 15 frames per second (fps), but other frame rates are also within the scope of the present disclosure. The image frames (also referred to herein as images) are pre-processed (step 106), such as down-sampling. The down-sampled images are analyzed by an activity recognition algorithm (step 108). The activity recognition algorithm detects the pixel-based position of the liquid meniscus in each image. The activity recognition algorithm is configured to determine whether the amount of change in the pixel-based position of the fluid meniscus within a predetermined time period is greater than a predetermined threshold. In other words, the activity recognition algorithm determines whether waste material is being drawn into the waste container 26 at a predetermined rate. In one example, the activity recognition algorithm may compare the pixel-based position of the fluid meniscus between consecutive image frames of the video stream and compare its change to a predetermined threshold. If the change remains below the predetermined threshold, the processor 44 maintains the system in the low power mode and abstains from executing the remainder of the method 100. If the change in the fluid meniscus is above the predetermined threshold, the processor 44 executes the remaining steps of the method 100. Additionally or alternatively, the user may provide input to a touch screen display of the imaging device 50, a paired external device, etc. to terminate the low power mode and initiate the eBL mode.

[0069] The method 100 includes activating a light source (step 110), such as activating a flashlight of the imaging device 50. This step is optional, and the flashlight of the imaging device 50 may alternatively be continuously activated. In another variation, the eBL mode may be utilized to increase the intensity or level of the flashlight of the imaging device 50. The container backlight of the waste container 26 may already be activated. Using light from both the flashlight and the container backlight may provide optimal illumination while minimizing glare. Notably, due to the precise position of the camera 50 supported by the device support 54, glare may be limited to a small point positioned above the insert 74 and the reference marker 76 (see Fig.19 and 20 ), which is an image area of ​​less concern for image analysis. In addition, a larger signal-to-noise ratio (e.g., blood color signal vs. background reflection) has less impact on image analysis, so a separate algorithm may not be needed to reduce or eliminate the effects of glare. In addition to activating the flash, the feed rate of the video stream can also be increased to improve data resolution.

[0070] The method 100 may include analyzing at least one of the image frames of the video stream (step 112), and preferably analyzing a plurality of image frames continuously according to the algorithm. In other words, each image frame may be analyzed, every other image frame may be analyzed, every third image frame may be analyzed, etc., depending on available computing resources and desired data resolution. In an alternative embodiment that does not utilize a video stream, the method 100 may include the step of capturing one or more multi-exposure photographs for subsequent analysis.

[0071] The method 100 includes a step of detecting a reference marker 76 (step 114), such as a QR code, a bar code, or another marker having optically readable data. Among other aspects to be described, the QR code can be used to account for mechanical differences in the volume of the waste container 26. In other words, manufacturing tolerances may cause slight variations in the internal volume of the waste container 26 and the volume of the insert 74 and insert mount located therein, thereby causing inaccuracies in the image-based volume determination. In addition, slight variations in the internal volume may be caused by vacuum-induced variations, wherein the pressure differential caused by the vacuum in the waste container 26 causes the side walls of the waste container 26 to "bend" inwardly and displace the waste contained therein. Further, the body temperature fluid from the patient in the waste container 26 is at a higher temperature relative to the environment, so thermal expansion of the waste container 26 may cause additional slight variations in the internal volume.

[0072] The present disclosure solves these problems by associating in memory the QR code with calibration-based data associated with the waste container 26. The calibration can be performed by a technician during deployment (e.g., assembly or modification) of the eBL subsystem with the medical waste collection system 20. The technician can attach at least two QR codes, preferably using a method such as Fig.10 The device holder 54 is mounted in the manner previously described. The camera 50 is activated in a calibration mode and at least one calibration run is performed, which draws at least one known fluid volume into the waste container 26. The processor 44 performs the image-based volume determination to be described, including determining the vertical y-axis value of the fluid meniscus, and the processor 44 associates each y-axis value and the known fluid volume with the unique code of the QR code 76. The calibration data is stored on the memory 88 (e.g., the memory of the imaging device 50 or a cloud-based memory). In operation, the camera 50 detects the QR code 76, and the processor 44 accesses the calibration data from the memory 88 to perform an image-based volume determination adjusted by or taking into account the calibration data.

[0073] Alternative methods for container-specific calibration are contemplated. In a first example, an on-site calibration may be performed, wherein a calibration mode displayed by the imaging device 50 guides the user to aspirate a target volume of fluid. After doing so, an input is provided to the user interface, and the processor 44 performs the image-based volume determination to be described, including determining the vertical y-axis value of the fluid meniscus, and the processor 44 associates each y-axis value. The user interface displays the predicted volume, and the user may provide a confirmation input to the user interface to indicate that the predicted volume as determined by the processor 44 is equal to the target volume of the fluid. In a second example, the reference marker 76 may be attached at a precise location and confirmed by testing (e.g., in a laboratory setting). The camera 50 is activated in calibration mode, the y-axis position of the reference marker 76 is determined and associated with the unique code of the reference marker 76. In a third example, container-specific calibration data may be stored on a memory chip, such as a near field communication (NFC) tag. The NFC tag may be detected by a complementary NFC reader of the imaging device 50, and the calibration data is transmitted to the imaging device 50. The calibration data is stored in the memory 88 together with the unique code of the reference marker 76. This alternative approach may require precise placement of the reference markers 76. The step of compensating for volume changes in the waste container 26 is optional, and the image-based volume determination may be sufficiently accurate even without this step.

[0074] The method 100 includes the step of adjusting the image (step 116). Since the evaluation of the fluid meniscus is on the vertical y-axis, it is desirable that the image of the tank be oriented vertically with sufficient accuracy. In other words, the step of adjusting the image can account for mechanical differences in the rotational positioning of the waste container 26 within the chassis 22. The adjustment can be based on features external to the waste container 26 (i.e., external adjustment), features associated with or inherent to the waste container (i.e., internal adjustment), or a combination thereof. Fig.10 The step of adjusting the image can be based on the step of detecting the reference marker 76. For example, each QR code can be attached to the waste container 26 using a clamp or guide to vertically arrange it with sufficient vertical accuracy. The processor 44 is configured to detect the QR code (such as Fig.10 The image is rotated accordingly. Adjustment techniques for orienting and positioning the image in three dimensions within the frame are within the scope of the present disclosure, including homography, perspective transformation, 3D-2D mapping, etc. Fig. 15B , 16B , 17B and 18B respectively show the Fig.15A , 16A , 17A and 18A image rotation effect.

[0075] Additionally or alternatively, the camera 50 may be configured to detect indicia or fiducial markings of the waste container 26 or chassis 22 and adjust the image accordingly. Fig.11 26. The machine vision output is shown with the internal structure 78 of the chassis 22 superimposed on the image. Additionally or alternatively, external indicia may be provided at fixed locations on the waste container 26, such as printed fiducial markers, such as additional QR codes or April tags attached to known locations on the waste container 26. Other reference structures are also contemplated, such as a sensor rod 80 of the fluid measurement subsystem, a mount for the insert 74, a subcomponent of the container cover 82.

[0076] Internal adjustments may include the processor 44 detecting, determining or identifying a container identifier. Figures 12A-12C, the container mark may include a lower edge 86', at least one side edge 84', and / or other identifiable marks on the waste container 26. These edges can be identified by comparing adjacent pixels, pixel clusters, or portions of the image. Sufficient contrast or contrast in the comparison can indicate a transition, i.e., an edge, from the internal structure of the chassis 22 to the waste container 26. The lower edge 86' can be an arc-shaped boundary associated with a proximal portion of the base of the waste container 26. This step can implement any suitable machine vision technology and / or machine learning technology to identify the edge and / or estimate the characteristics or size of the waste container 26. The processor 44 is configured to adjust the image to a predetermined alignment arrangement. For example, the processor 44 can be configured to orient the side edge to be substantially vertical, represented by reference numeral 84. For another example, the processor 44 can be configured to align the lowest portion of the lower edge 86' with the lower boundary of the frame of the image, represented by reference numeral 86. It should be understood that external adjustment can be used alone or in combination with internal adjustment.

[0077] The image adjustment step 116 may also include compensating for any tilt (e.g., misalignment in the longitudinal direction of the front and rear) of the waste container 26. Tilt may be caused by uneven floor surfaces, manufacturing variations, etc. FIG. 12 illustrates a tilt correction method 118. The method 118 includes the processor 44 receiving an image frame (step 120) and detecting the reference marker 92 (step 122). Referring again to FIG. Figure 8 , the reference markers 92 are arranged to facilitate determining the tilt relative to the image plane. For example, the markers 92 can be AruCo markers, wherein the adjustment frame 90 is attached to the waste container 26 and the contour is designed to match the curvature of the waste container 26, and the markers 92 are set on the four sides of the frame 90. The corner positioning provided by the AruCo markers is better than the corner positioning that can be achieved by QR codes. In one variation, the AruCo markers are not set on the frame 90, but a separate AruCo is attached to the predetermined position of the waste container 26. For example, Fig.14A and 14B A variation is shown in which the reference marker 92 has a color reference palette printed thereon as well as a human readable code and is positioned to limit or reduce unnecessary obstruction of the waste container 26 and volume indicia.

[0078] The image frame is segmented to determine the center of mass of the waste container 26. The method 100 may include the steps of performing camera calibration and determining the center of the optical sensor. The camera calibration may be performed by determining the orientation of the imaging device 50. When the imaging device 50 is an iPhone 13 In an exemplary embodiment, the orientation is determined based on an internal gyroscope and optionally by points associated with detection of a reference marker 92. The method 100 also includes a step of mapping image coordinates to canonical coordinates (step 124). The pixel point of the centroid and the pixel point of the center of the optical sensor can be converted to a three-dimensional point in the standard space of the waste container 26, respectively. From the orientation of the waste container 26 and the imaging device 50, a mathematical correction is determined. For example, the mathematical correction can be a coefficient associated with the posture of the waste container 26 to minimize a least squares formula. A correction volume is determined (step 128) based on the image-based volume determination (step 126) and the mathematical correction.

[0079] Reference again Fig.9A and 9B And further reference Fig.10 , method 100 includes the step of segmenting an image of a waste container 26 (including the fluid contained therein). Each neural network (the number of neural networks will not be emphasized below) can characterize whether a pixel is blood (B), a liquid meniscus (M), or non-blood (NB). In an optional variation, the neural network can further characterize whether a pixel is a bubble or foam (F), and / or a fluid surface (FS). In other words, each pixel is assigned a value from the neural network and then a class label is assigned based on the value. The neural network can be a deep learning neural network, such as U-Net, Mask-RCNN, DeepLab, etc., or any image-based segmentation algorithm, such as image segmentation, clustering, region growing, etc. By way of further example, neural networks can be used for object localization, segmentation (e.g., edge detection, background subtraction, algorithms based on image segmentation, etc.), measurement, clustering, pattern recognition, template matching, feature extraction, descriptor extraction (e.g., extraction of texture maps, color histograms, HOG, SIFT, MSER (maximum stable extreme regions for removing spot features from selected regions, etc.), feature dimensionality reduction (e.g., PCA, K-Means, linear discriminant analysis, etc.), feature selection, thresholding, localization, color analysis, parametric regression, non-parametric regression, unsupervised or semi-supervised parametric or non-parametric regression, neural network and deep learning based methods, or any other type of machine learning or machine vision. Fig.15D , 16D , 17D and 18D show representations of the segmentation masks with different colors for non-blood, meniscus and blood class labels. Fig.16D and 18D Reference marker 76 is also shown to be excluded from the segmentation mask.

[0080] The method 100 includes determining the location of the fluid meniscus (step 132). Step 132 is based on the segmentation mask and may include post-processing the segmentation using the entire width of the image. The aforementioned machine learning techniques may be implemented to train a segmentation network on the video stream to identify the meniscus under different conditions, including blood concentration levels, thickeners, illumination by a flash light, illumination by a container backlight, vacuum level of the vacuum source 30, foaming, turbidity, or opacity of the waste material 26, etc. Other means by which the meniscus can be detected are disclosed in commonly owned U.S. Patent No. 8,983,167, issued on March 17, 2015, the entire contents of which are incorporated herein by reference.

[0081] Step 132 may include fitting a parabola to pixels having a meniscus class label. Fig.15E , 16E , 17E and 18E show representations of the fluid meniscus where the non-blood and blood class labels have been removed. In the case where the fluid meniscus is below the camera 50, the parabola includes a lower vertex and opens upward. Conversely, if the fluid meniscus is above the camera 50, the parabola includes a higher vertex and opens downward. The position of the fluid meniscus is the y-axis value of the vertex of the parabola curve. Therefore, at the lower vertex, the position of the fluid meniscus is the minimum of the parabola curve; and at the higher vertex, the position of the fluid meniscus is the maximum of the parabola curve. Fig. 15B , 16B , 17B and 18B show representations of the machine vision output 94 indicating the y-axis values ​​of the vertices superimposed on the adjusted image of the waste container 26 .

[0082] Method 100 also includes a step of mapping the image from the image coordinate space to the canonical coordinate space (step 124), as previously introduced. Step 124 may require four positioning markers (e.g., two QR codes) to generate homographies and map the fluid meniscus from the image coordinates to the canonical coordinates. One or more of the adjustment blocks of the QR code (e.g., the upper left adjustment block) may be used to transform to two-dimensional canonical coordinates and then to three-dimensional canonical coordinates. Fig. 15C , 16C , 17C and 18C illustrate a representation of the machine vision output 94' in canonical coordinate form, indicating the y-axis values ​​representing the vertices superimposed on the adjusted image of the waste container 26, Fig.15F , 16F, 17F and 18F illustrate representations of a cropped adjusted image of the waste container 26 after superimposing the machine vision line 94'. The cropped image frame in the canonical coordinate space may include the at least two reference markers and a portion of the waste container located therebetween. The width of the cropped and mapped image frame is approximately equal to the width of the at least two reference markers. Cropping of the image limits the range within which the meniscus can be positioned (e.g., the width of the image). Cropping of the adjusted image is optional.

[0083] The method 100 includes a step of pixel to volume mapping (step 126) to determine the volume of fluid within the waste container. This step may include extracting or receiving calibration data from the memory 88, such as container-specific coefficients as described above for accounting for container-specific differences. The image-based volume determination is the y-axis value of the fluid meniscus (in canonical coordinate form) adjusted by the calibration data. In other words, the lowest point of the meniscus (along the y-axis defined in the image by the processor 44) can be mapped relative to a datum and the y-axis position of the meniscus converted to a volume in milliliters. Again, the above-described machine learning techniques can be implemented to train a segmentation network to convert the y-axis position of the meniscus into a fluid volume. As an example and with reference to Fig. 16C , a y-axis value of "2000" in canonical coordinates (left axis of the graph) may result in a volume determination of approximately 1,000 mL.

[0084] Method 100 may include extracting a region of interest (ROI) from the image (step 134). Fig.19 , the processor 44 is configured to identify the reference marker 76 and process the information contained therein to determine the corresponding position of the region of interest 96'. In other words, for example, the data within the QR code may enable the processor 44 to analyze the region of interest 96' of the image frame at a predetermined position relative to the QR code. In addition, in embodiments where the insert 74 includes first and second imaging surfaces 75a, 75b, the reference marker 76 may contain such data that enables the processor 44 to analyze the first and second regions of interest 98a, 98b, respectively.

[0085] In some embodiments, further optional step 136 may include extracting palette colors, eliminating noise, extracting features, and performing light standardization from the reference marker 76. Palette extraction, noise elimination, and feature extraction may be performed in a manner at least similar to that of the aforementioned U.S. Patent No. 9,824,441. Differences in illumination may be taken into account by using an image training segmentation network, in which only illumination from the camera 50 flash, only light from the container background light, or a combination of both at different levels is used. Among other advantages, this may allow blood components to be determined over a greater blood concentration range without signal saturation. Each light intensity level provided by the camera 50 flash may have a known brightness, color temperature, spectrum, etc. Additionally or alternatively, the container background light may provide light of a known brightness, color temperature, spectrum, etc.

[0086] Light normalization according to known solutions can be used primarily to account for differences in ambient lighting. By supporting the camera 50 and the flash of the imaging device in the device holder 54 and configured to be detected in an insert 74 disposed near the bottom of the waste container 26, the effect of the directional light becomes relatively more pronounced. Differences in the relative positioning between the camera 50 and the waste container 26 and the gradual change in the light from the flash of the imaging device 50 result in uncertain nonlinear relationships, such that empirical determinations and machine learning models may result in less accurate determinations of the concentration of blood components in the waste material. For example, slight differences in the posture of the imaging device 50 may result in different positions and intensities of the flash reflected on the surface of the waste container 26, and different qualities (e.g., color component values) of the image or image frame of the waste material. In other words, the light intensity as detected by the camera 50 may depend on the position of the flash on the surface of the waste container 26. Therefore, in certain embodiments, the analysis of the image or image frame to determine the concentration of the blood component can be facilitated by training a neural network on an image data set of various positions of the camera 50 relative to the waste container 26, wherein the various positions of the flash of the imaging device 50 are reflected on the surface of the waste container 26.

[0087] Thus, in certain embodiments, light normalization may require at least two reference markers, each with a known color configuration. In one example, a neural network may be trained to predict the color configuration of a lower reference marker 76 based on the color configuration of one or more upper reference markers 76. Light normalization may include applying color correction based on the color configuration predicted for the lower reference marker 76. In another example, by training a neural network on an image dataset of imaged color component values ​​of at least two, at least three, or four or more reference markers 76, an image or image frame used to determine blood component concentrations may be facilitated.

[0088] The images captured by the camera 50 may be captured under lighting conditions of constant intensity and other characteristics. Alternatively, step 106 is optional and may include capturing images using the camera 50 at different light intensity levels provided by a flash of the camera 50, wherein the processor 44 utilizes an algorithm that compensates for the various light intensity levels. For example, a first image is captured at a high flash intensity, a second image is captured at a medium flash intensity, a third image is captured at a low flash intensity, and so on, thereby allowing for an increased dynamic range of the captured images and, associated therewith, an increased range of blood components that may be measured by the algorithm. Another method of light normalization includes intensity gradient normalization, wherein the distance and / or angle of the flash is taken into account.

[0089] The method 100 includes step 138, which may also include analyzing the image area associated with the imaging feature of the insert 74 to quantify the concentration of blood components in the waste material. The analysis can be performed in the manner disclosed in the aforementioned U.S. Patent No. 8,792,693, wherein a parametric model or template matching algorithm is executed to determine the concentration of blood components associated with the fluid within the waste container 26. Specifically, the processor 44 can be configured to extract color component values ​​(e.g., red values) from the image and execute a trained algorithm to determine the concentration of blood components pixel by pixel or in other suitable manners. The hemoglobin (Hb) or blood loss (eBL) is estimated using a hemoglobin estimation algorithm (step 140) based on the image-based volume determination and the concentration of the blood components. Step 140 can be the product of the two, or based on another suitable calculation, such as Equation 1.

[0090]

[0091] Fig. 20 A variation of the method 100 in a machine learning environment is shown, in which certain features are extracted using the adjusted image and the positions of the reference markers 76, and light normalization is performed in color space. Fig. 9B Variations of the method 100 include incorporating certain steps by analyzing image frames of a video stream through a neural network. Specifically, by training the neural network on a diverse and representative training data set, the steps of region of interest extraction, color palette extraction, noise removal, feature extraction, and illumination normalization may not be discrete steps performed by method 100, but may be performed by the neural network in other ways without losing accuracy in hemoglobin estimation.

[0092] In another variation, Fig.9A The method 100 shown in FIG. 1 can be implemented in the boundary case to be described. Fig. 9BThe method 100 shown in the figure may be implemented in other ways by default. A boundary case may be a case where the volume of waste material is less than a predetermined volume. The predetermined volume may be based on the level of the waste material not being above the insert 74 disposed within the waste container 26. For example, the predetermined volume may be less than about 1,000 mL, which may include 800 mL of pre-filled liquid and an additional 400 mL of waste material collected during the surgical procedure. Below the predetermined volume, the ML model implemented Fig.9A (and Fig. 20 ) may be more suitable for these threshold situations. Otherwise, when the fluid volume is higher than the predetermined volume, the neural network implemented by the training Fig. 9B The method 100 may be more accurate over a wider range of blood concentrations. The processor 44 may be configured to implement any variation of the method 100 based on fluid volume or other considerations or determinations.

[0093] The estimated blood loss can be displayed on one or more displays (step 142), such as a touch screen display of the imaging device 50. Additionally or alternatively, the eBL can be wirelessly transmitted to the user interface 52 of the medical waste collection system 20 or another display terminal in the operating room. It can be easily understood from the foregoing disclosure that the eBL can be displayed and updated at desired intervals or in real time. Specifically, in video mode, the camera 50 can capture a video stream in which each of the blood components and fluid volume is repeatedly determined in a nearly instant manner. In addition, the touch screen display of the camera 50 displays the field of view of the camera, so the ability to visualize the internal volume of the waste container 26 is generally unimpeded, and usage information can also be added to the user.

[0094] It is contemplated that in certain embodiments, the camera 50 may be paired with the medical waste collection system 20 (and more specifically the controller 42) for two-way wireless communication. In such an arrangement, the fluid measurement subsystem may be used as an alternative to or in conjunction with image-based volume determination. Data from the fluid measurement subsystem may also be utilized to improve machine learning. If the patient's blood loss exceeds a predetermined or selected limit, an alert may be provided via the user interface 52. Blood loss data associated with the medical procedure may be transmitted to an electronic medical record.

[0095] The present disclosure contemplates additional alarms or guardrails. In certain embodiments, the camera 50 and processor 44 may be configured to determine and provide event detectors regarding the state of the waste container 26. In other words, the machine learning algorithm may be extended to determine whether suction is active or inactive, whether waste material is flowing into the waste container 26, whether the waste container 26 is static or emptying, and the like. Similarly, the machine learning algorithm may be extended to determine whether a blood clot or other debris is lodged near or in front of an imaging feature of the insert 74, and / or whether the insert 74 is missing, displaced, or otherwise improperly positioned beyond a predetermined threshold. Although a decrease in the concentration of a blood component is permitted, a false alarm may be provided if the processor 44 determines that the eBL has decreased.

[0096] Additionally or alternatively, a reference image of the insert 74 may be used to determine blood composition, e.g., a first image prior to a surgical procedure in which there is no blood in the container. The reference image may be compared to some of all subsequent images. For example, the reference image may be used to compensate for variations in color of imaging features of the insert 74 and differences in light intensity caused by repeated soiling and cleaning cycles during multiple uses within the waste container 26. It is noteworthy that the blood concentration ratio based on each image and the reference image may not be directly dependent on light intensity.

[0097] In certain embodiments, the machine learning algorithm can be extended to determine haze on the inner or outer surface of the waste container 26, damage to the outer surface of the waste container 26 (e.g., scratches), dirtiness of the lens of the camera 50, etc. For example, by capturing these images (such as reference images) before and during each procedure, the rate of dirtiness of the waste container 26 or the insert 74 can be determined. For example, it can be assumed that there is no dirtiness and the analysis of the reference image should be within a predetermined similarity threshold. If the analysis of the reference image exceeds the predetermined similarity threshold, it can be considered that the waste container 26 is too full (solid) to be reliably analyzed, and the confidence interval is exceeded. Corresponding predictive alerts or corrective actions can be provided. Specifically, based on this analysis, predictive maintenance can be invoked, in which a technician cleans the inner and outer surfaces of the waste container 26 and replaces the insert 74.

[0098] In addition, the machine learning algorithm can be extended to determine the heterogeneity of the waste material within the waste container 26. For example, by analyzing the imaged area containing the fluid as determined by the segmentation network. Based on the calculation of the heterogeneity characteristics (such as edge scores, standard deviation of pixel values, HOG or other similar characteristics), the user may be warned when the fluid contents are heterogeneous, because this situation may cause inaccurate analysis of the blood content in the fluid. In this case, the user can be instructed to agitate the fluid, such as by mechanical movement or pumping water at high suction pressure, so as to properly mix the fluid contents in the imaging container to accurately analyze the blood content. Once proper mixing is achieved and the heterogeneity score is below a certain threshold, the analysis of the blood content can proceed normally.

[0099] In another embodiment, the camera 50 can be integrated into the chassis 22 without having to support the mobile device on the device bracket 54. In such an embodiment, the camera 50 can be at least one digital camera coupled to any suitable location of the chassis 22, such as within the front housing 46. Additionally or alternatively, the digital camera can be coupled to the waste container 26 to be positioned inside and / or outside the waste volume. For example, the digital camera can be coupled to the container cover 82 and oriented downward. Multiple cameras can be used in combination, images analyzed therefrom, and combined for analysis by machine learning algorithms to improve accuracy and redundancy.

[0100] Several implementations have been discussed in the foregoing description. However, the embodiments discussed herein are not intended to be exhaustive or to limit the invention to any particular form. Modifications and variations may be made in accordance with the above teachings, and may be practiced in a manner different from that specifically described. For example, the method disclosed herein may be performed on a waste container that is not arranged on a mobile chassis. In other words, an equipment support in which an independent tank is placed may be provided-such as an equipment support disclosed in co-owned U.S. Patent No. 10,641,644, granted on May 5, 2020, the entire contents of which are incorporated herein by reference. For example, the blood component may be hemoglobin, or may be one or more of whole blood, red blood cells, platelets, plasma, white blood cells, analytes, and the like. The method may also be used to estimate the concentration and amount of non-blood components in the waste container 26, such as saline, ascites, bile, flushing fluid, saliva, gastric juice, mucus, pleural fluid, interstitial fluid, urine, feces, and the like. The medical waste collection system 20 may communicate with other systems to form a fluid management ecosystem for generating substantially comprehensive estimates of extracorporeal blood volume, total blood loss, patient blood volume status, etc. Additionally, certain inventive aspects of the present disclosure are made with reference to the following exemplary clauses.

[0101] CLAUSE 1 - A method for estimating blood loss from waste material within a waste container of a medical waste collection system, wherein an imaging device is supported in a device stand, the method comprising the steps of: capturing video of the waste container and waste material disposed therein using the imaging device; analyzing the video using one or more processors to determine the volume of waste material within the waste container; analyzing the image using one or more processors to determine the concentration of blood components within the waste material; estimating blood loss based on the determined volume and the determined concentration of blood components using the one or more processors; and displaying the estimated blood loss on a display.

[0102] Item 2 - A method for estimating the amount of blood loss of waste material within a waste container of a medical waste collection system, wherein an imaging device is supported in an equipment stand, the method comprising the following steps: when a vacuum source of the medical waste collection system draws waste material into the waste container, using the imaging device to capture a video stream of the waste container and the waste material placed therein; using one or more processors to segment image frames of the video stream by characterizing whether the pixel values ​​of each image frame are blood, meniscus or non-blood, and optionally whether it is a bubble or a fluid surface; using one or more processors to assign category labels based on the pixel values; using one or more processors to determine the position of a fluid meniscus based on the category label of the meniscus; mapping the image frames from an image coordinate space to a canonical coordinate space; and using one or more processors to determine the volume of waste material within the waste container based on the position of the fluid meniscus.

[0103] Clause 3 - The method according to any one of clauses 2, further comprising: detecting, using one or more processors, at least two reference markers attached to a waste container; and cropping the mapped image frame in a canonical coordinate space to include the at least two reference markers and a portion of the waste container located therebetween.

[0104] Clause 4 - The method of clause 3, further comprising adjusting the image using one or more processors based on the relative positioning of the at least two reference markers.

[0105] Item 5 - An equipment holder for supporting an imaging device to capture images of a waste container of a medical waste collection system, wherein a front shell of the medical waste collection system defines a window, the equipment holder comprising: a shield connected to the front shell, wherein the shield defines a rear groove configured to cover the window, a front groove, and an aperture providing fluid communication between the rear groove and the front groove, wherein the front groove is sized to receive the imaging device and the aperture is positioned relative to the front groove to align with a camera and flash of the imaging device.

[0106] Item 6 - The device holder according to Item 5 further includes: at least one arm that connects the shield to the front shell; and a hinge that connects the at least one arm and the shield and is configured to allow the shield to be pivoted to a configuration in which the window is visible when an imaging device is disposed therein.

[0107] Clause 7 - The equipment stand of clause 6, further comprising a front housing movably coupled to the shield.

[0108] Clause 8 - The device stand of any of clauses 5-7, wherein at least substantially the entirety of the waste container is configured to be visible in the field of view of the camera of the imaging device.

Claims

1. A method for estimating blood loss from waste material within a waste container of a medical waste collection system, wherein an imaging device is supported in a device holder, the method comprising the steps of: capturing a video stream of the waste container and the waste material disposed therein using an imaging device as a vacuum source of the medical waste collection system draws the waste material into the waste container; analyzing, using one or more processors, one or more image frames of the video stream to determine a volume of waste material within the waste container; analyzing, using one or more processors, the image frames to determine the concentration of blood components within the waste material; estimating, using one or more processors, blood loss based on the determined volume and the determined concentration of the blood component; as well as The estimated blood loss is shown on the display.

2. A method for estimating blood loss from waste material within a waste container of a medical waste collection system, wherein an imaging device is supported in a device holder, the method comprising the steps of: capturing a video stream of the waste container and the waste material disposed therein using an imaging device as a vacuum source of the medical waste collection system draws the waste material into the waste container; analyzing, with one or more processors, image frames of the video stream to determine a concentration of a blood component within the waste material, wherein the step of analyzing the image frames is facilitated by a neural network trained on an image data set of at least one of: a relative positioning of a camera of an imaging device with respect to a waste container, a location of flashes of light reflected on a surface of the waste container, color component values ​​associated with at least two reference markers attached to the surface of the waste container, and color component values ​​associated with an imaged surface of an insert disposed within the waste container; estimating, using one or more processors, blood loss based on the volume and the determined concentration of the blood component; as well as The estimated blood loss is shown on the display.

3. The method according to claim 2, wherein: The step of analyzing the image frames is facilitated by the neural network being also trained on an image dataset of a region of interest associated with at least two imaging surfaces of the insert, wherein each of the at least two imaging surfaces provides a different color component value for waste material disposed between the insert and the waste container.

4. The method according to claim 2, wherein: The step of analyzing the image frames is facilitated by the neural network being also trained on an image dataset of a region of interest associated with an imaging surface of the insert to provide a color gradient for waste material disposed between the insert and the waste container.

5. The method according to any one of claims 2 to 4, wherein: The step of analyzing the image frames is facilitated by a neural network that is also trained on a dataset of images of at least one of various volumes of waste material and different phase characteristics of waste material.

6. The method according to any one of claims 2 to 5, further comprising: Mapping the image frame from the image coordinate space to the canonical coordinate space; as well as An image frame in a canonical coordinate space is provided to the neural network.

7. The method of any of claims 2-6, further comprising, utilizing one or more processors, analyzing image frames of the video stream to determine the volume, wherein the step of analyzing the image frames is facilitated by a neural network also trained on the image dataset.

8. The method according to any one of claims 1 to 7, further comprising: capturing, using an imaging device, a video stream at a first frame rate; determining, using one or more processors, a rate at which a volume of waste material within a waste container increases; A video stream is captured, using an imaging device, at a second frame rate greater than the first frame rate if the rate is greater than a predetermined threshold.

9. The method of claim 8, further comprising activating a flash of an imaging device or increasing the intensity of a flash of an imaging device if the rate is greater than a predetermined threshold.

10. The method according to claim 9, further comprising: detecting, using one or more processors, a pixel-based position of a fluid meniscus in each image frame; as well as Using one or more processors, a determination is made as to whether an amount by which a pixel-based position of a fluid meniscus changes within a predetermined period of time is greater than a predetermined threshold.

11. The method of claim 10, further comprising downsampling each image frame prior to the step of detecting the pixel-based position of the fluid meniscus.

12. The method according to claim 1 or 2, further comprising: detecting, with one or more processors, at least two reference markers attached to the waste container; as well as The image frame is adjusted, using one or more processors, based on the relative positioning of the at least two reference markers.

13. The method according to claim 12, wherein: The at least two reference markers are Quick Response (QR) codes arranged in a substantially vertical configuration, wherein the step of adjusting the image frame further comprises adjusting each QR code to be perfectly vertical.

14. The method according to claim 1 or 2, further comprising: segmenting the image frames using one or more processors by characterizing pixel values ​​of each image frame as blood, meniscus, or non-blood, and optionally, bubbles or fluid surface; as well as Assign class labels based on pixel values.

15. The method of claim 14, further comprising determining a location of the fluid meniscus based on the class label of the meniscus.

16. The method of claim 1 or 2, further comprising mapping each image frame from an image coordinate space to a canonical coordinate space.

17. The method according to claim 16, wherein: The step of mapping the image frame also includes: Generate homography to locate the marker; transforming the image coordinates into two-dimensional canonical coordinates; and Optionally, the two-dimensional canonical coordinates are transformed into three-dimensional canonical coordinates.

18. The method of claim 16, further comprising cropping the mapped image frame in the canonical coordinate space to include the at least two reference markers and a portion of the waste container located therebetween.

19. The method according to claim 16, wherein: The step of analyzing the image frame to determine the volume of the waste material further comprises: receiving vessel-specific calibration data; Converts the y-axis values ​​of the mapped image frame in canonical coordinate form to volume values.

20. A method of estimating blood loss from waste material within a waste container of a medical waste collection system, wherein an imaging device is supported in a device holder, the method comprising the steps of: using an imaging device, capturing images of the waste container and waste material disposed therein; as well as Processing the image using one or more processors, wherein the step of processing the image further comprises: determining the position of the fluid meniscus; Adjust the image; Mapping the image from image coordinate space to canonical coordinate space; and Converting the y-axis value of the mapped image in canonical coordinate form to a determined volume; detecting, with one or more processors, at least two reference markers attached to a waste container, wherein the reference markers include position data; extracting, using one or more processors, a region of interest in the original image based on the position data; analyzing the image to determine the concentration of the blood component within the waste material using one or more processors; estimating, using one or more processors, blood loss based on the determined volume and the determined concentration of the blood component; and The estimated blood loss is shown on the display.

21. The method according to claim 20, wherein: The step of determining the position of the fluid meniscus further comprises: segmenting the image frames by characterizing pixel values ​​of each image frame as blood, meniscus, or non-blood, and optionally, bubbles or fluid surface, using one or more processors; and Using one or more processors, class labels are assigned based on the pixel values, wherein the location of the fluid meniscus is based on the class label of the meniscus.

22. The method according to claim 20, wherein: The step of adjusting the image further includes detecting at least two reference markers and adjusting the at least two reference markers to be perfectly vertical.

23. The method according to claim 20, wherein: The steps of mapping the image also include: generating, using one or more processors, a homography for positioning the marker; transforming the image coordinates into two-dimensional canonical coordinates using one or more processors; and Optionally, the two-dimensional canonical coordinates are transformed into three-dimensional canonical coordinates.

24. The method of claim 20, further comprising cropping the mapped image in the canonical coordinate space to include the at least two reference markers and a portion of the waste container located therebetween.

25. The method according to claim 20, wherein: The step of analyzing the image to determine the volume of the waste material also includes applying calibration data specific to the container.

26. A method of correcting tilt of a waste container or imaging equipment of a medical waste collection system, wherein the imaging equipment is supported in an equipment stand, the method comprising the steps of: utilizing an imaging device to capture a video stream of the waste container and the waste material disposed therein; At one or more processors, receiving one or more image frames of a video stream captured by a camera of an imaging device; detecting, using one or more processors, positions of at least two reference markers within an image frame; determining, with one or more processors, an orientation of the waste container based on the positions of the at least two reference markers; determining, using one or more processors, an orientation of an imaging device; determining, using one or more processors, a mathematical correction based on an orientation of the waste container and an orientation of the imaging device; Using one or more processors, a mathematical correction is applied to determine a corrected volume of waste material placed within the waste container.

27. The method according to claim 26, further comprising: analyzing, using one or more processors, the image frames to determine the concentration of blood components within the waste material; estimating, using one or more processors, blood loss based on the corrected volume and the determined concentration of the blood component; as well as The estimated blood loss is shown on the display.

28. A method of calibrating for mechanical differences in a waste container of a medical waste collection system, wherein an imaging device is supported in a device holder, the method comprising the steps of: capturing an image of a waste container and waste material disposed therein using an imaging device in a calibration mode, wherein a vacuum source of the medical waste collection system draws at least a known volume of waste material into the waste container; Analyzing, using one or more processors, image frames of the video stream to determine a y-axis value of a fluid meniscus of waste material within the waste container; and Calibration data relating at least one known value to a y-axis value of the fluid meniscus is stored on the memory.

29. The method according to claim 28, further comprising: detecting a reference marker attached to the waste container, wherein the reference marker includes optically readable data; The calibration data is correlated with the optically readable data of the reference markers using one or more processors.

30. The method according to claim 28 or 29, further comprising: analyzing, using one or more processors, image frames of the video stream to determine a volume of waste material within the waste container; showing the volume of waste material on a display; A confirmation input of the volume of waste material is received on the user interface.

31. A non-transitory computer readable medium having instructions stored thereon, the instructions being configured to be executed by one or more processors to perform the method of any one of claims 1-30.

Images