Venous reservoir blood volume sensor with computer vision

Abstract

A blood volume sensor system for measuring blood volume in a venous reservoir includes a camera configured to capture a video stream of a venous reservoir having volume markings thereon and configured to contain blood therein, and a computer unit configured to receive the video stream from the camera and to determine a volume of the blood contained within the venous reservoir using a computer vision algorithm. A method of measuring blood volume in a venous reservoir of a cardiopulmonary bypass circuit operated by a heart-lung machine using the blood volume sensor system includes capturing a video stream of the venous reservoir containing blood; inputting the video stream into the computer unit; using the computer vision algorithm to determine the volume of the blood in the venous reservoir from image frames of the video stream; and outputting the determined volume of blood in the venous reservoir to the heart-lung machine.

Description

VENOUS RESERVOIR BLOOD VOLUME SENSOR WITH COMPUTER VISIONRELATED APPLICATIONS

[0001] This application claims priority benefit of US Provisional Application Serial Number 63 / 707,913 filed October 16, 2024; the contents of which are hereby incorporated by reference.FIELD OF THE INVENTION

[0002] The present invention generally relates to a blood volume sensor for use with a heartlung machine, and more particularly to an automatic blood volume sensor system to record and can promptly notify a perfusionist or the heart-lung machine to any change or alarming trend of changing blood volume.BACKGROUND

[0003] Artificial support of a patient’s cardiovascular system using cardiopulmonary bypass (CPB) is essential for certain critical clinical procedures such as cardiac surgeries. During such procedures, a perfusionist operates the heart-lung machine to circulate the patient’s blood through an extracorporeal circuit, and one of the perfusionist’s key responsibilities is to maintain an optimal blood volume in the venous reservoir throughout the procedure. During a CPB procedure, if the blood volume falls outside the appropriate working range, or if the blood level rises or drops too rapidly, these are clear indicators of potential imminent complications or even accidents. The perfusionist is expected to take immediate actions to intervene with the CPB procedure to prevent injuries [l]-[3],

[0004] Most commercial reservoirs are equipped with a critical -level safety sensor to prevent formation of bubbles when the blood volume is too low. However, these reservoirs are limited toissuing warnings only when the blood level has already dropped below the critical level. There is nothing available to notify the perfusionist of an alarming trend of changing blood volume. Instead, In the current clinical practice, the perfusionists have to manually read the volume labels on the reservoir to measure the volume, and based on these manual readings, the perfusionists must determine if the blood volume is trending in a way that is indicative of a problem requiring intervention. Such a manual approach is inefficient, inconvenient, and prone to human errors, which can have direct negative impacts on the patient’s health conditions. With existing sensors and manual monitoring of existing sensors, it is also impractical to log blood volume with high temporal resolutions during a long procedure that may last hours.

[0005] Recently, multiple improved venous reservoir blood volume sensors based on different sensing mechanisms have been developed, but there is still room for improvement. For example, an optical sensor based on a contact image sensor (CIS) has been developed, comprising a linear array of CMOS pixels, which scans the blood level inside a venous reservoir to calculate the blood volume [4], [5], This sensor is highly accurate, precise, and non-intrusive; however, the installation of the CIS onto the reservoir blocks at least one side of the reservoir, thus partially obstructing the perfusionist’s view. Additionally, a gravimetric-based blood volume sensor has been developed, using a load cell to measure the weight of blood to calculate its volume [4], [5], similar with the concept introduced by Condello [6], This gravimetric sensor is highly accurate, non- intrusive, and non-obstructive to a perfusionist’s view; however, the system requires a manual zero-point calibration at the beginning of the operation and it is not robust against mechanical disturbances to the reservoir.

[0006] Thus, there exists a need for further improvements, such as an automatic blood volume sensor system that can promptly notify a perfusionist to an alarming trend of changing bloodvolume. Such a sensor must be non-intrusive to the perfusionist’s view and must measure blood without contact, ensuring no risk of contamination. There is additionally a need for automatic and continuous measurement and recording of the venous reservoir blood volume in real time during CPB procedures, over a period spanning the entire procedure.SUMMARY OF THE INVENTION

[0007] The present invention provides a blood volume sensor system and method for measuring blood volume in a venous reservoir. The blood volume sensor system includes a camera configured to capture a video stream of a venous reservoir having volume markings thereon and configured to contain blood therein, and a computer unit configured to receive the video stream from the camera and to determine a volume of the blood contained within the venous reservoir using a computer vision algorithm.

[0008] The method of measuring blood volume in a venous reservoir of a cardiopulmonary bypass (CPB) circuit operated by a heart-lung machine using the blood volume sensor system includes capturing, with the camera, a video stream of the venous reservoir containing blood; inputting the video stream into the computer unit; using the computer vision algorithm to determine the volume of the blood in the venous reservoir from image frames of the video stream; and outputting the determined volume of blood in the venous reservoir to the heart-lung machine.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present invention is further detailed with respect to the following drawings that are intended to show certain aspects of the present of invention, but should not be construed as limit on the practice of the invention, wherein:

[0010] FIG. 1 shows a schematic of a blood volume sensor system based on computer vision according to embodiments of the present invention;

[0011] FIG. 2 is a flowchart of an overall function of a method used by a computer vision algorithm for computing blood volume from the input video stream according to embodiments of the present invention;

[0012] FIGS 3A-3F show schematics of a blood volume sensor system positioned relative to a venous blood reservoir according to embodiments of the present invention;

[0013] FIGS. 4A-4H show a schematic and photo of the sensor setup for a mode Al of a method 400 according to embodiments, flow-chart for this specific mode, example image processing steps, and experimental results;

[0014] FIGS. 5A-5E show a schematic of the sensor setup for a mode A2 of a method 500 according to embodiments, flow-chart for this specific mode, example image processing steps, and experimental results;

[0015] FIGS. 6A-6C show photos of the sensor setup for a mode A2’ of a method 600 according to embodiments, flow-chart for this specific mode, and example image processing steps;

[0016] FIG. 7A-7C show a photo of the sensor setup for a mode A3 of a method 700 according to embodiments, flow-chart for this specific mode, and example image processing steps;

[0017] FIGS. 8A-8C show photos of the sensor setup for a mode A4 of a method 800 according to embodiments, flow-chart for this specific mode, and example image processing steps;

[0018] FIGS. 9A-9F show a schematic and a photo of the sensor setup for a mode Bl of a method 900 according to embodiments, flow-charts for this specific mode, and example image processing steps;

[0019] FIGS. 10A and 10B show a flowchart for this specific mode for a mode Bl ’ of a method 1000 according to embodiments and example image processing steps;

[0020] FIGS. 11A and 1 IB show sub-routines for triggering alarms for when a camera’s view is blocked and when a blood volume flow rate is abnormal, respectively;

[0021] FIG. 12 shows a flowchart of a subroutine 1200 for updating the volume label registration if the reservoir has been moved relative to the camera;

[0022] FIGS. 13A-13C show a sensor setup for a method 1300 for acquisition of a camera image with reduced interference from the ambient environment using Digital Subtraction Imaging (DSA) according to embodiments, a flow-chart of this method 1300, and example image processing steps;

[0023] FIG. 14 shows a sensor setup for a method 1400 for enhancing the blood volume sensor’s performance using reinforcement learning;

[0024] FIG. 15 shows a photograph of a user interface on the console to interact with the sensor;

[0025] FIG. 16 shows a super-clamp used on a Sorin reservoir, for mounting the camera at the bottom;

[0026] FIG. 17 shows a top mount fixture for Sorin reservoir. According to embodiments, the adapter feet directly fit into the top cap of the Sorin reservoir;

[0027] FIG. 18 shows a top mount fixture for a Terumo reservoir;

[0028] FIG. 19A is a photograph showing an experimental setup for testing the sensor system according to embodiments of the present invention in a simulated clinical CPB environment with a SORIN S5 heart-lung machine in a simulated surgical operation room at Cardiovascular Perfusion SIMLAB at Orrum Clinical Analytics;

[0029] FTG. 19B is a photograph showing an experimental setup for testing the sensor system according to embodiments of the present invention with a SORIN venous reservoir connected to the heart-lung machine;

[0030] FIG. 19C is a photograph of the reservoir equipped with a critical-level safety sensor for the experimental setup of FIGS. 19A and 19B;

[0031] FIG. 19D shows a table of the sensor’s performance in the experimental setup of FIGS. 19A-19C;

[0032] FIG. 20A is a photograph showing an experimental setup for testing the sensor system according to embodiments of the present invention with a TERUMO reservoir;

[0033] FIG. 20B is a photograph showing a processed image frame of the reservoir of FIG. 20A for OCR-based Volume Label Recognition and registration;

[0034] FIG. 20C is a photograph showing a processed image frame of the reservoir of FIG. 20A for detecting the blood level using HSV masking and Contour detection;

[0035] FIG. 20D is a photograph showing a processed image frame of the reservoir of FIG. 20A for blood volume calculation; and

[0036] FIG. 20E shows a table of the sensor’s performance in the experimental setup of FIGS. 20A-20D.

[0037] FIG. 21 shows a method of using ORB-based feature recognition to directly detect the blood volume.DETAILED DESCRIPTION OF THE INVENTION

[0038] The present invention has utility as an automatic blood volume sensor system that can promptly notify a perfusionist to an alarming trend of changing blood volume. The inventivesensor system is non-intrusive to the perfusionist’s view and measures blood without contact, ensuring no risk of contamination. The inventive sensor system is automatic and provides continuous measurement and recording of the reservoir liquid volume in real time over extended periods of time. While the present invention is largely detailed with respect to reservoirs used in cardiopulmonary bypass (CPB), it is appreciated that the present invention is operative with both hard-shell and soft-shell reservoirs. Exemplary of soft-shell reservoirs are Dover®, Medline ®, Rusch® and Holliuster ® urine bags. Also besides blood volume detection, the present invention is also operative with other liquids that illustratively include urine, surgical suction fluids, industrial fluids, and lavage fluids.

[0039] According to embodiments, a venous reservoir blood volume sensor system is provided based on computer vision, with the advantages of being non-intrusive to the blood, non-obstructive to the perfusionists’ operations, and versatile for different brands of reservoirs. Computer vision has emerged as a promising solution for certain medical technologies including medical imaging [7], [8], and surgical robots [9], In principle, repetitive tasks that can be performed by a human operator through manual vision can potentially be accomplished through computer vision. However, the present invention that reads the blood volume from a venous reservoir using computer vision is not merely a computer-based automated process mimicking a perfusionist’s visual inspection of blood volume. Instead, the present invention enables improvements in venous reservoir blood volume monitoring not possible with manual monitoring alone, such as real-time monitoring and logging of blood volume with high temporal resolutions during a long procedure that may last hours, providing alerts to alarming trends faster than manually possible, and high accuracy, such as a mean absolute percentage error as small as 2.5%.

[0040] The present invention will now be described with reference to the following embodiments. As is apparent by these descriptions, this invention is embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. For example, features illustrated with respect to one embodiment can be incorporated into other embodiments, and features illustrated with respect to a particular embodiment may be deleted from the embodiment. In addition, numerous variations and additions to the embodiments suggested herein will be apparent to those skilled in the art in light of the instant disclosure, which do not depart from the instant invention. Hence, the following specification is intended to illustrate some particular embodiments of the invention, and not to exhaustively specify all permutations, combinations, and variations thereof.

[0041] It is to be understood that in instances where a range of values are provided that the range is intended to encompass not only the end point values of the range but also intermediate values of the range as explicitly being included within the range and varying by the last significant figure of the range. By way of example, a recited range of from 1 to 4 is intended to include 1-2, 1-3, 2-4, 3-4, and 1-4.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0043] Unless indicated otherwise, explicitly or by context, the following terms are used herein as set forth below.

[0044] As used in the description of the invention and the appended claims, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0045] Also as used herein, “and / or” refers to and encompasses any and all possible combinations of one or more of the associated listed items, as well as the lack of combinations when interpreted in the alternative (“or”).

[0046] As shown in FIG. 1, embodiments of the automatic blood volume sensor system 100 are configured for use with a cardiopulmonary bypass (CPB) circuit 200 operated using a heart-lung machine 300. The sensor 100 is monitored by the perfusionist 70 as well as the heart-lung machine 300. Embodiments of the automatic blood volume sensor 100 include a camera 10 with an LED lamp 30 positioned to face the venous reservoir 40 of the CPB circuit 200, at a distance of about 1 foot or less. The sensor system 100 measures the volume of venous blood 50 inside the venous reservoir 40. The lamp 30 illuminates the reservoir 40 to provide optimal conditions for the camera 10. The camera 10 captures the video stream 12 from the venous reservoir 40 loaded with blood 50. The video stream 12 is processed by the computer unit 20 using a computer vision algorithm to detect the blood volume from the acquired images frames in order to calculate the blood volume 60 of the venous blood 50. The real-time measurement of blood volume 60 is displayed on a console 22 of the sensor 100 to notify the perfusionist 70, and streamed to the heart-lung machine 300. The sensor console 22 displays the sensor output, shows the crucial image processing, issues warning for abnormal blood volume or flow, provides user-interface to control and configure the sensor system The result of blood volume 60 is displayed to the perfusionist 70 and continuously sent to the heart-lung machine 300.

[0047] According to embodiments, the camera 10 is a black and white camera, a color camera, an near-infrared night vision camera, an infrared thermal camera, a wide-angle camera, a fisheye camera, a 360 degree camera, or a combination thereof. According to embodiments, the lamp 30 is a LED lamp, a light bulb, or a light source with infrared light emission. According to embodiments the lamp 30 is an optional component of the sensor 100. According to embodiments, the lamp is configurable with brightness settings to automatically adapt to ambient lighting conditions. According to embodiments, the computer unit 20 is provided as a separate component from the camera 10 and / or lamp 30. Alternatively, the computer unit 20 is integrated with the camera 10 into one component. According to embodiments, the computer unit 20 and / or the camera 10 and / or lamp 30 are integrated into the heart-lung machine 300 to directly communicate results to the heart-lung machine 300.

[0048] FIG. 2 shows an overall functional flowchart of an inventive blood volume sensor system 100. The computer unit 20 of the inventive sensor 100 uses an algorithm based on computer vision to determine the blood volume of the reservoir. As shown in FIG. 2, at Step (i), the computer unit receives the video stream from the camera and pre-process the image frames with transformation, conversion, and enhancement. At step (ii), volume labels are automatically detected from the image frame, and their locations on the image are registered. At Step (iii), the liquid / air interface is determined from the image to find their location. At Step (iv) The volume of liquid (blood volume) is calculated by interpolating the location of the interface relative to the registered positions of the volume labels. An algorithm operative herein is the Oriented FAST and Rotated BRIEF (ORB) algorithm that is available from OpenCV.

[0049] Each component of the sensor system and algorithm shown in FIGS. 1 and 2 has multiple options, and any combination of them can be used for measuring the blood volume. Table 1 below shows the options for these components.

[0050] Table 1- Component Options

[0051] The algorithm is compatible for different brands of commercial venous reservoirs, including SORIN INSPIRE HVR and TERUMO CAPIOX NX19. It is compatible for all venous reservoirs that were designed for perfusionists’ visual estimation of blood volume. Additional options for the component of the sensor system are shown in Table 2.

[0052] Table 2- Additional options for sensor system components.

[0053] As shown in FIGS. 3A-3F, the camera can be mounted by fixing it onto the reservoir, or off the reservoir. The camera can be positioned towards the reservoir from different angles. FIGS. 3A-3F show six exemplary configurations of mounting the camera. According to some embodiments, the camera is mounted on the reservoir by a clamp or a mounting fixture. According to other embodiments, the camera is mounted off the reservoir, for example attached to a heart lung machine or other mechanical support. Embodiments of the automatic blood volume sensor 100 are configured for use with a cardiopulmonary bypass (CPB) circuit 200 in multiple configurations. The camera 10 and LED lamp 30 are positioned to face the reservoir 40. The distance between the camera 10 and the reservoir 40 is about 0.25 foot to 3 feet. According to other embodiments, the camera 10 and LED lamp 10 are directly attached to the reservoir 40, by anysuitable means such as by stickers, or Velcro, or other convenient and robust means. According to embodiments, the camera 10 and LED lamp 30 are positioned to view the reservoir from above, or from the side, or from below.

[0054] According to embodiments, the sensor system is categorized by at least one of two different modes in recognizing volume labels: based on Optical Character Recognition (OCR); and based on Oriented FAST and Rotated BRIEF (ORB).

[0055] FIGS. 4A-4H show a photo of the sensor setup for a mode Al of a method 400 according to embodiments, flow-chart (a customized version of Fig 2 for this specific mode), example image processing steps, and experimental results. In mode Al, the camera is a visible light color camera, such as ArduCam B0448. The camera is mounted off the reservoir. The camera is facing normal to the reservoir surface. The system finds volume labels using OCR. The liquid / air interface is detected using contour detection based on HSV masking. In Step 402, the input video stream is processed with transformation and correction, such as image rotation, as in FIG. 4C. In Step 404, Optical character recognition (OCR) is used to identify volume labels on the reservoir from the image frame and register their positions on the image frame, as in FIG. 4D. In Step 406, the liquid / air interface is extracted from the image using HSV masking and contour detection, as in FIGS. 4E and 4F. The software routine detects the full area of blood in the reservoir by finding the largest “blob” of blood-colored pixels in the image. Pixels are masked based on their hue, saturation, and value (HSV) numbers, which can be tuned for liquids of different shades and colors. When a blob is identified, the liquid / air interface is determined from the top edge of this blob. In Step 408, using the positions of blood volume labels registered in step 404 and the position of liquid / air interface extracted from step 406, the volume of blood is calculated in step 408, by interpolating the interface position relative to the registered positions of the volume labels, as inFIG. 4G. Experimental results of this method 400 for calculating a blood volume using an inventive blood volume sensor system 100 are shown in FIG. 4H. The most crucial performance characteristics of the inventive sensor are the accuracy and precision in reading the blood volume. The mean absolute percentage error (MAPE) is used as an indicator of error while the standard deviation is used as the measure of precision. The relative error (MAPE) is less than 5% and the precision (standard deviation) is a few m . Mode Al has been tested on other reservoirs, such as Terumo CAPIOX NX19. This proves this sensor system is automatically compatible with wide range of commercial reservoirs.

[0056] FIGS. 5A-5E show a photo of the sensor setup for a mode A2 of a method 500 according to embodiments, flow-chart (a customized version of Fig 2 for this specific mode), example image processing steps, and experimental results. In mode A2 of method 500, the camera is a visible light color camera, such as ArduCam B0200. The camera is mounted on the reservoir at the bottom. The camera is pointing up. The volume labels are found using OCR. The liquid / air interface is detected using contour detection based on HSV masking. In step 502, a raw image from the video stream is used to find the four ArUco markers, identify the area of interest, and transform the image. In Step 504, the volume labels are found using OCR and to register their locations. In Step 506, HSV masking is used and contour detection is used to find the liquid / air interface. In Step 508, the volume is calculated using the position of the liquid / air interface and the registered locations of the volume labels. Four ArUco markers (two from the top mount, two from the superclamps) define an area of interest covering a set of volume labels. The image is cropped from the area of interest is stretched and transformed to compensate the image distortion due to viewing the labels from a close distance and a steep angle with the camera. FIG. 5E shows performance results of mode A2 of method 500.

[0057] FIGS. 6A-6C show a photo of the sensor setup for a mode A2’ of a method 600 according to embodiments, flow-chart (a customized version of Fig 2 for this specific mode), and example image processing steps. Mode A2’ is using a similar approach as mode A2, but the imaging is for invisible near-infrared light and the image is in grayscale. Here, the camera is a grayscale near-infrared night vision camera, such as ArduCam B0322. The camera is mounted on the reservoir at the bottom. The camera is pointing up. The Near-infrared (NIR) LED ring is the lamp. An NIR long-pass filter is installed in front of the camera lens, which blocks visible light to show only NIR images. The volume labels are found using OCR. The liquid / air interface is detected using contour detection based on grayscale masking. IN Step 604, the volume labels are found using OCR and their locations are registered from the transformed image of the area of interest defined by the ArUco markers. In Step 606, grayscale masking is done and a contour detection is used to find the liquid / air interface. In Step 608 the volume is calculated using the position of the liquid / air interface and registered locations of the volume labels. Grayscale masking works by finding the grayscale values matching the characteristics of blood. In this example, the volume label locations also contain information on the orientation, which matches the orientation of the liquid / air interface.

[0058] FIG. 7A-7C show a photo of the sensor setup for a mode A3 of a method 700 according to embodiments, flow-chart (a customized version of Fig 2 for this specific mode), and example image processing steps. Here, the camera is a visible light color camera, such as ArduCam B0200. The camera is mounted on the reservoir at the top. The camera is pointing down. The volume labels are found using OCR. The liquid / air interface is detected using edge detection. In Step 702: the ArUco markers are found, the AOI is identified defined by four ArUco makers, and the image is transformed to compensate distortion. In Step 704, a predefined sub-region is found in the AOI,the volume labels are found using OCR, and their locations are registered. In Step 706, the position of liquid / air interface is found from the subregion in the AOI using edge detection. In Step 708, the volume is calculated using the position of liquid / air interface and registered locations of volume labels. Here, from the Canny edge detection, the longest continuous horizontal line will be selected and recognized as the liquid / air interface.

[0059] FIGS. 8A-8C show a photo of the sensor setup for a mode A4 of a method 800 according to embodiments, flow-chart (a customized version of Fig 2 for this specific mode), and example image processing steps. Here, the camera module is, for example, a Topdon TC001 Plus, with dual modes of thermal image (mid-infrared) + visible image. The camera is mounted on the reservoir at the bottom. The camera is pointing up. The volume labels are found using OCR from the visible color image. The liquid / air interface is detected from the thermal image using contour detection based on grayscale masking. An acquired visible color image used in Step 802 & 804. In Step 806, an acquired thermal image is used to find the liquid / air interface using contour detection based on grayscale masking. Thermal image is a map of surface temperature of the reservoir, acquired by the mid-infrared thermal radiation. The blood is usually warmer than the ambient environment which gives a contrast at the liquid / air interface. Contour detection based on grayscale masking is used to find the interface. The blood volume is calculated using the registered blood volume labels attained from the visible color image, and the liquid / air interface position found from the thermal image.

[0060] FIGS. 9A-9F show a photo of the sensor setup for a mode B 1 of a method 900 according to embodiments, flow-chart (a customized version of Fig 2 for this specific mode), and example image processing steps. In this embodiments, the camera is a visible light color camera, such as an ArduCam B0497C. The camera is mounted on the reservoir at the top. The camera is pointingdown. The volume labels are found using ORB. The liquid / air interface is detected using motion detection. FIG. 9C shows an overall flowchart of blood volume sensor system mode Bl (Method 900. FIG. 9D shows a flowchart of sub-routine for step 906 for finding volume label positions for blood volume sensor system mode Bl. FIG. 9E shows a flowchart of sub-routine for step 908 for finding liquid / air interface for blood volume sensor system mode Bl. According to embodiments, the volume labels are found using an ORB-based feature matching algorithm, and the liquid / air interface is found using motion detection. In step 902, the input video steam is processed with an image transformation and correction, such as image rotation, and a “template” image is saved. In step 904, preprocessing is completed where the location and value of each volume label is manually input onto the “template” image and saved to a preprocessed image library for all target reservoirs. In step 906, an Oriented Fast and Rotated Brief (ORB) algorithm is used to match features between the live feed and the preprocessed image library. After the best match is found, a homography matrix is calculated and the location and value of each volume label from the template image is stretched onto the live feed. In step 908, after the volume labels are overlaid in step 906, motion detection is completed on the live frame, and the corresponding mask is created. The liquid / air interface is extracted from the motion detection mask based on location, orientation, and area. In step 910, using the locations of the volume labels found in step 906 and the location of the liquid / air interface found in step 908, the volume of blood is calculated in step 910. The liquid / air interface position relative to the positions of the volume labels give the calculated volume by interpolation. Several different modalities can be used simultaneously in the extraction of the liquid / air interface, such as HSV detection, grayscale (night vision), thermal, and edge detection, which aid in a motionless video feed. When motion ceases, the previously extractedblood volume is displayed. Recalibration of the homography matrix and volume label overlay occurs automatically / continuously while the sensor is in use.

[0061] FIGS. 10A and 10B show a flowchart (a customized version of Fig 2 for this specific mode) for a mode B 1’ of a method 1000 according to embodiments and example image processing steps. Mode Bl’ uses the same hardware configuration as Bl. Bl’ implements the same preprocessing, and ORB-based algorithm to find volume label as mode B 1. The difference in mode Bl’ is the use of Canny edge detection to find the liquid / air interface while mode Bl uses motion detection. In this embodiments, the camera is a visible light color camera, such as ArduCam B0497C. The camera is mounted on the reservoir at the top. The camera is pointing down. The volume labels are found using ORB. The liquid / air interface is detected using edge detection. FIG. 10B shows example image processing the workflow of the blood volume sensor system mode B 1 ’ (Method 1000).

[0062] According to embodiments, the inventive system and method has specific sub-routines in operating the blood volume sensor to deal with the following specific scenarios related to CPB operation with the venous reservoir: Camera’s view is blocked; Abnormal blood volume is detected; and Camera’s position has been moved relative to the reservoir.

[0063] FIGS. 11A and 1 IB show sub-routines for triggering alarms for when a camera’s view is blocked and when a blood volume flow rate is abnormal, respectively. Given that the sensor system 100 relies on views of the reservoir 40 to measure the blood volume, if the view is blocked by any object between the reservoir 40 and the camera 10, the volume cannot be read or measured. When this occurs, the system triggers a warning to alert the perfusionist about the blockage of the view and wait for the perfusionist or other operators to correct it. Furthermore, using a CPB procedure, if the blood volume falls outside the appropriate working range, or if the blood levelORRU-OIOIPCT rises or drops too rapidly, these are clear indicators of potential imminent complications or even accidents. The perfusionist is expected to take immediate actions to intervene with the CPB procedure to prevent injuries. According to the sub-routines, these are achieved with the sensor system. If the blood volume is abnormal, a warning will be triggered to notify the perfusionist of the abnormal volume. The flow rate is calculated from the blood volume as a function of time. If the blood flow is abnormal (rising or dropping too rapidly), a warning will be triggered to notify the perfusionist of the alarming trend of changing volume. If both the blood volume and the flow rate are in the good working ranges, no warning will be issued and the sub-routine exits and returns to the main program. In FIG. 11 A, the sensor system relies on views of the reservoir to measure the blood volume. If the view is blocked by any object between the reservoir and the camera, the volume cannot be read or measured. The system triggers warning to alert the perfusionist about the blockage of the view and wait for the perfusionist or other operators to correct it. In FIG. 1 IB, during a CPB procedure, if the blood volume falls outside the appropriate working range, or if the blood level rises or drops too rapidly, these are clear indicators of potential imminent complications or even accidents. The perfusionist is expected to take immediate actions to intervene with the CPB procedure to prevent injuries. These are achieved with the sensor system. If the blood volume is abnormal, a warning will be triggered to notify the perfusionist of the abnormal volume. The flow rate is calculated from the blood volume as a function of time. If the blood flow is abnormal (rising or dropping too rapidly), a warning will be triggered to notify the perfusionist of the alarming trend of changing volume. If both the blood volume and the flow rate are in the good working ranges, no warning will be issued and the sub-routine exist and returns to the main program.

[0064] As shown in the flowchart of FIG. 2, the algorithm considers whether volume label locations need to be corrected if the reservoir is moved relative to the camera. Notably, before the sensor's continuous operation in reading blood volume, the sensor must be first calibrated with the registered locations of volume labels. Moreover, during the operation of the sensor system, the reservoir’s positions may be shifted and the calibration procedure must be automatically repeated to register new positions of volume labels. The flowchart for re-calibration procedure is described in FIG. 12. FIG. 12 shows a flowchart of a subroutine 1200 for updating the volume label registration if the reservoir has been moved relative to the camera. If the reservoir has been moved relative to the camera during the operation, the locations of previously registered volume labels will no longer be valid. The sensor’ s reading will be unreliable. The volume labels locations must be updated. The new image frame of the reservoir is compared with the stored frame to check if the reservoir has been shifted. If shifting has been identified, initiate the sequence to update the locations of volume labels as well as the reservoir position and orientation in the image. Otherwise, continue with the sensor’s operation with existing calibration. Notably, the sub-routine 1200 is mainly used in Method 400, 500, 600, 700 & 800. The method 900 and 1000 already have the subroutine 1200 built into their regular workflow.

[0065] According to embodiments, the inventive sensor system includes configurations to enhance the sensor’s performance in terms of reliability. According to embodiments, such enhancements include using Digital Subtraction Imaging to reduce interference from ambient environment. FIGS. 13A-13C show a sensor setup for a method 1300 for acquisition of a camera image with reduced interference from the ambient environment using Digital Subtraction Imaging according to embodiments, a flow-chart of this method 1300, and example image processing steps. During such Digital Subtraction operations, the system modulates the LED lamp to turn it ON / OFFperiodically, via control signal from the computer unit and synchronizes the camera’s image acquisition with the LED lamp ON / OFF state. When the LED is ON, the system acquires an image (Image A of FIG. 13C) of the reservoir illuminated by LED and ambient background. When the LED is OFF, the system acquires an image (Image B of FIG. 13C) of the reservoir illuminated by ambient background. The system then uses digital subtraction to obtain Image C of FIG. 13C, which isolates the view of reservoir illuminated by LED only: Image C= Image A - Image B. The modulation of the LED must be fast enough to make the flashing pattern of LED un-noticeable by the operators and other personnel in the clinical environment. For example, the LED can be modulated at 50 Hz per second (50 ON / OFF cycles per second), and the camera captures the images at a frame rate of 100 frames per second (two images per cycle: One image for LED ON; One image for LED OFF). According to embodiments, a sensor system enhancement includes using reinforcement learning to optimize the sensor’s setting used for finding the liquid / air interface. FIG. 14 shows a sensor setup for a method 1400 for enhancing the blood volume sensor’s performance using reinforcement learning. In the reinforcement learning of method 1400, the blood volume sensor’s algorithm relies on contour detection, edge detection or motion detection to find the liquid / air interface. In this algorithm, the parameters, such as thresholds in determining the interface, affect the results. Reinforcement Learning (RL) is applied to automatically determine the optimal parameters for finding the liquid / air interface under various lighting conditions. An RL agent learns to adjust parameters by interacting with the blood volume sensor system and receiving rewards based on performance metrics such as detection accuracy.

[0066] According to embodiments, the sensor system additionally includes supporting components for the blood volume sensor system. These include at least one of a user interface on the console to interact with the sensor, a super-clamp used for a Sorin reservoir for mounting thecamera at the bottom, and a top-mount fixture used for Sorin reservoir and Terumo reservoir for mounting the camera at the top. FIG. 15 shows a photograph of a user interface on the console to interact with the sensor. According to embodiments, the Venous Reservoir Blood Volume Monitor / User Interface contains the following characteristics: current reservoir volume, current flow rate, live feed display of venous reservoir: the sensor’s live feed is displayed for the perfusionist’s viewing / interpretation, graph of blood volume vs time, low-level alarms, high-level alarms, abnormal flow rate alarms, and abnormal trends alarms, operator inputs for configuration of warning thresholds, data logging and access, and manual recalibration. FIG. 16 shows a superclamp used on a Sorin reservoir, for mounting the camera at the bottom. FIG. 17 shows atop mount fixture for Sorin reservoir. According to embodiments, the adapter feet directly fit into the top cap of the Sorin reservoir. FIG. 18 shows a top mount fixture for a Terumo reservoir. According to embodiments, the adapter feet directly fit into the top cap of the Terumo reservoir.

[0067] As the sensor’s hardware has no contact with the reservoir, it is non-intrusive to the venous blood inside the reservoir. Since the camera is not mounted onto the reservoir surface, it will not obstruct the perfusionist’s view to the reservoir. The present invention is also versatile across different models of venous reservoirs, as it requires no modifications or custom designs for the reservoirs.

[0068] As noted above, the sensor system 100 consists of three major components: a camera 10, an LED lamp 30, and a computer 20. A high-definition webcam with High Dynamic Range (HDR) capabilities, an Arducam IMX291, is used as the camera in the examples. An LED lamp 30 is mounted alongside with the camera 10 sensor, pointed at the reservoir 40, to maintain a bright and consistent lighting condition on the reservoir surface, independent of other ambient light. A paper-based light diffusing layer is attached to the LED lamp to improve the uniformity ofillumination. The intensity of light at the reservoir surface is measured to be 2,270 lux. A Lenovo ThinkPad Pl Gen 5 laptop, equipped with a NVIDIA RTX A2000 GPU, is used as the computer 20 to process the video stream 12 and calculate the blood volume in real time. The GPU allows advanced techniques like automatic calibration to be processed in a few seconds. In order for the system to remain non-obtrusive to the perfusionist, the camera / LED lamp are mounted on a flexible arm, which is conveniently attached to a mounting post of a heart-lung machine in a clinical setting.

[0069] To test the sensor’s versatility across different brands of reservoirs, two brands of commercial venous reservoirs are used: SORIN INSPIRE HVR and TERUMO CAPIOX NX19. Both reservoirs are popularly used in clinical CPB procedures. To test the sensor’s performance, test liquids mimicking human blood are used in the reservoir. Since the sensor 100 relies on computer vision to identify and quantify blood in the reservoir, the color appearance of the test liquids play a crucial role. Two test liquids are used: a mixture of milk and red food coloring dye, and bovine blood. Both liquids resemble human blood closely in terms of color appearance.

[0070] According to embodiments, the algorithms and software of the blood volume sensor 100 are written in the Python programming language, and is tasked with processing camera frames into readings of blood volume. According to embodiments, the video stream 12 is acquired from the camera 10 and processed with proper transformation and correction, such as image rotation, using OpenCV

[0010] ,

[0071] The most crucial performance characteristics of the sensor 100 are the accuracy and precision in reading the blood volume. The following protocol and criteria are implemented to quantify the accuracy and precision. The reservoir is filled with a test liquid until its level reaches one of the major grid lines on the reservoir’s surface. The grid line serves as the “ground truth” fordetermining the accuracy of the readings. For a given grid line and volume, 20 consecutive blood volume readings are recorded from the sensor, and the mean absolute percentage error (MAPE) is used as a measure of accuracy and calculated according to Equation EEquation 1 :

[0072] where Xi is the / th recorded blood volume value, X is the ground truth value, and n =20 is the total number of readings. The standard deviation of the 20 readings is used as the measure of precision. After the accuracy and precision is calculated for the given volume, the procedure is repeated on all major grid lines present on the reservoir surface.

[0073] The sensor’s response to the SORIN reservoir is tested with dye / milk mixture. As shown in FIGS. 19A-19D. To assess the sensor’s potentials for clinical applications, the sensor is tested in a simulated CPB environment in the Cardiovascular Perfusion SIMLAB at Orrum Clinical Analytics Inc. The lab provides state-of-the-art, accurate, realistic setting for mimicking clinical scenarios, enabling essential tests of medical devices to be conducted in a simulated clinical environment. The experimental setup is shown in FIG. 19A. A SORIN S5 heart-lung machine and a SORIN reservoir are used in the experiment. Bovine blood is circulated into the CPB circuit to mimic human blood. As shown in FIG. 19C, a critical level safety sensor is installed onto the reservoir surface as it is a standard configuration for clinical CPBs. However, it blocked the 150 mL and 200 ml volume labels and restricted the working range of the sensor. To test the sensor’s performance, especially the blood volume in the range blocked by the safety sensor, varying blood volume up to 2000 mL are tested in the experiments and the results are listed in the table of FIG. 19D. The inventive sensor 100 is able to read the blood volume with high accuracy and precision for volume in the range from 300 mL to 2000 mL. Although the range below 300mL could not be covered by the sensor, it is effectively safeguarded by the critical -level safety sensor.

[0074] Inside the simulated CPB environment, two certified clinical perfusionists evaluate the entire CPB setup and confirmed the sensor has negligible effects to their direct view to the entire reservoir during an operation. The sensor has therefore been qualitatively validated as nonobstructive. It is important to note that the sensor provides the perfusionists with convenient, real time blood volume readings, and enables automated venous blood management.

[0075]

[0076] The sensor’s response to TERUMO reservoir is tested with dye / milk mixture, as shown in FIGS. 20A-20D. To test the full working range of the reservoir, no critical-level safety sensor is installed on the reservoir surface. The readings of the sensor from both reservoirs matched with the ground truth very closely, as shown in FIG. 20E. For a volume above 500 mL, the MAPE from SORIN reservoir is below 2.27% while from TERUMO reservoir below 2.2%. The average MAPE, calculated across all tests, is about 2.5%. The precision of the sensor is about a few mL range. For a volume smaller than 500 mL, the precision is even better due to volume marks in grid lines are spaced with finer increments. This accuracy and precision are comparable with previous CIS-based sensor [4], [5], Since the sensor’s performances for two different brands of reservoirs are similar, the versatility of the inventive sensor is validated.

[0077] The demonstrated computer vision-based sensor system is able to read the blood volume in the venous reservoir with high accuracy and precision, while not obstructing a perfusionist’s view to the venous reservoir. The system operates completely independent of the perfusion instruments, and thereby does not hinder the perfusionists’ operations on the heart-lung machine. Since the critical -level safety sensor on the reservoir blocks volume labels below the critical level,the sensor is limited to measuring volume above the critical level. The sensor is versatile and adaptable to different models of venous reservoirs and it delivers similar accuracy and precision for both SORIN and TERUMO reservoirs. In summary, the computer vision based venous reservoir blood volume sensor presented in this work is unique and effective, meeting all the requirements specified for clinical cardiopulmonary bypass procedures.

[0078] FIG. 21 shows a method developed out of method 900, by combining the volume label recognition and liquid / air interface detection into one step. Step (i): Create the image library - Process video feed with transformation and rotation. Acquire images of reservoir at every label position and name file with corresponding volume. Step (ii): Volume Recognition - ORB compares features between the live video feed and the preprocessed image library. The file name of the image with the highest match represents the volume in real time. This approach can also be used to acquire an ROI for use with all other described modalities and image acquisition methods. This allows for interpolation and extrapolation of the volume position when it falls in between the volume labels registered in the preprocessed image library, d the liquid / air interface detection into one step of recognizing the volume directly.

[0079] With the sensor system 100’s capability, many further potential applications can be developed, centered on the blood volume reading. For example, the perfusionist can use the recorded blood volume to track the history of the CPB operation which is crucial for evaluating the performance of the CPB procedures. Furthermore, with the sensor’s reading and special algorithm, the trending of the blood volume can be forecasted to enable advanced control of the heart-lung machine.

[0080] The example embodiments are based on hard-shell venous reservoirs, but it can be extended to work with soft-shell venous reservoirs. While the invention is positioned as a bloodvolume sensor for CPB, it can be adapted to any other scenario which requires measurement of volume of other liquid such as urine.

[0081] While at least one exemplary embodiment has been presented in the foregoing description, it should be appreciated that a vast number of variations exist. It should also be appreciated that the exemplary embodiment or exemplary embodiments are only examples, and are not intended to limit the scope, applicability, or configuration of the described embodiments in any way. Rather, the foregoing description and incorporated references will provide those skilled in the art with a convenient roadmap for implementing the exemplary embodiment or exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the scope as set forth in the appended claims and the legal equivalents thereof.

[0082] References[1] W. S. Stoney, “Evolution of cardiopulmonary bypass,” Circulation, vol. 119, no. 21, pp. 2844- 2853, 2009.[2] M. Sarkar and V. Prabhu, “Basics of cardiopulmonary bypass,” Indian Journal of Anaesthesia, vol. 61, no. 9, p. 760, 2017.[3] G. P. Gravlee, Cardiopulmonary bypass: principles and practice. Lippincott Williams & Wilkins, 2008.[4] T. Foley, A. F.-R. i Sabala, M. Fockler, Z. Alzayer, S. Murtha, and H. Jiang, “Continuous contactless measurement of blood volume inside venous reservoirs for cardiopulmonary bypass,” IEEE Sensors Journal, vol. 23, no. 13, pp. 14882- 14890, 2023.[5] S. Murtha, H. Jiang, T.Foley,A.F.-R.iSabala,M.Fockler,Z. Alzayer, andA. Wayne, “Blood volume sensor system,” PCT Patent Application, p. WO2023 / 107591A1, 2023.[6] I. Condello, “Venous reservoir volume measurement and monitoring, perspective in goal directed perfusion,” Perfusion, vol. 1, p. 2, 2021.[7] J. Li, Z. Zhou, J. Yang, A. Pepe, C. Gsaxner, G. Luijten, C. Qu, T. Zhang, X. Chen, W. Li et al., “Medshapenet-a large-scale dataset of 3d medical shapes for computer vision,” arXiv preprint arXiv:2308.16139, 2023.[8] A. Parvaiz, M. A. Khalid, R. Zafar, H. Ameer, M. Ali, and M. M. Fraz, “Vision transformers in medical computer vision — a contemplative retrospection,” Engineering Applications of Artificial Intelligence, vol. 122, p. 106126, 2023.[9] M. Yip, S. Salcudean, K. Goldberg, K. Althoefer, A. Menciassi, J. D. Opfermann, A. Krieger, K. Swaminathan, C. J. Walsh, H. Huang et al., “Artificial intelligence meets medical robotics,” Science, vol. 381, no. 6654, pp. 141-146, 2023.

[0010] G. Bradski, “The OpenCV Library,” Dr. Dobb 's Journal of Software Tools, 2000.

Claims

CLAIMS1. A liquid volume sensor system for measuring a volume in a reservoir, the volume sensor system comprising: a camera configured to automatically find a volume label on the reservoir, the reservoir having volume markings thereon and configured to contain a liquid therein; and a computer unit configured to receive a video stream from the camera and to determine a volume of the liquid contained within the reservoir using a computer vision algorithm.

2. The liquid volume sensor system of claim 1 wherein the camera is a black and white camera, a color camera, a near-infrared night vision camera, an infrared thermal camera, a wide- angle camera, a fisheye camera, a 360 degree camera, or a combination thereof.

3. The liquid volume sensor system of claim 1 further comprising a lamp configured to illuminate the reservoir.

4. The liquid volume sensor system of claim 1 wherein the camera is configured to be positioned between 0.25 (0.0762 meters) to 3 feet (0.9144 meters) from the reservoir.

5. The liquid volume sensor system of claim 1 wherein the sensor system is configured to be used with a cardiopulmonary bypass (CPB) circuit.

6. The liquid volume sensor system of any one of claims 1 to 5 wherein at least the camera of the sensor system is configured to be mounted to a structure other than the reservoir.

7. The liquid volume sensor system of any one of claims 1 to 5 wherein at least the camera of the sensor system is configured to be mounted to the reservoir.

8. A method of measuring blood volume in a venous reservoir of a cardiopulmonary bypass (CPB) circuit operated by a heart-lung machine using the liquid volume sensor system of claim 1, the method comprising: capturing, with the camera, a video stream of the venous reservoir containing blood; inputting the video stream into the computer unit; using the computer vision algorithm to automatically find a volume label on the venous reservoir, the venous reservoir having volume markings thereon and configured to contain blood therein and determine the volume of the blood in the venous reservoir from image frames of the video stream; and outputting the determined volume of blood in the venous reservoir to the heart-lung machine.

9. The method of claim 8 further comprising processing the video stream with transformation and correction.

10. The method of claim 9 wherein the transformation and correction includes image rotation.

11. The method of claim 8 further comprising determining whether the liquid volume sensor system is in need of calibration.ORRU-OIOIPCT12. The method of claim 11 wherein determining whether the liquid volume sensor system is in need of calibration includes using optical character recognition (OCR) to identify the volume markings on the venous reservoir from the image frames and registering their positions on the image frame.

13. The method of any one of claims 11 or 12 further comprising calibrating the liquid volume sensor system when it is determined to be needed.

14. The method of any one of claims 11 or 12 wherein the computer vision algorithm is an Oriented FAST and Rotated BRIEF algorithm.

15. The method of claim 8 further comprising extracting a level of the blood in the venous reservoir.

16. The method of claim 15 wherein the extracting of the level of the blood in the venous reservoir is accomplished by hue, saturation, and value (HSV) masking, grayscale masking, contour detection, Canny -Hough line detection, motion detection, or a combination thereof.

17. The method of any one of claims 15 or 16 further comprising calculating the volume of the blood in the venous reservoir based on the extracted level of the blood relative to the volume markings on the venous reservoir.

18. The method of claim 8 wherein the method is conducted continuously in real time.

19. The method of claim 8 further comprising displaying the determined volume of blood in the venous reservoir to a perfusionist.

20. The method of claim 8 further comprising providing an alert if the computer vision algorithm determines that the camera does not have a clear view of the venous reservoir.

21. The method of claim 8 further comprising providing an alert if the blood volume sensor system determines that the volume of blood in the venous reservoir is outside an appropriate working range or if the volume of the blood in the venous reservoir changes rapidly.

22. A method of combining the volume label recognition and liquid / air interface detection for a reservoir defining a volume for containing a liquid, the method comprising: creating an image library; processing a video feed with transformation and rotation; acquiring images of the reservoir at a plurality of label positions and name files, each associated with a gradation of the volume; and using an Oriented FAST and Rotated BRIEF algorithm to compare features between a live video feed and the image library, wherein a file name of an image with the highest match represents an actual volume gradation of the liquid in real time.ORRU-OIOIPCT23. The method of claim 22 further comprising updating a volume label registration if the reservoir has been moved relative to the camera.

Images