System, method, and computer program product for tracking real-time syringe volume

Abstract

A system, method, and computer program product for tracking real-time syringe volume may: obtain a plurality of images of a syringe including a fiducial marker, the fiducial marker encapsulating syringe data associated with the syringe; determine, based on the fiducial marker in at least one image, an initial volume associated with the syringe; for each image of the syringe of the plurality of images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and subtract, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image; determine, based on the subtracted image generated from a current image, a volume of a fluid dispensed from and / or remaining in the syringe; and provide the volume of the fluid dispensed from and / or remaining in the syringe.

Description

SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR TRACKING REAU-TIME SYRINGE VOEUMECROSS-REFERENCE TO RELATED APPLICATION

[0001] The present application claims priority to United States Provisional Application No. 63 / 514,218 entitled “System, Method, and Computer Program Product for Tracking Real-Time Syringe Volume” filed July 18, 2023, the disclosure of which is hereby incorporated by reference in its entirety.BACKGROUND

[0002] Incorrect dosing and incorrect administration rate are two common intravenous (IV) push medication errors that can have significant impacts on patient safety and recovery.SUMMARY

[0003] Accordingly, provided are improved systems, devices, products, apparatus, and / or methods for tracking real-time syringe volume.

[0004] According to some non-limiting embodiments or aspects, provided is a system, including: at least one processor coupled to a memory and configured to: obtain a plurality of images of a syringe including a fiducial marker, wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the plurality images of the syringe, an initial volume associated with the syringe; for each image of the syringe of the plurality of images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and subtract, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image; determine, based on the subtracted image generated from a current image of the plurality of images, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

[0005] In some non-limiting embodiments or aspects, the plurality of images includes color images, and wherein the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: convert the region of interest of that image to a grayscale image.

[0006] In some non-limiting embodiments or aspects, the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: apply at least one morphological operation to the subtracted image to generate a morphed image, wherein the morphed image includes an area of interest associated with a moving object and defined by at least one contour, and wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the morphed image.

[0007] In some non-limiting embodiments or aspects, the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: determine, based on the morphed image, at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the at least one area of the moving object in the morphed image.

[0008] In some non-limiting embodiments or aspects, the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: filter, based on at least one area threshold, the at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the filtered at least one area of the moving object in the morphed image.

[0009] In some non-limiting embodiments or aspects, for each image of the syringe of the plurality of images of the syringe, the fiducial marker in that image identifies the region of interest in that image as a predetermined area in that image including a predetermined width relative to the fiducial marker in that image, a predetermined length relative to the fiducial marker in that image, and a predetermined location relative to the fiducial marker in that image.

[0010] In some non-limiting embodiments or aspects, the plurality of images of the syringe include at least one of the following: an infrared image, a microwave image, an ultrasound image, or any combination thereof.

[0011] According to some non-limiting embodiments or aspects, provided is a method, including: obtaining, with at least one processor, a plurality of images of a syringe including a fiducial marker, wherein the fiducial marker encapsulates syringe data associated with the syringe; determining, with the at least one processor, based on the fiducial marker in at least one image of the syringe of the plurality images of the syringe, an initial volume associated with the syringe; for each image of the syringe of the plurality of images of the syringe: determining, with the at least one processor, based on the fiducial marker in that image, a regionof interest in that image; and subtracting, with the at least one processor, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image; determining, with the at least one processor, based on the subtracted image generated from a current image of the plurality of images, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and providing, with the at least one processor, the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

[0012] In some non-limiting embodiments or aspects, the plurality of images includes color images, and wherein the method further includes: for each image of the syringe of the plurality of images of the syringe: converting, with the at least one processor, the region of interest of that image to a grayscale image.

[0013] In some non-limiting embodiments or aspects, the method further includes: for each image of the syringe of the plurality of images of the syringe: applying, with the at least one processor, at least one morphological operation to the subtracted image to generate a morphed image, wherein the morphed image includes an area of interest associated with a moving object and defined by at least one contour, and wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the morphed image.

[0014] In some non-limiting embodiments or aspects, the method further includes: for each image of the syringe of the plurality of images of the syringe: determining, with the at least one processor, based on the morphed image, at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the at least one area of the moving object in the morphed image.

[0015] In some non-limiting embodiments or aspects, the method further includes: for each image of the syringe of the plurality of images of the syringe: filtering, with the at least one processor, based on at least one area threshold, the at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the filtered at least one area of the moving object in the morphed image.

[0016] In some non-limiting embodiments or aspects, for each image of the syringe of the plurality of images of the syringe, the fiducial marker in that image identifies the region of interest in that image as a predetermined area in that image including a predetermined widthrelative to the fiducial marker in that image, a predetermined length relative to the fiducial marker in that image, and a predetermined location relative to the fiducial marker in that image.

[0017] In some non-limiting embodiments or aspects, the plurality of images of the syringe include at least one of the following: an infrared image, a microwave image, an ultrasound image, or any combination thereof.

[0018] According to some non-limiting embodiments or aspects, provided is a computer program product including at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: obtain a plurality of images of a syringe including a fiducial marker, wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the plurality images of the syringe, an initial volume associated with the syringe; for each image of the syringe of the plurality of images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and subtract, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image; determine, based on the subtracted image generated from a current image of the plurality of images, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

[0019] In some non-limiting embodiments or aspects, the plurality of images includes color images, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the plurality of images of the syringe: convert the region of interest of that image to a grayscale image.

[0020] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the plurality of images of the syringe: apply at least one morphological operation to the subtracted image to generate a morphed image, wherein the morphed image includes an area of interest associated with a moving object and defined by at least one contour, and wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the morphed image.

[0021] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the plurality of images of the syringe: determine, based on the morphedimage, at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the at least one area of the moving object in the morphed image.

[0022] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the plurality of images of the syringe: filter, based on at least one area threshold, the at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the filtered at least one area of the moving object in the morphed image.

[0023] In some non-limiting embodiments or aspects, for each image of the syringe of the plurality of images of the syringe, the fiducial marker in that image identifies the region of interest in that image as a predetermined area in that image including a predetermined width relative to the fiducial marker in that image, a predetermined length relative to the fiducial marker in that image, and a predetermined location relative to the fiducial marker in that image.

[0024] Further non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0025] Clause 1. A system, comprising: at least one processor coupled to a memory and configured to: obtain a plurality of images of a syringe including a fiducial marker, wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the plurality images of the syringe, an initial volume associated with the syringe; for each image of the syringe of the plurality of images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and subtract, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image; determine, based on the subtracted image generated from a current image of the plurality of images, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

[0026] Clause 2. The system of clause 1, wherein the plurality of images includes color images, and wherein the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: convert the region of interest of that image to a grayscale image.

[0027] Clause 3. The system of any of clauses 1 or 2, wherein the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: apply at least one morphological operation to the subtracted image to generate a morphed image, wherein the morphed image includes an area of interest associated with a moving object and defined by at least one contour, and wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the morphed image.

[0028] Clause 4. The system of any of clauses 1-3, wherein the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: determine, based on the morphed image, at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the at least one area of the moving object in the morphed image.

[0029] Clause 5. The system of any of clauses 1-4, wherein the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: filter, based on at least one area threshold, the at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the filtered at least one area of the moving object in the morphed image.

[0030] Clause 6. The system of any of clauses 1-5, wherein, for each image of the syringe of the plurality of images of the syringe, the fiducial marker in that image identifies the region of interest in that image as a predetermined area in that image including a predetermined width relative to the fiducial marker in that image, a predetermined length relative to the fiducial marker in that image, and a predetermined location relative to the fiducial marker in that image.

[0031] Clause 7. The system of any of clauses 1-6, wherein the plurality of images of the syringe include at least one of the following: an infrared image, a microwave image, an ultrasound image, or any combination thereof.

[0032] Clause 8. A method, comprising: obtaining, with at least one processor, a plurality of images of a syringe including a fiducial marker, wherein the fiducial marker encapsulates syringe data associated with the syringe; determining, with the at least one processor, based on the fiducial marker in at least one image of the syringe of the plurality images of the syringe, an initial volume associated with the syringe; for each image of the syringe of the plurality of images of the syringe: determining, with the at least one processor, based on the fiducial marker in that image, a region of interest in that image; and subtracting, with the at least one processor,from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image; determining, with the at least one processor, based on the subtracted image generated from a current image of the plurality of images, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and providing, with the at least one processor, the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

[0033] Clause 9. The method of clause 8, wherein the plurality of images includes color images, and wherein the method further comprises: for each image of the syringe of the plurality of images of the syringe: converting, with the at least one processor, the region of interest of that image to a grayscale image.

[0034] Clause 10. The method of any of clauses 8 or 9, further comprising: for each image of the syringe of the plurality of images of the syringe: applying, with the at least one processor, at least one morphological operation to the subtracted image to generate a morphed image, wherein the morphed image includes an area of interest associated with a moving object and defined by at least one contour, and wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the morphed image.

[0035] Clause 11. The method of any of clauses 8-10, further comprising: for each image of the syringe of the plurality of images of the syringe: determining, with the at least one processor, based on the morphed image, at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the at least one area of the moving object in the morphed image.

[0036] Clause 12. The method of any of clauses 8-11, further comprising: for each image of the syringe of the plurality of images of the syringe: filtering, with the at least one processor, based on at least one area threshold, the at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the filtered at least one area of the moving object in the morphed image.

[0037] Clause 13. The method of any of clauses 8-12, wherein, for each image of the syringe of the plurality of images of the syringe, the fiducial marker in that image identifies the region of interest in that image as a predetermined area in that image including a predetermined widthrelative to the fiducial marker in that image, a predetermined length relative to the fiducial marker in that image, and a predetermined location relative to the fiducial marker in that image.

[0038] Clause 14. The method of any of clauses 8-13, wherein the plurality of images of the syringe include at least one of the following: an infrared image, a microwave image, an ultrasound image, or any combination thereof.

[0039] Clause 15. A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: obtain a plurality of images of a syringe including a fiducial marker, wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the plurality images of the syringe, an initial volume associated with the syringe; for each image of the syringe of the plurality of images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and subtract, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image; determine, based on the subtracted image generated from a current image of the plurality of images, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

[0040] Clause 16. The computer program product of clause 15, wherein the plurality of images includes color images, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the plurality of images of the syringe: convert the region of interest of that image to a grayscale image.

[0041] Clause 17. The computer program product of any of clauses 15 or 16, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the plurality of images of the syringe: apply at least one morphological operation to the subtracted image to generate a morphed image, wherein the morphed image includes an area of interest associated with a moving object and defined by at least one contour, and wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the morphed image.

[0042] Clause 18. The computer program product of any of clauses 15-16, wherein the program instructions, when executed by the at least one processor, further cause the at leastone processor to: for each image of the syringe of the plurality of images of the syringe: determine, based on the morphed image, at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the at least one area of the moving object in the morphed image.

[0043] Clause 19. The computer program product of any of clauses 15-18, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the plurality of images of the syringe: filter, based on at least one area threshold, the at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the filtered at least one area of the moving object in the morphed image.

[0044] Clause 20. The computer program product of any of clauses 15-19, wherein, for each image of the syringe of the plurality of images of the syringe, the fiducial marker in that image identifies the region of interest in that image as a predetermined area in that image including a predetermined width relative to the fiducial marker in that image, a predetermined length relative to the fiducial marker in that image, and a predetermined location relative to the fiducial marker in that image.BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Additional advantages and details are explained in greater detail below with reference to the exemplary embodiments that are illustrated in the accompanying schematic figures, in which:

[0046] FIG. 1A is a diagram of non-limiting embodiments or aspects of an environment in which systems, devices, products, apparatus, and / or methods, described herein, can be implemented;

[0047] FIG. IB is a perspective view of implementations of non-limiting embodiments or aspects of a syringe;

[0048] FIG. 2 is a diagram of non-limiting embodiments or aspects of components of one or more devices and / or one or more systems of FIG. 1 A;

[0049] FIG. 3 is a flow chart of non-limiting embodiments or aspects of a process for tracking real-time syringe volume;

[0050] FIG. 4 is a flow chart of non-limiting embodiments or aspects of a process for tracking real-time syringe volume; and

[0051] FIGS. 5 A and 5B illustrate an implementation of non-limiting embodiments or aspects of a region of interest of images of a syringe.DETAILED DESCRIPTION

[0052] It is to be understood that the present disclosure may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary and non-limiting embodiments or aspects. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0053] For purposes of the description hereinafter, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and derivatives thereof shall relate to embodiments or aspects as they are oriented in the drawing figures. However, it is to be understood that embodiments or aspects may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply non-limiting exemplary embodiments or aspects. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects of the embodiments or aspects disclosed herein are not to be considered as limiting unless otherwise indicated.

[0054] No aspect, component, element, structure, act, step, function, instruction, and / or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and may be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise.

[0055] As used herein, the terms “communication” and “communicate” may refer to the reception, receipt, transmission, transfer, provision, and / or the like of information (e.g., data, signals, messages, instructions, commands, and / or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) to be incommunication with another unit means that the one unit is able to directly or indirectly receive information from and / or transmit information to the other unit. This may refer to a direct or indirect connection that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and / or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit (e.g., a third unit located between the first unit and the second unit) processes information received from the first unit and communicates the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet and / or the like) that includes data. It will be appreciated that numerous other arrangements are possible.

[0056] As used herein, the term “computing device” may refer to one or more electronic devices that are configured to directly or indirectly communicate with or over one or more networks. A computing device may be a mobile or portable computing device, a desktop computer, a server, and / or the like. Furthermore, the term “computer” may refer to any computing device that includes the necessary components to receive, process, and output data, and normally includes a display, a processor, a memory, an input device, and a network interface. A “computing system” may include one or more computing devices or computers. An “application” or “application program interface” (API) refers to computer code or other data sorted on a computer-readable medium that may be executed by a processor to facilitate the interaction between software components, such as a client-side front-end and / or server-side back-end for receiving data from the client. An “interface” refers to a generated display, such as one or more graphical user interfaces (GUIs) with which a user may interact, either directly or indirectly (e.g., through a keyboard, mouse, touchscreen, etc.). Further, multiple computers, e.g., servers, or other computerized devices directly or indirectly communicating in the network environment may constitute a “system” or a “computing system”.

[0057] It will be apparent that systems and / or methods, described herein, can be implemented in different forms of hardware, software, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, itbeing understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0058] Some non-limiting embodiments or aspects are described herein in connection with thresholds. As used herein, satisfying a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0059] Non-limiting embodiments or aspects of the present disclosure may provide systems, methods, and computer program products that can determine an amount of fluid (e.g., liquid medication, etc.) delivered from and / or remaining in a syringe and / or can verify whether a medication delivery is complete in syringes of different volumes in real-time using video / image processing and fiducial markers. For example, non-limiting embodiments or aspects of the present disclosure may obtain a plurality of images of a syringe including a fiducial marker, wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the plurality images of the syringe, an initial volume associated with the syringe; for each image of the syringe of the plurality of images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and subtract, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image; determine, based on the subtracted image generated from a current image of the plurality of images, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

[0060] In this way, non-limiting embodiments or aspects of the present disclosure may use image processing techniques and / or algorithms to process images or frames in a manner that reduces a computation load, thereby enabling edge computation; may use a fiducial marker to detect a syringe and classify a volume under medication and / or medication details; may place a position of the syringe at a defined angle to an image capture device to improve design; may be used with syringes of various types and / or volumes with minimal changes or no modifications; may avoid calibration processes, thereby making detection and tracking of a syringe volume more efficient; may use existing off-the-shelf image capture devices without requiring infrared and / or thermal imaging devices.

[0061] Referring now to FIG. 1A, FIG. 1A is a diagram of an example environment 100 in which devices, systems, methods, and / or products described herein, may be implemented. As shown in FIG. 1A, environment 100 includes user device 102, syringe 103, management system 104, and / or communication network 106. Systems and / or devices of environment 100 can interconnect via wired connections, wireless connections, or a combination of wired and wireless connections.

[0062] User device 102 may include one or more devices capable of receiving information and / or data from management system 104 (e.g., via communication network 106, etc.) and / or communicating information and / or data to management system 104 (e.g., via communication network 106, etc.). For example, user device 102 may include one or more computing systems including one or more processors (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, one or more tablet computers, etc.). In some non-limiting embodiments or aspects, user device 102 may include a tablet computer or a mobile computing device, such as an Apple® iPad, an Apple® iPhone, an Android® tablet, an Android® phone, and / or the like.

[0063] User device 102 may include one or more image capture devices (e.g., one or more cameras, one or more sensors, etc.) configured to capture one or more images of an environment (e.g., environment 100, etc.) surrounding the one or more image capture devices. For example, user device 102 may include one or more image capture devices configured to capture one or more images of syringe 103 (e.g., a plurality of images, a series of images captured over a period of time, etc.). As an example, an image capture device of user device 102 may include at least one of the following: a plurality of image capture devices, a monocular camera, a stereo camera, a color camera configured to capture and / or detect one or more predetermined wavelengths of light, a camera including a filter configured to filter a predetermined wavelength of light, an infrared (IR) camera (e.g., a sensor configured to detect a changes in temperatures based on which heat and / or movement may be detected, etc.), a thermal sensor configured to capture thermal images of syringe 103 illuminated by infrared wavelengths of a light source, a pan, tilt, and zoom (PTZ) camera including a variable field-of- view (FOV) and an automatic zoom function, a master-slave camera system including a static camera and a dynamic camera, a LiDAR system, a RADAR system, a microwave sensor (a sensor configured to detect movement of objects by emitting microwave pulses and measuring reflected microwaves of a moving object, etc.), an ultrasonic motion sensor, and / or the like, or any combination thereof.

[0064] In some non-limiting embodiments or aspects, an image capture device of user device 102 includes a stereo camera. For example 3D position information associated with a 3D position of syringe 103 and / or components thereof (e.g., of plunger 134, of stopper 136, of fiducial marker 138, etc.) relative to the image capture device (and / or user device 102) may be determined using a Structure from Motion (SfM) algorithm. As an example, user device 102 may include a stereo camera setup, which is available in many mobiles devices, such Apple® iPads, Apple® iPhones, Android® tablets, Android® phones, and / or the like. User device 102 may process images from the stereo camera using SfM algorithms to extract 3D information which may enhance object feature recognition of fiducial marker 138 and / or of the components of syringe 103. As an example, the SfM processing may improve extraction of the 3D features from the components of syringe 103, which may improve chances of image feature accumulation, e.g., by using a burst image capture / video mode which captures images in a proper direction / registration (e.g., pan / tilt, etc.) based on the setup or location of syringe 103. In such examples, user device 102 may generate a 3D depth or point cloud of syringe 103 using the stereo camera and detect and classify components thereof based on the depth information of the 3D point cloud and structural feature mapping of the syringe components. For example, user device 102 may use depth information to detect and track the distance between plunger 134 and the image capture device (and / or user device 102) and / or to map the real-time volume of the medication inside the syringe, which may enhance an accuracy of the tracking of the plunger depth and movement to provide better pixel-to-volume mapping for tracking the volume of the medication in the syringe.

[0065] In some non-limiting embodiments or aspects, an image capture device of user device 102 includes a LiDAR system and / or a RADAR system. For example, an image may include a LiDAR point cloud and / or a RADAR point cloud. As an example, user device 102 may include a mini-LiDAR system, which is available in many mobiles devices, such Apple® iPads, Apple® iPhones, Android® tablets, Android® phones, and / or the like. In such an example, use of LiDAR images may improve accuracy for 3D feature detection because LiDAR images directly provide 3D world information as point clouds, which may accelerate the 3D data collection with reduced or minimum protocols when compared to stereo setup. For example, user device 102 may use existing image registration and / or transformation techniques to overlap 2D object information from camera images, such as color, texture and / or the like, with the 3D LiDAR point cloud to detect 3D features to detect and track the distance between the syringe components and the image capture device (and / or user device 102) for mapping the real-time volume of the medication inside syringe 103, which may enhance an accuracy of thetracking of the plunger depth and movement to provide better pixel-to-volume mapping for tracking the volume of the medication in the syringe.

[0066] Referring also to FIG. IB, which is a perspective view of an implementation of nonlimiting embodiments or aspects of syringe 103, syringe 103 may include syringe barrel 132, plunger rod 134, and / or stopper 136. Syringe barrel 132 may extend between a proximal end including a proximal opening configured to receive stopper 136 and / or plunger rod 134, and a distal end including a distal opening (e.g., a luer lock opening, etc.) configured to be connected to an IV line or catheter (e.g., via a needless connector, etc.) and / or a needle or cannula and through which a fluid (e.g. a medication, etc.) can be expelled from syringe 103.

[0067] Syringe 103 may include fiducial marker 138 (e.g., a tag, a label, a code, etc.). Fiducial marker 138 may be associated with (e.g., removably attached to, permanently attached to, integrated with, implemented on, etc.) syringe 103. For example, as shown in FIG. IB, fiducial marker 138 may be located on an exterior surface of syringe barrel 132 of syringe 103 such that fiducial marker 138 is included in an image of syringe 103 captured by an image capture device. In some non-limiting embodiments or aspects, syringe 103 may include a plurality of fiducial markers 138 arranged about an axis of syringe 103, which may enable at least one fiducial marker 138 being presented to a FOV of an image capture device regardless of an orientation of the syringe 103.

[0068] Fiducial marker 138 may encapsulate syringe data associated with syringe 103. For example, syringe data may include at least one of the following parameters associated with syringe 103: a volume of syringe 103 (e.g., an initial volume of fluid contained in syringe 103, etc.), a size of syringe 103, a type of syringe 103 (e.g., a syringe type identifier, etc.), a unique identifier of syringe 103 (e.g., that uniquely identifies syringe 103 from other syringes, etc.), a type of medication contained in syringe 103, a concentration of medication contained in syringe 103, a prescribed rate of delivery of the medication contained in syringe 103, and / or the like, or any combination thereof.

[0069] Fiducial marker 138 may define or identify a region of interest in an image that has predetermined width relative to fiducial marker 138 in the image, a predetermined length relative to fiducial marker 138 in the image, and / or a predetermined location relative to fiducial marker 138 in the image. For example, and referring also to FIGS. 5A and 5B, which illustrate an implementation 500 of non-limiting embodiments or aspects of a region of interest of images of a syringe, fiducial marker 138 defines or identifies a region of interest (ROI) in each image that has a predetermined width relative to fiducial marker 138 in that image, a predeterminedlength relative to fiducial marker 138 in that image, and / or a predetermined location relative to fiducial marker 138 in that image.

[0070] Fiducial marker 138 may encapsulate pose information associated with a 3D position of the fiducial marker 138. For example, fiducial marker 138 may include markings that, when captured in an image, enable computing of a precise 3D position of fiducial marker 138 with respect to an image capture device that captured the image (e.g., an x, y, z coordinate position of the fiducial marker 138, etc.) and / or a precise 2D position of fiducial marker 138 in the image itself (e.g., an x, y coordinate positon of fiducial marker 138 in the image, etc.).

[0071] In some non-limiting embodiments or aspects, fiducial marker 138 includes an AprilTag. For example, fiducial marker 138 may include an AprilTag V3 of type custom Tag 48hl2, which enables using AprilTag V3 detection to determine a unique ID, which may indicate a type of syringe 103 associated with fiducial marker 138 (e.g., in leading digits, etc.) and / or a unique serial number for that specific syringe 103 (e.g., in the trailing digits, etc.), and / or a location (e.g., x, y, and z coordinates, directional vectors for Z, Y, and X axes, etc.) of fiducial marker 138 in a field-of-view (FOV) of an image capture device. As an example, an AprilTag may store information (e.g., tag ID, syringe ID, etc.), while also enabling simple and accurate 6D (x, y, z, roll, pitch, yaw) pose estimation of the tag.

[0072] However, non-limiting embodiments or aspects are not limited thereto, and fiducial marker 138 may include a QR code, a barcode (e.g., a ID barcode, a 2D barcode, etc.), an Aztec code, a Data Matrix code, an ArUco marker, a colored pattern, a reflective pattern, a fluorescent pattern, a predetermined shape and / or color (e.g., a red pentagon, a blue hexagon, etc.), an LED pattern, a hologram, and / or the like that encapsulates syringe data associated with syringe 103 and / or encapsulates pose information associated with a 2D and / or 3D position of fiducial marker 138.

[0073] In some non-limiting embodiments or aspects, syringe 103 may be maintained at a same position and / or orientation relative to an image capture device of user device 102 to provide a consistent region of interest and a fixed FOV. For example, syringe 103 may be maintained at the same position and / or orientation relative to the image capture device of user device 102 (e.g., via a hardware configuration or fixture, etc.) as the image capture devices captures a series of images of syringe 103 over a period of time for tracking a real-time volume of fluid dispensed from and / or remaining in syringe 103. However, non-limiting embodiments or aspects of the present disclosure are not limited thereto, and in some non-limiting embodiments or aspects a position and / or orientation of syringe 103 relative to an image capture device of user device 102 may vary (e.g., due to being held in a hand of a user, etc.) asthe image capture device captures a series of images of syringe 103 over a period of time for tracking a real-time volume of fluid dispensed from and / or remaining in syringe 103.

[0074] Management system 104 may include one or more devices capable of receiving information and / or data from user device 102 (e.g., via communication network 106, etc.) and / or communicating information and / or data to user device 102 (e.g., via communication network 106, etc.). For example, management system 104 may include one or more computing systems including one or more processors (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, etc.). In some non-limiting embodiments or aspects, management system 104 includes and / or is accessible via a nurse station or terminal in a hospital. For example, management system 104 may provide bedside nurse support, nursing station manager support, retrospective reporting for nursing administration, and / or the like.

[0075] Communication network 106 may include one or more wired and / or wireless networks. For example, communication network 106 may include a cellular network (e.g., a long-term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a sixth generation (6G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber optic -based network, a cloud computing network, and / or the like, and / or a combination of these or other types of networks.

[0076] The number and arrangement of systems and devices shown in FIGS. 1A and IB are provided as an example. There can be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, or differently arranged systems and / or devices than those shown in FIGS. 1A and IB. Furthermore, two or more systems or devices shown in FIGS. 1A and IB can be implemented within a single system or a single device, or a single system or a single device shown in FIGS. 1A and IB can be implemented as multiple, distributed systems or devices. Additionally, or alternatively, a set of systems or a set of devices (e.g., one or more systems, one or more devices, etc.) of environment 100 can perform one or more functions described as being performed by another set of systems or another set of devices of environment 100.

[0077] Referring now to FIG. 2, FIG. 2 is a diagram of example components of a device 200. Device 200 may correspond to user device 102 (e.g., one or more devices of a system of user device 102, etc.) and / or one or more devices of management system 104. In some non-limitingembodiments or aspects, user device 102 (e.g., one or more devices of a system of user device 102, etc.) and / or one or more devices of management system 104 may include at least one device 200 and / or at least one component of device 200. As shown in FIG. 2, device 200 may include bus 202, processor 204, memory 206, storage component 208, input component 210, output component 212, and communication interface 214.

[0078] Bus 202 may include a component that permits communication among the components of device 200. In some non-limiting embodiments or aspects, processor 204 may be implemented in hardware, software, or a combination of hardware and software. For example, processor 204 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an application- specific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 206 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 204.

[0079] Storage component 208 may store information and / or software related to the operation and use of device 200. For example, storage component 208 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid state disk, etc.), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of computer-readable medium, along with a corresponding drive.

[0080] Input component 210 may include a component that permits device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally or alternatively, input component 210 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, an image capture device, etc.). Output component 212 may include a component that provides output information from device 200 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.).

[0081] Communication interface 214 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 214 may permit device 200 to receive information from another device and / or provide information to another device. For example, communication interface 214 may include an Ethernet interface, an optical interface,a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0082] Device 200 may perform one or more processes described herein. Device 200 may perform these processes based on processor 204 executing software instructions stored by a computer-readable medium, such as memory 206 and / or storage component 208. A computer- readable medium (e.g., a non-transitory computer-readable medium) is defined herein as a non- transitory memory device. A memory device includes memory space located inside of a single physical storage device or memory space spread across multiple physical storage devices.

[0083] Software instructions may be read into memory 206 and / or storage component 208 from another computer-readable medium or from another device via communication interface 214. When executed, software instructions stored in memory 206 and / or storage component 208 may cause processor 204 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments or aspects described herein are not limited to any specific combination of hardware circuitry and software.

[0084] Memory 206 and / or storage component 208 may include data storage or one or more data structures (e.g., a database, etc.). Device 200 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage or one or more data structures in memory 206 and / or storage component 208.

[0085] The number and arrangement of components shown in FIG. 2 are provided as an example. In some non-limiting embodiments or aspects, device 200 may include additional components, fewer components, different components, or differently arranged components than those shown in FIG. 2. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0086] Referring now to FIG. 3, FIG. 3 is a flowchart of non-limiting embodiments or aspects of a process 300 for tracking real-time syringe volume. In some non-limiting embodiments or aspects, one or more of the steps of process 300 may be performed (e.g., completely, partially, etc.) by user device 102 (e.g., one or more devices of a system of user device 102, etc.). In some non-limiting embodiments or aspects, one or more of the steps of process 300 may be performed (e.g., completely, partially, etc.) by another device or a group of devices separate from or including user device 102, such as management system 104 (e.g., one or more devices of management system 104, etc.).

[0087] As shown in FIG. 3, at step 302, process 300 includes obtaining a plurality of images of a syringe including a fiducial marker. For example, user device 102 (and / or management system 104) may obtain a plurality of images of syringe 103 including fiducial marker 138. The plurality of images of syringe 103 may include a current image of syringe 103 and a previous image of syringe 130 (e.g., one or more previous images, a plurality of previous images, etc.) captured before the current image of syringe 103 (e.g., captured before the current image of syringe 103 in a time series of images including the plurality of images captured over a period of time, such as one or more earlier images and / or a first image of syringe 103 in the time series of images of syringe 103, and / or the like, etc.). In such examples, fiducial marker 138 may encapsulate syringe data associated with syringe 103.

[0088] An image may include at least one of the following: a digital image (e.g., a TIFF, a JPEG, a GIF, a PNG, and a raw image file, etc.), an infrared image, a microwave image (e.g., an image generated by a microwave sensor using a microwave imaging technique, etc.), an ultrasound image, a LiDAR image or point cloud, a RADAR image or point cloud, or any combination thereof, and / or the like.

[0089] As shown in FIG. 3, at step 304, process 300 includes determining, based on the fiducial marker in at least one image of the syringe, an initial volume associated with the syringe. For example, user device 102 (and / or management system 104) may determine, based on fiducial marker 138 in at least one image of syringe 103 of the plurality of images of syringe 103 (e.g., in the previous image of syringe 103, in the current image of syringe 103, etc.), an initial volume associated with syringe 103. As an example, user device 102 (and / or management system 104) may process the at least one image of syringe 103 to detect fiducial marker 138 therein, read, from fiducial marker 138 in the at least one image of syringe 103, the syringe data associated with syringe 103 to classify a type and / or an initial or full volume of syringe 103, and / or retrieve the initial or full volume of syringe 103 based on the type and / or an identifier associated with syringe 103.

[0090] As shown in FIG. 3, at step 306, process 300 includes, for each image of the syringe, determining, based on the fiducial marker in that image, a region of interest in that image. For example, user device 102 (and / or management system 104) may, for each image of syringe 103 of the plurality of images of syringe 103 (e.g., for the previous image of syringe 103, for the current image of syringe 103, etc.), determine, based on fiducial marker 138 in that image, a region of interest in that image. As an example, fiducial marker 138 may define or identify a region of interest in an image that has a predetermined width relative to fiducial marker 138 in the image, a predetermined length relative to fiducial marker 138 in the image, and / or apredetermined location relative to fiducial marker 138 in the image. For example, for each image of the syringe 103 of the plurality of images of the syringe 103, fiducial marker 138 in that image may identify the region of interest in that image as a predetermined area in that image including a predetermined width relative to fiducial marker 138 in that image (e.g., a predetermined pixel width relative to a pixel dimension of fiducial marker 138 in that image, etc.), a predetermined length relative to the fiducial marker 138 in that image (e.g., a predetermined pixel length relative to a pixel dimension of fiducial marker 138 in that image, etc.), and / or a predetermined location relative to the fiducial marker 138 in that image (e.g., a predetermined pixel location relative to a pixel dimension of fiducial marker 138 in that image, etc.). In such an example, as shown in FIGS. 5A and 5B, the region of interest in the image may be a movement path of stopper 136 and / or plunger 134 from an initial or full syringe volume position thereof (e.g., as shown in FIG. 5A) to a final or fully dispensed syringe volume position thereof (e.g., as shown in FIG. 5B). For example, fiducial marker 138 may identify or define, as a region of interest, a middle section of syringe body 132 such that only this exact region of interest of each image is processed instead of processing the full image, thereby reducing a processing load associated with processing the images and / or increasing an accuracy when detecting stopper 136 and / or plunger 134 in the images.

[0091] In some non-limiting embodiments or aspects, user device 102 (and / or management system 104) may, for each image of syringe 103 of the plurality of images of syringe 103 (e.g., for the previous image of syringe 103, for the current image of syringe 103, etc.), determine, based on fiducial marker 138 in that image, position information associated with a 3D position of fiducial marker 138 (and / or syringe 103) relative to the image capture device and / or a 2D position of fiducial marker 138 (and / or syringe 103) in the image itself. For example, user device 102 (and / or management system 104) may determine the position information associated with the 3D position of fiducial marker 138 (and / or syringe 103) relative to the image capture device and / or the 2D position of the fiducial marker 138 (and / or syringe 103) in the image itself by determining or identifying, based on the image, fiducial marker 138 and the pose information associated with the 3D positions and / or 2D positions of fiducial marker 138. For example, fiducial marker 138 may include an AprilTag, and user device 102 (and / or management system 104) may process the image using AprilTag detection software to determine syringe data encapsulated by fiducial marker 138 and / or a unique serial number for syringe 103 and to compute a precise 3D position, orientation, and / or identity of fiducial marker 138 (and / or syringe 103) relative to the image capture device that captured the image and / or a precise 2D position of fiducial marker 138 (and / or syringe 103) in the image itself.As an example, the position information associated with the 3D position of the fiducial marker 138 (and / or syringe 103) relative to the image capture device may be determined as the 3D position of the fiducial marker 138 associated with syringe 103 relative to the image capture device and / or the position information associated with the 2D positon of syringe 103 in the image itself may be determined as the 2D position of fiducial marker 138 associated with syringe 103. In such an example, the 3D position of fiducial marker 138 (and / or syringe 103) may include x, y, and z coordinates of fiducial marker 138 and / or directional vectors for Z, Y, and X axes of fiducial marker 138. In such an example, the 2D position of fiducial marker 138 in the image itself may include x, y coordinates of fiducial marker 138 in the image and / or direction vectors for Y and X axes of fiducial marker 138.

[0092] In some non-limiting embodiments or aspects, user device 102 (and / or management system 104) may, for each image of syringe 103 of the plurality of images of syringe 103 (e.g., for the previous image of syringe 103, for the current image of syringe 103, etc.), determine position information associated with a 3D position fiducial marker 138 (and / or syringe 103) relative to the image capture device using a stereo camera, a LiDAR system, a RADAR system, and / or the like generate a 3D depth or point cloud of syringe 103.

[0093] As shown in FIG. 3, at step 308, process 300 includes, for each image of the syringe, applying a frame subtraction to the region of interest in that image to generate a subtracted image. For example, user device 102 (and / or management system 104) may, for each image of syringe 103 of the plurality of images of syringe 103 (e.g., for the previous image of syringe 103, for the current image of syringe 103, etc.), apply a frame subtraction to the region of interest in that image to generate a subtracted image. As an example, user device 102 (and / or management system 104) may take the current image of syringe 103 and the previous image of syringe 103 as input and generate a subtracted image including pixel values that are equal to the pixel values of the current image minus the corresponding pixel values of the previous image. In this way, usage of a frame or pixel subtraction method may render non-limiting embodiments or aspects computationally less expensive than solutions which are based on other motion detection algorithms, such as a background subtraction method, a background estimation method, an optical flow method, a Gaussian Mixture Model-based foreground and background segmentation, and / or the like. Moreover, usage of fiducial markers enables nonlimiting embodiments or aspects to more quickly find a region of in interest in each image, which may avoid the additional computational overhead of finding and discarding the background of the image as required by other motion detection solutions.

[0094] Further details regarding non-limiting embodiments or aspects of step 308 of process 300 are provided below with regard to FIG. 4.

[0095] Referring now to FIG. 4, FIG. 4 is a flowchart of non-limiting embodiments or aspects of a process 400 for tracking real-time syringe volume. In some non-limiting embodiments or aspects, one or more of the steps of process 400 may be performed (e.g., completely, partially, etc.) by user device 102 (e.g., one or more devices of a system of user device 102, etc.). In some non-limiting embodiments or aspects, one or more of the steps of process 300 may be performed (e.g., completely, partially, etc.) by another device or a group of devices separate from or including user device 102, such as management system 104 (e.g., one or more devices of management system 104, etc.).

[0096] As shown in FIG. 4, at step 402, process 400 includes, for each image of the syringe, converting the region of interest of that image to a grayscale image. For example, user device 102 (and / or management system 104) may, for each image of syringe 103 of the plurality of images of syringe 103 (e.g., for the previous image of syringe 103, for the current image of syringe 103, etc.), convert the region of interest of that image to a grayscale image (and / or convert that entire image to a grayscale image, etc.). As an example, the plurality of images of syringe 103 (e.g., the previous image of syringe 103, the current image of syringe 103, etc.) may include color images (e.g., RGB images, HSV images, etc.), and user device 102 (and / or management system 104) may convert the region of interest in those color images to grayscale images (e.g., using OpenCV, etc.).

[0097] As shown in FIG. 4, at step 404, process 400 includes, for each image of the syringe, subtracting, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image. For example, user device 102 (and / or management system 104) may, for each image of syringe 103 of the plurality of images of syringe 103 (e.g., for the previous image of syringe 103, for the current image of syringe 103, etc.), subtract, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image. As an example, user device 102 (and / or management system 104) may, for each image of syringe 103 of the plurality of images of syringe 103 (e.g., for the previous image of syringe 103, for the current image of syringe 103, etc.), apply a frame subtraction method in which a digital numeric value of one pixel or a whole image (e.g., each pixel of the previous image, etc.) is subtracted from another image (e.g., subtracted from each corresponding pixel of the current image to detect changes between the two images (e.g., an area of interest associated with a moving object, etc.), which may be used to determine if stopper 136 and / or plunger 134 moved. For example, framesubtraction may detect moving objects in a sequence of frames from static cameras by detecting moving objects from the difference between the current frame and reference or previous frame, which is often called a ‘Background Image’ or ‘Background Model’. In some non-limiting embodiments or aspects, an image capture device and syringe body 132 may be static and only stopper 136 and / or plunger 134 may move between images to deliver a medication or fluid from syringe 103. In this way, frame subtraction of the current frame with the previous frame may enable understanding of the movement of stopper 136 and / or plunger 134.

[0098] As shown in FIG. 4, at step 406, process 400 includes, for each image of the syringe, applying at least one morphological operation to the subtracted image to generate a morphed image. For example, user device 102 (and / or management system 104) may, for each image of syringe 103 of the plurality of images of syringe 103 (e.g., for the previous image of syringe 103, for the current image of syringe 103, etc.), apply at least one morphological operation to the subtracted image to generate a morphed image. The morphed image may include an area of interest associated with a moving object (e.g., stopper 136 and / or plunger 134, etc.) and / or defined by at least one contour. As an example, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof may be determined based on the morphed image. For example, the subtracted image or frame may include a black and white or grayscale image with common static pixels having become zero (e.g., black pixels, etc.) and moved object pixels having become non-zero (e.g., white pixels, etc.). In such an example, at least one morphological operation may be applied to the subtracted image or frame to enhance the area of interest associated with the moving object which may be defined by at least one contour or outline representing or bounding the area of interest. For example, a contour may include an outline representing or bounding a shape or form of something, and one or more contours defining an area of interest associated with the moving object may be generated by applying the at least one morphological operation to the subtracted image. As an example, these generated contours may be continuous and / or include edges with an identity and geometrical parameters.

[0099] Morphology may include a comprehensive set of image processing operations that process images based on shapes. Morphological operations may apply a structuring element to an input image, creating an output image of the same size. In a morphological operation, the value of each pixel in the output image may be based on a comparison of the corresponding pixel in the input image with its neighbors. A morphological operation may include an erosion operation, a dilation operation, an opening operation, a closing operation, and / or the like.

[0100] As shown in FIG. 4, at step 408, process 400 includes, for each image of the syringe, determining, based on the morphed image, at least one area defined by the at least one contour in the morphed image. For example, user device 102 (and / or management system 104) may, for each image of syringe 103 of the plurality of images of syringe 103 (e.g., for the previous image of syringe 103, for the current image of syringe 103, etc.), determine, based on the morphed image, at least one area defined by the at least one contour in the morphed image. As an example, a contour area may be calculated by counting a number of pixels in the area enclosed inside a closed contour. In such an example, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof may be determined based on the area of the moving object in the morphed image.

[0101] As shown in FIG. 4, at step 410, process 400 includes, for each image of the syringe, filtering, based on at least one area threshold, the at least one area defined by the at least one contour in the morphed image. For example, user device 102 (and / or management system 104) may, for each image of syringe 103 of the plurality of images of syringe 103 (e.g., for the previous image of syringe 103, for the current image of syringe 103, etc.), filter, based on at least one area threshold, the at least one area defined by the at least one contour in the morphed image. As an example, there may be multiple contours from the area of contour calculation output, and user device 102 (and / or management system 104) may apply the at least one area threshold to filter unwanted contours and keep an area of stopper 136 and / or plunger 134. In such an example, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the filtered at least one area of the moving object in the morphed image.

[0102] As shown in FIG. 3, at step 310, process 300 includes determining, based on the subtracted image generated from a current image of the plurality of images, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof. For example, user device 102 (and / or management system 104) may determine, based on the subtracted image generated from current image of the plurality of images, at least one of a volume of a fluid dispensed from syringe 103, a volume of a fluid remaining in syringe 103, or any combination thereof. As an example, user device 102 (and / or management system 104) may determine, based on the filtered at least one area of the moving object in the morphed image, at least one of a volume of a fluid dispensed from syringe 103, a volume of a fluid remaining in syringe 103, or any combination thereof. In such an example, user device 102 (and / or management system 104) may track the filtered at least one area of the moving object in the morphed image generated from each image over the plurality of images(e.g., the contour(s) output after the area thresholds are applied per image or frame, etc.) to calculate a direction and position of the at least one area of the moving object to generate a localization that may be used to estimate the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in the syringe 103, or any combination thereof. For example, user device 102 (and / or management system 104) may use OpenCV to calculate a direction and position of the at least one area of the moving object to generate a localization that may be used to estimate the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in the syringe 103, or any combination thereof.

[0103] As an example, assuming the region of interest represents a volume, which may be initially set or assumed as the full or initial volume of syringe 103, in each image, that volume may be “filled” with pixels. When fluid is delivered between a first image and a second image, a portion of the pixels representing an undelivered volume of fluid is present in each of the first image and the second image. The subtraction image may omit these pixels and / or report a pixel value for these pixel locations as zero. A portion of the pixels within the region of interest that remain may represent the portion of the fluid expelled from the syringe (e.g., delivered to the patient). User device 102 (and / or management system 104) may assess a number of pixels remaining in the subtraction image and / or a number of pixels with a value corresponding to a threshold value. The number of pixels, alone or in conjunction with information about syringe 103 identified from the fiducial marker (e.g., the initial volume, known dimensions of the fiducial marker used for scaling, etc.), may be used to estimate the volume delivered. In some implementations, the percentage of pixels remaining within the subtraction image may be used to calculate a volume delivered. For example, a percentage estimation to identify a volume delivered between a first time (t 1) and a second time (t2) may be determined according to the following Equation (1):where / 1 is the first time, tl is the second time subsequent to the first time / 1 , v is the volume, c is the container, z is the number of pixels in the region of interest, and zx- zyis a number of pixels in the subtraction image with a value exceeding a threshold value, and r is a total number of pixels in the region of interest.

[0104] As an example, if a tl - t2 subtraction image includes 40 pixels with values exceeding the threshold value (e.g., not 0), and the total region of interest encompasses 400 pixels, it can be estimated that 10% of the volume was lost between tl and tl. If the fiducial marker indicates the volume of the container (vc) is 50 mL syringe, the volume delivered between tl and tl may be estimated to be 10% of 50mL or 5 mL.

[0105] As shown in FIG. 3, at step 312, process 300 includes, providing the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof. For example, user device 102 (and / or management system 104) may provide the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof. As an example, user device 102 (and / or management system 104) may control a display (e.g., output component 212, etc.) to display, in real-time, the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof. For example, as shown in FIGS. 5 A and 5B, the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof may be displayed in real-time (e.g., overlaid with a region of interest on an image of syringe 103 to visualize tracking of stopper 136 and / or plunger 134 in real-time, etc.) as stopper 136 and / or plunger 134 move relative to syringe barrel 132 over time in a series of images of syringe 103 as fluid is dispensed from syringe 103.

[0106] In some non-limiting embodiments or aspects, user device 102 (and / or management system 104) may compare the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof (and / or a rate of fluid delivery determined based thereon) to one or more thresholds and, if the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof (and / or the rate of fluid delivery determined based thereon) violates the one or more thresholds, provide an alert associated therewith (e.g., via user device 102, etc.).

[0107] In some non-limiting embodiments or aspects, user device 102 (and / or management system 104) may aggregate the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof (and / or a rate of fluid delivery determined based thereon) from a plurality of injections or fluid delivery procedures to determine one or more metrics associated with a syringe type and / or volume of syringe 103 and / or a user that administered the plurality of injections.

[0108] In some non-limiting embodiments or aspects, user device 102 (and / or management system 104) may aggregate the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof (and / or a rate of fluid delivery determined based thereon) from a injection or fluid delivery procedure to determine one or more metrics associated with a the injection or fluid delivery procedure. For example, multiple volume measurements taken over a period of time may be used to calculate a flow rate for the fluid delivery. The flow rate may be used for one or more processes or devices such as to: control delivery of the fluid in real time, assess safety of the fluid delivery (e.g., not exceeding a flow rate threshold), or analyzed to determine clinical best practices or to identify clinician compliance with administration guidelines.

[0109] Although embodiments or aspects have been described in detail for the purpose of illustration and description, it is to be understood that such detail is solely for that purpose and that embodiments or aspects are not limited to the disclosed embodiments or aspects, but, on the contrary, are intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect. In fact, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

Claims

WHAT IS CLAIMED IS:

1. A system, comprising: at least one processor coupled to a memory and configured to: obtain a plurality of images of a syringe including a fiducial marker, wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the plurality images of the syringe, an initial volume associated with the syringe; for each image of the syringe of the plurality of images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and subtract, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image; determine, based on the subtracted image generated from a current image of the plurality of images, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

2. The system of claim 1, wherein the plurality of images includes color images, and wherein the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: convert the region of interest of that image to a grayscale image.

3. The system of claim 1, wherein the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: apply at least one morphological operation to the subtracted image to generate a morphed image, wherein the morphed image includes an area of interest associated with a moving object and defined by at least one contour, and wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the morphed image.

4. The system of claim 3, wherein the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: determine, based on the morphed image, at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the at least one area of the moving object in the morphed image.

5. The system of claim 4, wherein the at least one processor is further configured to: for each image of the syringe of the plurality of images of the syringe: filter, based on at least one area threshold, the at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the filtered at least one area of the moving object in the morphed image.

6. The system of claim 1, wherein, for each image of the syringe of the plurality of images of the syringe, the fiducial marker in that image identifies the region of interest in that image as a predetermined area in that image including a predetermined width relative to the fiducial marker in that image, a predetermined length relative to the fiducial marker in that image, and a predetermined location relative to the fiducial marker in that image.

7. The system of claim 1, wherein the plurality of images of the syringe include at least one of the following: an infrared image, a microwave image, an ultrasound image, or any combination thereof.

8. A method, comprising: obtaining, with at least one processor, a plurality of images of a syringe including a fiducial marker, wherein the fiducial marker encapsulates syringe data associated with the syringe; determining, with the at least one processor, based on the fiducial marker in at least one image of the syringe of the plurality images of the syringe, an initial volume associated with the syringe;for each image of the syringe of the plurality of images of the syringe: determining, with the at least one processor, based on the fiducial marker in that image, a region of interest in that image; and subtracting, with the at least one processor, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image; determining, with the at least one processor, based on the subtracted image generated from a current image of the plurality of images, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and providing, with the at least one processor, the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

9. The method of claim 8, wherein the plurality of images includes color images, and wherein the method further comprises: for each image of the syringe of the plurality of images of the syringe: converting, with the at least one processor, the region of interest of that image to a grayscale image.

10. The method of claim 8, further comprising: for each image of the syringe of the plurality of images of the syringe: applying, with the at least one processor, at least one morphological operation to the subtracted image to generate a morphed image, wherein the morphed image includes an area of interest associated with a moving object and defined by at least one contour, and wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the morphed image.

11. The method of claim 10, further comprising: for each image of the syringe of the plurality of images of the syringe: determining, with the at least one processor, based on the morphed image, at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining inthe syringe, or any combination thereof is determined based on the at least one area of the moving object in the morphed image.

12. The method of claim 11, further comprising: for each image of the syringe of the plurality of images of the syringe: filtering, with the at least one processor, based on at least one area threshold, the at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the filtered at least one area of the moving object in the morphed image.

13. The method of claim 8, wherein, for each image of the syringe of the plurality of images of the syringe, the fiducial marker in that image identifies the region of interest in that image as a predetermined area in that image including a predetermined width relative to the fiducial marker in that image, a predetermined length relative to the fiducial marker in that image, and a predetermined location relative to the fiducial marker in that image.

14. The method of claim 8, wherein the plurality of images of the syringe include at least one of the following: an infrared image, a microwave image, an ultrasound image, or any combination thereof.

15. A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: obtain a plurality of images of a syringe including a fiducial marker, wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the plurality images of the syringe, an initial volume associated with the syringe; for each image of the syringe of the plurality of images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and subtract, from the region of interest in that image, the region of interest of a previous image of the plurality of images to generate a subtracted image;determine, based on the subtracted image generated from a current image of the plurality of images, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

16. The computer program product of claim 15, wherein the plurality of images includes color images, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the plurality of images of the syringe: convert the region of interest of that image to a grayscale image.

17. The computer program product of claim 15, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the plurality of images of the syringe: apply at least one morphological operation to the subtracted image to generate a morphed image, wherein the morphed image includes an area of interest associated with a moving object and defined by at least one contour, and wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the morphed image.

18. The computer program product of claim 17, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the plurality of images of the syringe: determine, based on the morphed image, at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the at least one area of the moving object in the morphed image.

19. The computer program product of claim 18, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to:for each image of the syringe of the plurality of images of the syringe: filter, based on at least one area threshold, the at least one area defined by the at least one contour in the morphed image, wherein the at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof is determined based on the filtered at least one area of the moving object in the morphed image.

20. The computer program product of claim 15, wherein, for each image of the syringe of the plurality of images of the syringe, the fiducial marker in that image identifies the region of interest in that image as a predetermined area in that image including a predetermined width relative to the fiducial marker in that image, a predetermined length relative to the fiducial marker in that image, and a predetermined location relative to the fiducial marker in that image.

Images