System, method, and computer program product for tracking real-time syringe volume

Abstract

Systems, methods, and computer program products for tracking real-time syringe volume may: acquire at least two images of a syringe including a fiducial marker, the at least two images including an initial image and a current image, the fiducial marker encapsulating syringe data; for each image of the syringe: determining a region of interest in the image based on a fiducial marker in the image, and segmenting the region of interest in the image based on a color threshold of the image to generate a mask image; determining a volume of fluid dispensed from the syringe and / or remaining in the syringe based on a pixel-to-volume map associated with the syringe, a mask image generated from the initial image, and a mask image generated from the current image; and provide a volume of fluid dispensed from and / or remaining in the syringe.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 509,607, filed June 22, 2023, entitled “System, Method, and Computer Program Product for Tracking Real-Time Syringe Volume,” the disclosure of which is incorporated herein by reference in its entirety. Background Technology

[0002] Incorrect dosage and incorrect administration rate are two common intravenous (IV) bolus administration errors that can have a significant impact on patient safety and recovery. Summary of the Invention

[0003] Therefore, improved systems, devices, products, apparatuses, and / or methods for tracking real-time syringe volumes are provided.

[0004] According to some non-limiting embodiments or aspects, a system is provided, comprising: at least one processor coupled to a memory and configured to: acquire at least two images of a syringe including reference markers, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the reference markers encapsulate syringe data associated with the syringe; determine a pixel-to-volume mapping associated with the syringe based on the reference markers in at least one of the at least two images of the syringe; for each of the at least two images of the syringe: determine a region of interest in the image based on the reference markers in the image, and segment the region of interest in the image based on a color threshold of the image to generate a mask image; determine at least one of the following based on the pixel-to-volume mapping associated with the syringe, the mask image generated from the initial image of the syringe, and the mask image generated from the current image of the syringe: a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof; and provide the at least one of the following: the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof.

[0005] In some non-limiting embodiments or aspects, the syringe includes at least one color calibration mark, which includes at least one color, wherein a movable part of the syringe includes the at least one color, and wherein at least one processor is further configured to: for each of at least two images of the syringe, generate a color threshold for the image based on the at least one color of the at least one color calibration mark in the image.

[0006] In some non-limiting embodiments or aspects, the at least one color calibration mark includes at least two color calibration marks, the at least two color calibration marks including a first color calibration mark and a second color calibration mark, the first color calibration mark including a first color, the second color calibration mark including a second color different from the first color, wherein a first portion of the movable part of the syringe includes the first color, wherein a second portion of the movable part of the syringe, different from the first portion, includes the second color, and wherein at least one processor is further configured to: for each image of the syringe in at least two images of the syringe: generate a color threshold for the image based on the first color of the first color calibration mark in the image and the second color of the second color calibration mark in the image.

[0007] In some non-limiting embodiments or aspects, at least two images of the syringe include a first color space, and wherein at least one processor is further configured to: for each of the at least two images of the syringe, convert the image from including the first color space to including a second color space different from the first color space, wherein a color threshold of the image is generated based on at least one color of at least one color calibration mark in the converted image, and wherein a region of interest in the converted image is segmented based on the color threshold of the image to generate a mask image.

[0008] In some non-limiting embodiments or aspects, at least one processor is further configured to determine at least one of the following by: fluid volume allocated from the syringe, fluid volume remaining in the syringe, or any combination thereof: for each of at least two images of the syringe, determining at least one centroid pixel of at least a group of pixels in a mask image that satisfies a color threshold of the image, wherein at least one of the fluid volume allocated from the syringe, fluid volume remaining in the syringe, or any combination thereof is determined based on a pixel-to-volume mapping associated with the syringe, at least one centroid pixel of a mask image generated from an initial image of the syringe, and at least one centroid pixel of a mask image generated from a current image of the syringe.

[0009] In some non-limiting embodiments or aspects, at least one processor is further configured to: for each of at least two images of the syringe, determine at least two centroid pixels in a mask image that satisfy at least two groups of pixels of a plurality of pixels, the at least two centroid pixels including a first centroid pixel of a first plurality of pixels satisfying a first color range threshold of the color threshold of the image and a second centroid pixel of a second plurality of pixels satisfying a second color range threshold of the image, wherein at least one of the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on a pixel-to-volume mapping associated with the syringe, a first centroid pixel of a mask image generated from an initial image of the syringe, a second centroid pixel of a mask image generated from an initial image of the syringe, a first centroid pixel of a mask image generated from a current image of the syringe, and a second centroid pixel of a mask image generated from the current image of the syringe.

[0010] In some non-limiting embodiments or aspects, pixel-to-volume mapping includes hard-coded or predetermined pixel-to-volume mapping.

[0011] In some non-limiting embodiments or aspects, the pixel-to-volume mapping includes a machine learning model configured to generate at least one of the following: a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof, based on a mask image generated from an initial image of the syringe and a mask image generated from a current image of the syringe.

[0012] In some non-limiting embodiments or aspects, at least one processor is further configured to: for each of at least two images of the syringe, control a display to display the image of the syringe on the display, the image having a mask image generated from the image overlaid thereon.

[0013] In some non-limiting embodiments or aspects, at least one processor is further configured to: for each of at least two images of the syringe, determine positional information associated with the three-dimensional (3D) position of the syringe relative to the image capture device that captured the image, based on reference markers in the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

[0014] According to some non-limiting embodiments or aspects, a method is provided, the method comprising: acquiring at least two images of a syringe including reference markers using at least one processor, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the reference markers encapsulate syringe data associated with the syringe; determining a pixel-to-volume mapping associated with the syringe based on the reference markers in at least one of the at least two images of the syringe using the at least one processor; for each image of the syringe in the at least two images of the syringe: determining a region of interest in the image based on the reference markers in the image using the at least one processor, and segmenting the region of interest in the image based on a color threshold of the image to generate a mask image using the at least one processor; determining at least one of the following based on the pixel-to-volume mapping associated with the syringe, the mask image generated from the initial image of the syringe, and the mask image generated from the current image of the syringe: a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof; and providing at least one of the following: a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof using the at least one processor.

[0015] In some non-limiting embodiments or aspects, the syringe includes at least one color calibration mark, the at least one color calibration mark including at least one color, wherein a movable part of the syringe includes the at least one color, and wherein the method further includes: for each of at least two images of the syringe: using at least one processor, generating a color threshold for the image based on the at least one color of the at least one color calibration mark in the image.

[0016] In some non-limiting embodiments or aspects, at least one color calibration mark includes at least two color calibration marks, the at least two color calibration marks including a first color calibration mark and a second color calibration mark, the first color calibration mark including a first color, the second color calibration mark including a second color different from the first color, wherein a first portion of the movable part of the syringe includes the first color, wherein a second portion of the movable part of the syringe different from the first portion includes the second color, and wherein the method further includes: for each image of the syringe in at least two images of the syringe: using at least one processor, generating a color threshold for the image based on the first color of the first color calibration mark in the image and the second color of the second color calibration mark in the image.

[0017] In some non-limiting embodiments or aspects, at least two images of the syringe include a first color space, and wherein the method further includes: for each of the at least two images of the syringe: using at least one processor, converting the image from including the first color space to including a second color space different from the first color space, wherein a color threshold of the image is generated based on at least one color of at least one color calibration mark in the converted image, and wherein a region of interest in the converted image is segmented based on the color threshold of the image to generate a mask image.

[0018] In some non-limiting embodiments or aspects, determining at least one of the following: the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, further includes: for each of at least two images of the syringe: using at least one processor, determining at least one centroid pixel of at least a group of pixels in a mask image that satisfies a color threshold of the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on a pixel-to-volume mapping associated with the syringe, at least one centroid pixel of a mask image generated from an initial image of the syringe, and at least one centroid pixel of a mask image generated from a current image of the syringe.

[0019] In some non-limiting embodiments or aspects, determining at least one of the following: the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, further includes: for each image of the syringe in at least two images of the syringe: using at least one processor, determining at least two centroid pixels in a mask image that satisfy at least two groups of pixels of a plurality of pixels, the at least two centroid pixels including a first centroid pixel of a first plurality of pixels satisfying a first color range threshold of the color threshold of the image and a second centroid pixel of a second plurality of pixels satisfying a second color range threshold of the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on a pixel-to-volume mapping associated with the syringe, a first centroid pixel of a mask image generated from an initial image of the syringe, a second centroid pixel of a mask image generated from the initial image of the syringe, a first centroid pixel of a mask image generated from a current image of the syringe, and a second centroid pixel of a mask image generated from the current image of the syringe.

[0020] In some non-limiting embodiments or aspects, pixel-to-volume mapping includes hard-coded or predetermined pixel-to-volume mapping.

[0021] In some non-limiting embodiments or aspects, the pixel-to-volume mapping includes a machine learning model configured to generate at least one of the following based on a mask image generated from an initial image of the syringe and a mask image generated from a current image of the syringe: the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof.

[0022] In some non-limiting embodiments or aspects, the method further includes: for each of at least two images of the syringe: using at least one processor, controlling a display to display the image of the syringe on the display, the image having a mask image generated from the image overlaid thereon.

[0023] In some non-limiting embodiments or aspects, the method further includes: for each of at least two images of the syringe: using at least one processor, determining positional information associated with the three-dimensional (3D) position of the syringe relative to the image capture device capturing the image, based on reference markers in the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

[0024] According to some non-limiting embodiments or aspects, a computer program product is provided, comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: acquire at least two images of a syringe including reference markers, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the reference markers encapsulate syringe data associated with the syringe; determine a pixel-to-volume mapping associated with the syringe based on the reference markers in at least one of the at least two images of the syringe; for each of the at least two images of the syringe: determine a region of interest in the image based on the reference markers in the image, and segment the region of interest in the image to generate a mask image based on a color threshold of the image; determine at least one of the following based on the pixel-to-volume mapping associated with the syringe, the mask image generated from the initial image of the syringe, and the mask image generated from the current image of the syringe: a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof; and provide at least one of the following: the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof.

[0025] In some non-limiting embodiments or aspects, the syringe includes at least one color calibration mark, which includes at least one color, wherein a movable part of the syringe includes the at least one color, and wherein the program instructions, when executed by at least one processor, further cause the at least one processor to: for each of at least two images of the syringe: generate a color threshold for the image based on the at least one color of the at least one color calibration mark in the image.

[0026] In some non-limiting embodiments or aspects, at least one color calibration mark includes at least two color calibration marks, the at least two color calibration marks including a first color calibration mark and a second color calibration mark, the first color calibration mark including a first color, the second color calibration mark including a second color different from the first color, wherein a first portion of the movable part of the syringe includes the first color, wherein a second portion of the movable part of the syringe, different from the first portion, includes the second color, and wherein the program instructions, when executed by at least one processor, further cause the at least one processor to: for each of at least two images of the syringe: generate a color threshold for the image based on the first color of the first color calibration mark in the image and the second color of the second color calibration mark in the image.

[0027] In some non-limiting embodiments or aspects, at least two images of the syringe include a first color space, and wherein, when executed by at least one processor, the program instructions further cause the at least one processor to: for each of the at least two images of the syringe: convert the image from including the first color space to including a second color space different from the first color space, wherein a color threshold of the image is generated based on at least one color of at least one color calibration mark in the converted image, and wherein a region of interest in the converted image is segmented based on the color threshold of the image to generate a mask image.

[0028] In some non-limiting embodiments or aspects, when executed by at least one processor, the program instructions further cause the at least one processor to determine at least one of the following: for each of at least two images of the syringe: determining at least one centroid pixel of at least a group of pixels in a mask image that satisfies a color threshold of the image, wherein at least one of the following is determined based on a pixel-to-volume mapping associated with the syringe, at least one centroid pixel of a mask image generated from an initial image of the syringe, and at least one centroid pixel of a mask image generated from a current image of the syringe.

[0029] In some non-limiting embodiments or aspects, the program instructions, when executed by at least one processor, further cause the at least one processor to: for each of at least two images of the syringe, determine at least two centroid pixels in a mask image that satisfy at least two groups of pixels of a plurality of pixels, the at least two centroid pixels including a first centroid pixel of a first plurality of pixels satisfying a first color range threshold of the color threshold of the image and a second centroid pixel of a second plurality of pixels satisfying a second color range threshold of the image, wherein at least one of the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on a pixel-to-volume mapping associated with the syringe, the first centroid pixel of a mask image generated from an initial image of the syringe, the second centroid pixel of a mask image generated from an initial image of the syringe, the first centroid pixel of a mask image generated from a current image of the syringe, and the second centroid pixel of a mask image generated from the current image of the syringe.

[0030] In some non-limiting embodiments or aspects, pixel-to-volume mapping includes hard-coded or predetermined pixel-to-volume mapping.

[0031] In some non-limiting embodiments or aspects, the pixel-to-volume mapping includes a machine learning model configured to generate at least one of the following: a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof, based on a mask image generated from an initial image of the syringe and a mask image generated from a current image of the syringe.

[0032] In some non-limiting embodiments or aspects, the program instructions, when executed by at least one processor, also cause the at least one processor to: for each of at least two images of the syringe, control a display to display the image of the syringe on the display, the image having a mask image generated from the image overlaid thereon.

[0033] In some non-limiting embodiments or aspects, the program instructions, when executed by at least one processor, further cause the at least one processor to: for each of at least two images of the syringe, determine positional information associated with the three-dimensional (3D) position of the syringe relative to the image capture device of the captured image, based on reference markers in the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

[0034] Further non-limiting embodiments or aspects are described in the following numbered entries: Article 1: A system comprising: at least one processor coupled to a memory and configured to: acquire at least two images of a syringe including reference markers, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the reference markers encapsulate syringe data associated with the syringe; determine a pixel-to-volume mapping associated with the syringe based on the reference markers in at least one image of the syringe among the at least two images of the syringe; for each image of the syringe among the at least two images of the syringe: determine a region of interest in the image based on the reference markers in the image, and segment the region of interest in the image based on a color threshold of the image to generate a mask image; determine at least one of a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof, based on the pixel-to-volume mapping associated with the syringe, the mask image generated from the initial image of the syringe, and the mask image generated from the current image of the syringe; and provide at least one of the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof.

[0035] Article 2: The system according to Article 1, wherein the syringe includes at least one color calibration mark, the at least one color calibration mark includes at least one color, wherein a movable part of the syringe includes the at least one color, and wherein at least one processor is further configured to: for each of at least two images of the syringe: generate a color threshold for the image based on at least one color of the at least one color calibration mark in the image.

[0036] Article 3: A system according to either Article 1 or Article 2, wherein at least one color calibration mark comprises at least two color calibration marks, the at least two color calibration marks comprising a first color calibration mark and a second color calibration mark, the first color calibration mark comprising a first color, the second color calibration mark comprising a second color different from the first color, wherein a first portion of a movable part of the syringe comprises the first color, wherein a second portion of the movable part of the syringe, different from the first portion, comprises the second color, and wherein at least one processor is further configured to: for each image of the syringe in at least two images of the syringe: generate a color threshold for the image based on the first color of the first color calibration mark in the image and the second color of the second color calibration mark in the image.

[0037] Article 4: A system according to any one of Articles 1 to 3, wherein at least two images of the syringe include a first color space, and wherein at least one processor is further configured to: for each of the at least two images of the syringe: convert the image from including the first color space to including a second color space different from the first color space, wherein a color threshold of the image is generated based on at least one color of at least one color calibration mark in the converted image, and wherein a region of interest in the converted image is segmented based on the color threshold of the image to generate a mask image.

[0038] Article 5: A system according to any one of Articles 1 to 4, wherein at least one processor is further configured to determine at least one of the following: a fluid volume dispensed from the syringe, a remaining fluid volume in the syringe, or any combination thereof, by: for each of at least two images of the syringe, determining at least one centroid pixel of at least a group of pixels in a mask image that satisfies a color threshold of the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on a pixel-to-volume mapping associated with the syringe, at least one centroid pixel of a mask image generated from an initial image of the syringe, and at least one centroid pixel of a mask image generated from a current image of the syringe.

[0039] Article 6: A system according to any one of Articles 1 to 5, wherein at least one processor is further configured to: for each of at least two images of the syringe, determine at least two centroid pixels in a mask image that satisfy a color threshold of the image, the at least two centroid pixels including a first centroid pixel of a first plurality of pixels satisfying a first color range threshold of the color threshold of the image and a second centroid pixel of a second plurality of pixels satisfying a second color range threshold of the image, wherein at least one of the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on a pixel-to-volume mapping associated with the syringe, a first centroid pixel of a mask image generated from an initial image of the syringe, a second centroid pixel of a mask image generated from an initial image of the syringe, a first centroid pixel of a mask image generated from a current image of the syringe, and a second centroid pixel of a mask image generated from the current image of the syringe.

[0040] Article 7: A system according to any one of Articles 1 to 6, wherein the pixel-to-volume mapping includes hard-coded or predetermined pixel-to-volume mapping.

[0041] Article 8: A system according to any one of Articles 1 to 7, wherein the pixel-to-volume mapping includes a machine learning model configured to generate at least one of the following: a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof, based on a mask image generated from an initial image of the syringe and a mask image generated from a current image of the syringe.

[0042] Article 9: A system according to any one of Articles 1 to 8, wherein at least one processor is further configured to: for each of at least two images of the syringe: control a display to display the image of the syringe on the display, the image having a mask image generated from the image overlaid thereon.

[0043] Article 10: A system according to any one of Articles 1 to 9, wherein at least one processor is further configured to: for each of at least two images of the syringe, determine positional information associated with the three-dimensional (3D) position of the syringe relative to the image capture device of the captured image, based on a reference marker in the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

[0044] Article 11: A method comprising: acquiring at least two images of a syringe including reference markers using at least one processor, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the reference markers encapsulate syringe data associated with the syringe; determining a pixel-to-volume mapping associated with the syringe based on the reference markers in at least one image of the syringe among the at least two images of the syringe using the at least one processor; for each image of the syringe among the at least two images of the syringe: determining a region of interest in the image based on the reference markers in the image using the at least one processor, and segmenting the region of interest in the image based on a color threshold of the image to generate a mask image using the at least one processor; determining at least one of a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof, based on the pixel-to-volume mapping associated with the syringe, the mask image generated from the initial image of the syringe, and the mask image generated from the current image of the syringe using the at least one processor; and providing at least one of a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof, using the at least one processor.

[0045] Article 12: The method according to Article 11, wherein the syringe includes at least one color calibration mark, the at least one color calibration mark including at least one color, wherein a movable part of the syringe includes the at least one color, and wherein the method further includes: for each of at least two images of the syringe: using at least one processor, generating a color threshold for the image based on the at least one color of the at least one color calibration mark in the image.

[0046] Article 13: The method according to any one of Articles 11 or 12, wherein at least one color calibration mark comprises at least two color calibration marks, the at least two color calibration marks comprising a first color calibration mark and a second color calibration mark, the first color calibration mark comprising a first color, the second color calibration mark comprising a second color different from the first color, wherein a first portion of a movable part of the syringe comprises the first color, wherein a second portion of the movable part of the syringe, different from the first portion, comprises the second color, and wherein the method further comprises: for each image of the syringe in at least two images of the syringe: using at least one processor, generating a color threshold for the image based on the first color of the first color calibration mark in the image and the second color of the second color calibration mark in the image.

[0047] Article 14: The method according to any one of Articles 11 to 13, wherein at least two images of the syringe include a first color space, and wherein the method further comprises: for each of the at least two images of the syringe: using at least one processor, converting the image from including the first color space to including a second color space different from the first color space, wherein a color threshold of the image is generated based on at least one color of at least one color calibration mark in the converted image, and wherein a region of interest in the converted image is segmented based on the color threshold of the image to generate a mask image.

[0048] Article 15: The method according to any one of Articles 11 to 14, wherein determining at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, further comprises: for each of at least two images of the syringe: using at least one processor, determining at least one centroid pixel of at least a group of pixels in a mask image that satisfies a color threshold of the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on a pixel-to-volume mapping associated with the syringe, at least one centroid pixel of a mask image generated from an initial image of the syringe, and at least one centroid pixel of a mask image generated from a current image of the syringe.

[0049] Article 16: The method according to any one of Articles 11 to 15, wherein determining at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, further comprises: for each image of the syringe in at least two images of the syringe: using at least one processor, determining at least two centroid pixels of at least two groups of pixels in a mask image that satisfy a color threshold of the image, the at least two centroid pixels including a first centroid pixel of a first plurality of pixels satisfying a first color range threshold of the color threshold of the image and a second centroid pixel of a second plurality of pixels satisfying a second color range threshold of the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on a pixel-to-volume mapping associated with the syringe, a first centroid pixel of a mask image generated from an initial image of the syringe, a second centroid pixel of a mask image generated from the initial image of the syringe, a first centroid pixel of a mask image generated from a current image of the syringe, and a second centroid pixel of a mask image generated from the current image of the syringe.

[0050] Article 17: The method according to any one of Articles 11 to 16, wherein the pixel-to-volume mapping includes hard-coded or predetermined pixel-to-volume mapping.

[0051] Article 18: The method according to any one of Articles 11 to 17, wherein the pixel-to-volume mapping includes a machine learning model configured to: generate at least one of the following: a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof, based on a mask image generated from an initial image of the syringe and a mask image generated from a current image of the syringe.

[0052] Article 19: The method according to any one of Articles 11 to 18 further comprises: for each of at least two images of the syringe: using at least one processor, controlling a display to display the image of the syringe on the display, the image having a mask image generated from the image overlaid thereon.

[0053] Article 20: The method according to any one of Articles 11 to 19 further comprises: for each of at least two images of the syringe: using at least one processor, determining positional information associated with the three-dimensional (3D) position of the syringe relative to the image capture device of the captured image based on reference markers in the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

[0054] Article 21: A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: acquire at least two images of a syringe including reference markers, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the reference markers encapsulate syringe data associated with the syringe; determine a pixel-to-volume mapping associated with the syringe based on the reference markers in at least one of the at least two images of the syringe; for each of the at least two images of the syringe: determine a region of interest in the image based on the reference markers in the image, and segment the region of interest in the image to generate a mask image based on a color threshold of the image; determine at least one of a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof, based on the pixel-to-volume mapping associated with the syringe, the mask image generated from the initial image of the syringe, and the mask image generated from the current image of the syringe; and provide the at least one of the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof.

[0055] Article 22: A computer program product according to Article 21, wherein the syringe includes at least one color calibration mark, the at least one color calibration mark including at least one color, wherein a movable part of the syringe includes the at least one color, and wherein the program instructions, when executed by at least one processor, further cause the at least one processor to: for each of at least two images of the syringe: generate a color threshold for the image based on the at least one color of the at least one color calibration mark in the image.

[0056] Article 23: A computer program product according to any one of Articles 21 or 22, wherein at least one color calibration mark comprises at least two color calibration marks, the at least two color calibration marks comprising a first color calibration mark and a second color calibration mark, the first color calibration mark comprising a first color, the second color calibration mark comprising a second color different from the first color, wherein a first portion of a movable part of the syringe comprises the first color, wherein a second portion of the movable part of the syringe, different from the first portion, comprises the second color, and wherein the program instructions, when executed by at least one processor, further cause the at least one processor to: for each of at least two images of the syringe: generate a color threshold for the image based on the first color of the first color calibration mark in the image and the second color of the second color calibration mark in the image.

[0057] Article 24: A computer program product according to any one of Articles 21 to 23, wherein at least two images of a syringe include a first color space, and wherein, when executed by at least one processor, the program instructions further cause the at least one processor to: for each of the at least two images of the syringe: convert the image from including the first color space to including a second color space different from the first color space, wherein a color threshold of the image is generated based on at least one color of at least one color calibration mark in the converted image, and wherein a region of interest in the converted image is segmented based on the color threshold of the image to generate a mask image.

[0058] Article 25: A computer program product according to any one of Articles 21 to 24, wherein the program instructions, when executed by at least one processor, further cause the at least one processor to determine at least one of the following: for each of at least two images of the syringe: determining at least one centroid pixel of at least one set of a plurality of pixels in a mask image that satisfies a color threshold of the image, wherein at least one of the following is determined based on a pixel-to-volume mapping associated with the syringe, at least one centroid pixel of a mask image generated from an initial image of the syringe, and at least one centroid pixel of a mask image generated from a current image of the syringe.

[0059] Article 26: A computer program product according to any one of Articles 21 to 25, wherein the program instructions, when executed by at least one processor, further cause the at least one processor to: for each of at least two images of the syringe, determine at least two centroid pixels in a mask image that satisfy at least two groups of pixels of a plurality of pixels, the at least two centroid pixels including a first centroid pixel of a first plurality of pixels satisfying a first color range threshold of the color threshold of the image and a second centroid pixel of a second plurality of pixels satisfying a second color range threshold of the image, wherein at least one of the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on a pixel-to-volume mapping associated with the syringe, a first centroid pixel of a mask image generated from an initial image of the syringe, a second centroid pixel of a mask image generated from an initial image of the syringe, a first centroid pixel of a mask image generated from a current image of the syringe, and a second centroid pixel of a mask image generated from the current image of the syringe.

[0060] Article 27: A computer program product according to any one of Articles 21 to 26, wherein the pixel-to-volume mapping includes hard-coded or predetermined pixel-to-volume mapping.

[0061] Article 28: A computer program product according to any one of Articles 21 to 27, wherein the pixel-to-volume mapping includes a machine learning model configured to generate at least one of the following: a fluid volume allocated from the syringe, a remaining fluid volume in the syringe, or any combination thereof, based on a mask image generated from an initial image of the syringe and a mask image generated from a current image of the syringe.

[0062] Article 29: A computer program product according to any one of Articles 21 to 28, wherein the program instructions, when executed by at least one processor, further cause the at least one processor to: control a display to display the image of the syringe on a display for each of at least two images of the syringe, the image having a mask image generated from the image overlaid thereon.

[0063] Article 30: A computer program product according to any one of Articles 21 to 29, wherein the program instructions, when executed by at least one processor, further cause the at least one processor to: for each of at least two images of the syringe: determine positional information associated with the three-dimensional (3D) position of the syringe relative to the image capture device of the captured image, based on a reference marker in the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image. Attached Figure Description

[0064] With reference to the exemplary embodiments shown in the accompanying drawings, additional advantages and details are explained in more detail below: Figure 1A These are schematic diagrams of non-limiting embodiments or aspects of the environments in which the systems, devices, products, apparatuses and / or methods described herein may be implemented; Figure 1B and Figure 1C This is a perspective view of a non-limiting embodiment or aspect of the syringe; Figure 2 yes Figure 1A A schematic diagram of a non-limiting embodiment or aspect of a component of one or more devices and / or one or more systems; Figure 3 This is a flowchart of a non-limiting embodiment or aspect of a process for tracking real-time syringe volume; and Figures 4 to 6 The embodiments or aspects of the syringe covered with an image of a corresponding mask image are shown. Detailed Implementation

[0065] It should be understood that, unless expressly stated to the contrary, this disclosure may present various alternative variations and sequences of steps. It should also be understood that the specific devices and processes shown in the accompanying drawings and described in the following specification are merely exemplary and non-limiting embodiments or aspects. Therefore, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein should not be considered limiting.

[0066] In the following text, for descriptive purposes, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and their derivatives shall be used in connection with embodiments or aspects oriented as shown in the accompanying drawings. However, it should be understood that various alternative variations and sequences of steps may be employed in embodiments or aspects unless explicitly stated otherwise. It should also be understood that the specific devices and processes shown in the accompanying drawings and described in the following specification are merely non-limiting exemplary embodiments or aspects. Therefore, unless otherwise indicated, specific dimensions and other physical characteristics relating to the embodiments disclosed herein, or aspects thereof, or aspects thereof, should not be considered limiting.

[0067] Unless explicitly stated otherwise, aspects, parts, elements, structures, actions, steps, functions, and / or instructions used herein should not be construed as critical or essential. Furthermore, as used herein, the article “a (a, an)” is intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” Additionally, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with “one or more” or “at least one.” Where only one item is referred to, the term “one” or similar language is used. Furthermore, as used herein, the terms “having,” “possessing,” or “with” are intended as open-ended terms. Additionally, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.”

[0068] As used herein, the term "communication" can refer to the reception, receipt, transmission, transfer, and / or provision of information (e.g., data, signals, messages, instructions, and / or commands). Communication between one unit (e.g., a device, system, component of a device or system, and / or a combination thereof) and another unit means that one unit is able to receive information from and / or send information to the other unit, directly or indirectly. This can refer to a direct or indirect connection that is inherently wired and / or wireless. Furthermore, two units can communicate with each other even if the transmitted information can be modified, processed, relayed, and / or routed between the first and second units. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively send information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediate unit (e.g., a third unit located between the first and second units) processes information received from the first unit and sends the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network data packet (e.g., a data packet and / or the like) that includes data. It will be appreciated that many other arrangements are also possible.

[0069] As used herein, the term "computing device" can refer to one or more electronic devices configured to communicate directly or indirectly with or through one or more networks. A computing device can be a mobile or portable computing device, a desktop computer, and / or a server, etc. Furthermore, the term "computer" can refer to any computing device that includes the necessary components for receiving, processing, and outputting data, and typically includes a display, processor, memory, input devices, and network interfaces. A "computing system" can include one or more computing devices or computers. "Application" or "application program interface" (API) refers to computer code or other data stored on a computer-readable medium that can be executed by a processor to facilitate interaction between various software components, such as a client-side front-end and / or a server-side back-end for receiving data from a client. "Interface" refers to generated display content, such as one or more graphical user interfaces (GUIs) that a user can interact with directly or indirectly (e.g., via a keyboard, mouse, touchscreen, etc.). Furthermore, multiple computers (e.g., servers or other computerized devices) communicating directly or indirectly in a networked environment can constitute a "system" or "computing system."

[0070] It will be apparent that the systems and / or methods described herein can be implemented in various forms, including hardware, software, or a combination of both. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited to these implementations. Therefore, while the operation and behavior of the systems and / or methods are described herein without mentioning specific software code, it should be understood that software and hardware can be designed to implement the systems and / or methods based on the descriptions herein.

[0071] This document describes some non-limiting embodiments or aspects in conjunction with thresholds. As used herein, satisfying a threshold can mean: a value greater than a threshold, exceeding a threshold, higher than a threshold, greater than or equal to a threshold, less than a threshold, less than a threshold, lower than a threshold, less than or equal to a threshold, equal to a threshold, etc.

[0072] For reference Figure 1A , Figure 1A This is a schematic diagram of an example environment 100 in which the devices, systems, methods, apparatuses, and / or products described herein may be implemented. Figure 1A As shown, environment 100 includes user equipment 102, syringe 103, management system 104, and / or communication network 106. The systems and / or devices of environment 100 can be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0073] User equipment 102 may include one or more devices capable of receiving information and / or data from management system 104 (e.g., via communication network 106, etc.) and / or transmitting information and / or data to management system 104 (e.g., via communication network 106, etc.). For example, user equipment 102 may include one or more computing systems (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, one or more tablet computers, etc.) including one or more processors. In some non-limiting embodiments or aspects, user equipment 102 may include tablet computers or mobile computing devices, such as Apple® iPads, Apple iPhones, Android® tablets and / or Android phones, etc.

[0074] User equipment 102 may include one or more image capture devices (e.g., one or more cameras, one or more sensors, etc.) configured to capture one or more images of the environment surrounding the one or more image capture devices (e.g., environment 100, etc.). For example, user equipment 102 may include one or more image capture devices configured to capture one or more images of syringe 103. As an example, the image capture device of user equipment 102 may include at least one of the following: multiple image capture devices; a monocular camera; a stereo camera; a color camera configured to capture and / or detect light of one or more predetermined wavelengths; a camera including a filter configured to filter light of predetermined wavelengths; an infrared (IR) camera; a thermal sensor configured to capture a thermal image of a syringe 103 irradiated by an infrared wavelength of a light source; a pan, tilt, and zoom (PTZ) camera including a field-of-view (FOV) and automatic zoom functions; a master-slave camera system including a still camera and a motion camera; a LiDAR system and / or a RADAR system; or any combination thereof.

[0075] In some non-limiting embodiments or aspects, the image capture device of user equipment 102 includes a stereo camera. For example, a Structure from Motion (SfM) algorithm can be used to determine 3D positional information associated with the 3D position of syringe 103 and / or its components (e.g., plunger 134, stopper 136, reference marker 138, etc.) relative to the image capture device (and / or user equipment 102). As an example, user equipment 102 may include a stereo camera setup that is available in many mobile devices, such as Apple tablets, iPhones, Android tablets, and / or Android phones. User equipment 102 can use the SfM algorithm to process images from the stereo camera to extract 3D information that enhances the identification of object features of reference marker 138 and / or components of syringe 103. As an example, SfM processing can improve 3D feature extraction from the components of syringe 103, which can enhance the opportunity for image feature accumulation, for example, through a burst image capture / video mode that captures images in the appropriate orientation / registration (e.g., translation / tilt, etc.) based on the setup or position of syringe 103. In such an example, user device 102 can use a stereo camera to generate a 3D depth or point cloud of syringe 103 and detect and classify syringe components based on the depth information of the 3D point cloud and the structural feature mapping of syringe components. For example, user device 102 can use the depth information to detect and track the distance between plunger 134 and the image capture device (and / or user device 102), and / or map the real-time volume of the drug within the syringe, which can improve the accuracy of tracking plunger depth and movement, thereby providing a better pixel-to-volume mapping for tracking the volume of the drug within the syringe.

[0076] In some non-limiting embodiments or aspects, the image acquisition device of user equipment 102 includes a LiDAR system and / or a radar system. For example, the image may include a LiDAR point cloud and / or a radar point cloud. As an example, user equipment 102 may include a mini LiDAR system that can be used in many mobile devices, such as Apple tablets, iPhones, Android tablets, and / or Android phones. In such an example, using LiDAR images can improve the accuracy of 3D feature detection because LiDAR images directly provide 3D world information as point clouds, which can accelerate 3D data collection using reduced or minimal protocols compared to stereo setups. For example, user equipment 102 can use existing image registration and / or transformation techniques to overlay 2D object information (e.g., color and / or texture, etc.) from camera images with 3D LiDAR point clouds to detect 3D features, thereby detecting and tracking the distance between the syringe component and the image capture device (and / or user equipment 102) for mapping the real-time volume of the drug within syringe 103. This can improve the accuracy of tracking plunger depth and motion to provide better pixel-to-volume mapping for tracking the volume of the drug within the syringe.

[0077] Also refer to Figure 1B and Figure 1C , Figure 1B and Figure 1C This is a perspective view of a non-limiting embodiment or aspect of a syringe 103, which may include a syringe barrel 132, a plunger rod 134, and / or a stopper 136. The syringe barrel 132 may extend between a proximal and a distal end, the proximal end including a proximal opening configured to receive the stopper 136 and / or the plunger rod 134, and the distal end including a distal opening (e.g., a Luer lock opening, etc.) configured (e.g., via a needleless connector, etc.) to connect to an IV line or catheter and / or a needle or cannula, and fluid (e.g., a drug, etc.) may be discharged from the syringe 103 through the distal opening.

[0078] Syringe 103 may include a reference mark 138 (e.g., a label, mark, code, etc.). The reference mark 138 may be associated with syringe 103 (e.g., removably attached to syringe 103, permanently attached to syringe 103, integrated into syringe 103, implemented on syringe 103, etc.). For example, as Figure 1B and Figure 1CAs shown, the reference mark 138 may be located on the outer surface of the syringe barrel 132 of the syringe 103, such that the reference mark 138 is included in the image of the syringe 103 captured by the image capturing device. In some non-limiting embodiments or aspects, the syringe 103 may include a plurality of reference marks 138 arranged around the axis of the syringe 103, which may cause at least one reference mark 138 to be presented to the FOV of the image capturing device, regardless of the orientation of the syringe 103.

[0079] Reference mark 138 may encapsulate syringe data associated with syringe 103. For example, syringe data may include at least one of the following parameters associated with syringe 103: the volume of syringe 103 (e.g., the initial volume of fluid contained in syringe 103); the size of syringe 103; the type of syringe 103 (e.g., syringe type identifier); a unique identifier of syringe 103 (e.g., uniquely distinguishing syringe 103 from other syringes); the type of drug contained in syringe 103; the concentration of drug contained in syringe 103; and / or the specified delivery rate of drug contained in syringe 103; or any combination thereof.

[0080] The reference marker 138 may encapsulate pose information associated with the 3D position of the reference marker 138. For example, the reference marker 138 may include a marker that, when captured in an image, enables the calculation of the precise 3D position of the reference marker 138 relative to the image capture device that captured the image (e.g., the x, y, z coordinates of the reference marker 138, etc.) and / or the precise two-dimensional (2D) position of the reference marker 138 in the image itself (e.g., the x, y coordinates of the reference marker 138, etc. in the image).

[0081] In some non-limiting embodiments or aspects, the reference marker 138 includes an AprilTag. For example, the reference marker 138 may include an AprilTag V3 of type Custom Tag 48h12, which enables the use of AprilTag V3 detection to determine a unique ID that may (e.g., in leading numbers, etc.) indicate the type of syringe 103 associated with the reference marker 138, and / or (e.g., in trailing numbers, etc.) indicate a unique serial number of a particular syringe 103, and / or indicate the position of the reference marker 138 in the field of view (FOV) of the image capture device (e.g., x, y, and z coordinates, direction vectors of the Z, Y, and X axes, etc.). However, non-limiting embodiments or aspects are not limited thereto. The reference mark 138 may include a quick response (QR) code, a barcode (e.g., a one-dimensional (1D) barcode, a 2D barcode, etc.), an Aztec code, a data matrix code, an ArUco mark, a colored pattern, a reflective pattern, a fluorescent pattern, a predetermined shape and / or color (e.g., a red pentagon, a blue hexagon, etc.), a light-emitting diode (LED) pattern, and / or a hologram, etc. The above items encapsulate syringe data associated with the syringe 103 and / or encapsulate pose information associated with the 2D position and / or 3D position of the reference mark 138.

[0082] For reference Figure 1B The syringe 103 may include a color calibration mark 139 (e.g., a label, marker, etc.). The color calibration mark 139 may be associated with the syringe 103 (e.g., removably attached to the syringe 103, permanently attached to the syringe 103, integrated into the syringe 103, implemented on the syringe 103, etc.). For example, as... Figure 1B and Figure 1C As shown, the color calibration mark 139 may be located on the outer surface of the syringe barrel 132 of the syringe 103, such that the color calibration mark 139 is included in the image of the syringe 103 captured by the image capture device. In some non-limiting embodiments or aspects, the syringe 103 may include a plurality of color calibration marks 139 arranged around the axis of the syringe 103, which may cause at least one color calibration mark 139 to be presented to the FOV of the image capture device, regardless of the orientation of the syringe 103.

[0083] Color calibration mark 139 may include a color corresponding to the color of a movable part of syringe 103 (e.g., plunger 134, stopper 136, etc.). For example, color calibration mark 139 and stopper 136 (e.g., a portion of stopper 136, a color-marked ring on stopper 136, stopper 136 as a whole, etc.) may include the same color (e.g., blue, red, green, fluorescent, a color or surface that absorbs or reflects light in the near-infrared band, etc.), which can be used to calibrate the color used to detect stopper 136 under varying or a wider range of illumination conditions. Color calibration mark 139 may be positioned adjacent to reference mark 138. For example, color calibration mark 139 may be located on syringe 103 in association with reference mark 138, such that color calibration mark 139 is included in the region of interest defined or identified by reference mark 138 in an image of syringe 103 including reference mark 138. As an example, the reference mark 138 can define or identify a region of interest in an image, which has a predetermined width, a predetermined length, and / or a predetermined position relative to the reference mark 138 in the image.

[0084] For reference Figure 1C The color calibration mark 139 may include at least two color calibration marks, including a first color calibration mark 139a and a second color calibration mark 139b. The first color calibration mark 139a includes a first color, and the second color calibration mark 139b includes a second color different from the first color. A first portion 136a of the stopper 136 (and / or plunger 134) of the syringe 103 may include the first color, and a second portion 136b of the stopper 136 (and / or plunger 134) of the syringe 103, different from the first portion 136a, may include the second color. For example, using two or more color calibration marks corresponding to two or more identical or similar colored portions of the stopper 136 (and / or plunger 134) can improve noise retention when processing images of the stopper 136 (and / or plunger 134) compared to using a single color calibration mark.

[0085] In some non-limiting embodiments or aspects, the syringe 103 may be held in the same position and / or orientation relative to the image capture device of the user equipment 102 to provide a consistent region of interest and a fixed field of view. For example, when the image capture device of the user equipment 102 captures a series of images of the syringe 103 over a period of time to track the real-time volume of fluid dispensed from and / or remaining in the syringe 103, the syringe 103 may be held in the same position and / or orientation relative to the image capture device (e.g., by hardware configuration or fixing device, etc.). However, the non-limiting embodiments or aspects of this disclosure are not limited thereto, and in some non-limiting embodiments or aspects, the position and / or orientation of the syringe 103 relative to the image capture device may vary (e.g., due to being held in the user's hand, etc.) when the image capture device of the user equipment 102 captures a series of images of the syringe 103 over a period of time to track the real-time volume of fluid dispensed from and / or remaining in the syringe 103.

[0086] The management system 104 may include one or more devices capable of receiving information and / or data from user equipment 102 (e.g., via communication network 106, etc.) and / or transmitting information and / or data to user equipment 102 (e.g., via communication network 106, etc.). For example, the management system 104 may include one or more computing systems (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, etc.) containing one or more processors. In some non-limiting embodiments or aspects, the management system 104 includes and / or can be accessed through a nursing station or terminal in a hospital. For example, the management system 104 may provide bedside nurse support, nursing station administrator support, and / or retrospective reports on nursing management, etc.

[0087] The communication network 106 may include one or more wired and / or wireless networks. For example, the communication network 106 may include cellular networks (e.g., long-term evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, sixth-generation (6G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMN), local area networks (LAN), wide area networks (WAN), metropolitan area networks (MAN), telephone networks (e.g., public switched telephone networks (PSTN)), private networks, ad hoc networks, intranets, the Internet, fiber-optic networks and / or cloud computing networks, and / or combinations of these networks, or other types of networks.

[0088] Figures 1A to 1C The number and arrangement of systems and equipment shown are provided as examples. Figures 1A to 1C Compared to the systems and / or devices shown, there may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, or systems and / or devices arranged differently. Furthermore, Figures 1A to 1C The two or more systems or devices shown can be implemented in a single system or a single device, or Figures 1A to 1C The single system or single device shown can be implemented as multiple distributed systems or devices. Additionally or alternatively, a group of systems or a group of devices in environment 100 (e.g., one or more systems, one or more devices, etc.) can perform one or more functions described as being performed by another group of systems or another group of devices in environment 100.

[0089] For reference Figure 2 , Figure 2This is a schematic diagram of example components of device 200. Device 200 may correspond to user equipment 102 (e.g., one or more devices of the system of user equipment 102, etc.) and / or one or more devices of management system 104. In some non-limiting embodiments or aspects, user equipment 102 (e.g., one or more devices of the system of user equipment 102, etc.) and / or one or more devices of management system 104 may include at least one device 200 and / or at least one component of device 200. Figure 2 As shown, device 200 may include bus 202, processor 204, memory 206, storage unit 208, input unit 210, output unit 212 and communication interface 214.

[0090] Bus 202 may include components that allow communication between the various components of device 200. In some non-limiting embodiments or aspects, processor 204 may be implemented in hardware, software, or a combination of hardware and software. For example, processor 204 may include components that can be programmed to perform functions such as: processors (e.g., central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), etc.), microprocessors, digital signal processors (DSPs), and / or any processing unit (e.g., field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), etc.). Memory 206 may include random access memory (RAM), read-only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic storage, optical storage, etc.) storing information and / or instructions for use by processor 204.

[0091] Storage component 208 may store information and / or software related to the operation and use of device 200. For example, storage component 208 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, solid-state disks, etc.), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cartridges, magnetic tapes, and / or other types of computer-readable media and corresponding drives.

[0092] Input component 210 may include components that allow device 200 to receive information, such as through user input (e.g., touchscreen display, keyboard, keypad, mouse, button, switch, microphone, etc.). Additionally or alternatively, input component 210 may include sensors for sensing information (e.g., global positioning system (GPS) components, accelerometers, gyroscopes, actuators, image capture devices, etc.). Output component 212 may include components that provide output information from device 200 (e.g., display, speaker, one or more light-emitting diodes (LEDs), etc.).

[0093] Communication interface 214 may include transceiver-like components (e.g., transceiver, separate receiver and transmitter, etc.) that enable device 200 to communicate with other devices, for example, via wired connection, wireless connection, or a combination of wired and wireless connection. Communication interface 214 may allow device 200 to receive information from another device and / or provide information to another device. For example, communication interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, and / or a cellular network interface, etc.

[0094] Device 200 can perform one or more of the processes described herein. Device 200 can perform these processes based on processor 204 executing software instructions stored in a computer-readable medium (e.g., memory 206 and / or storage unit 208). A computer-readable medium (e.g., a non-transitory computer-readable medium) is defined herein as a non-transitory storage device. A storage device includes storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0095] Software instructions can be read into memory 206 and / or storage unit 208 from another computer-readable medium or from another device via communication interface 214. When executed, the software instructions stored in memory 206 and / or storage unit 208 can cause processor 204 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with the software instructions to perform one or more processes described herein. Therefore, the embodiments or aspects described herein are not limited to any particular combination of hardware circuitry and software.

[0096] The memory 206 and / or storage component 208 may include a data storage device or one or more data structures (e.g., a database). The device 200 may be able to receive information from the data storage device or one or more data structures in the memory 206 and / or storage component 208, store information in the data storage device or one or more data structures, transmit information to the data storage device or one or more data structures, or search for information stored in the data storage device or one or more data structures.

[0097] Figure 2 The number and arrangement of components shown are provided as examples. In some non-limiting embodiments or aspects, with Figure 2 Compared to the components shown, device 200 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a set of components of device 200 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of device 200.

[0098] For reference Figure 3 , Figure 3 This is a flowchart of a non-limiting embodiment or aspect of a process 300 for tracking real-time syringe volume. In some non-limiting embodiments or aspects, one or more steps of the multiple steps of process 300 may be performed by user equipment 102 (e.g., one or more devices of the system of user equipment 102, etc.) (e.g., fully performed, partially performed, etc.). In some non-limiting embodiments or aspects, one or more steps of process 300 may be performed by another device or set of devices separate from or including user equipment 102 (e.g., fully performed, partially performed, etc.), such as management system 104 (e.g., one or more devices of management system 104, etc.).

[0099] like Figure 3 As shown, in step 302, process 300 includes acquiring at least two images of the syringe, including a reference marker. For example, user device 102 (and / or management system 104) may acquire at least two images of syringe 103, including a reference marker 138. The at least two images of syringe 103 may include an initial image of syringe 103 and a current image of syringe 103. For example, the initial image of syringe 103 may include an image of syringe 103 captured over a period of time, in an image time series including the at least two images, captured before the current image of syringe 103, such as an earlier image and / or the first image of syringe 103 in the image time series of syringe 103. In such an example, reference marker 138 may encapsulate syringe data associated with syringe 103.

[0100] like Figure 3 As shown, in step 304, process 300 includes: determining a pixel-to-volume mapping associated with the syringe based on a reference marker in at least one image of the syringe. For example, user device 102 (and / or management system 104) may determine the pixel-to-volume mapping associated with syringe 103 based on a reference marker 138 in at least one image of syringe 103 (e.g., in an initial image of syringe 103, a current image of syringe 103, etc.). As an example, user device 102 (and / or management system 104) may process the at least one image of syringe 103 to detect the reference marker 138 therein, read syringe data associated with syringe 103 from the reference marker 138 in the at least one image of syringe 103 to classify the type and / or initial or full volume of syringe 103, and / or retrieve the pixel-to-volume mapping associated with syringe 103 based on the type and / or initial or full volume of syringe 103 (or provide the type and / or initial or full volume of syringe 103 and other inputs to a machine learning model to generate a pixel-to-volume mapping). Further details regarding non-limiting embodiments or aspects of pixel-to-volume mapping are provided below. Figure 3 Step 312 is provided.

[0101] like Figure 3 As shown, in step 306, process 300 includes: for each image of the syringe, determining a region of interest (ROI) in the image based on a reference marker in the image. For example, for each image of syringe 103 in at least two images (e.g., for an initial image of syringe 103, for a current image of syringe 103, etc.), user device 102 (and / or management system 104) may determine the ROI in the image based on a reference marker 138 in the image. As an example, the reference marker 138 may define or identify the ROI in the image having a predetermined width, a predetermined length, and / or a predetermined position relative to the reference marker 138 in the image. In such an example, as Figures 4 to 6As shown, the region of interest in an image may include the movement path of color calibration marker 139 (or first color calibration marker 139a and second color calibration marker 139b) and / or plug 136 and / or plunger 134 from their initial or full syringe volume position to their final or fully dispensed syringe volume position. For example, reference marker 138 may identify or define the middle portion of color calibration marker 139 and / or syringe body 132 as a region of interest only, i.e., only this exact region of interest in each image is processed, rather than the entire image, thereby reducing the processing load associated with processing the image and / or increasing the accuracy when detecting plug 136 and / or plunger 134 in the image.

[0102] In some non-limiting embodiments or aspects, for each of at least two images of the syringe 103 (e.g., for an initial image of the syringe 103, for a current image of the syringe 103, etc.), the user device 102 (and / or management system 104) may determine location information associated with the 3D position of the reference marker 138 (and / or the syringe 103) relative to the image capturing device and / or the 2D position of the reference marker 138 (and / or the syringe 103) in the image itself, based on the reference marker 138 in the image. For example, the user device 102 (and / or management system 104) may determine the location information associated with the 3D position and / or 2D position of the reference marker 138 by determining or recognizing the reference marker 138 (and / or the syringe 103) based on the image and the pose information associated with the 3D position of the reference marker 138 relative to the image capturing device and / or the 2D position of the reference marker 138 (and / or the syringe 103) in the image itself. For example, reference marker 138 may include an AprilTag, and user equipment 102 (and / or management system 104) may use AprilTag detection software to process the image to determine the syringe data and / or unique serial number of syringe 103 encapsulated by reference marker 138, and calculate the precise 3D position, orientation, and / or identification of reference marker 138 (and / or syringe 103) relative to the image capture device capturing the image, and / or the precise 2D position of reference marker 138 (and / or syringe 103). As an example, positional information associated with the 3D position of reference marker 138 (and / or syringe 103) relative to the image capture device may be determined as the 3D position of reference marker 138 associated with syringe 103 relative to the image capture device, and / or positional information associated with the 2D position of syringe 103 in the image itself may be determined as the 2D position of reference marker 138 associated with syringe 103. In such an example, the 3D position of the reference marker 138 (and / or syringe 103) may include the x, y, and z coordinates of the reference marker 138 and / or the direction vectors of the Z, Y, and X axes of the reference marker 138. In such an example, the 2D position of the reference marker 138 in the image itself may include the x, y coordinates of the reference marker 138 in the image and / or the direction vectors of the Y and X axes of the reference marker 138.

[0103] In some non-limiting embodiments or aspects, for each of at least two images of the syringe 103 (e.g., for an initial image of the syringe 103, for a current image of the syringe 103, etc.), the user equipment 102 (and / or management system 104) may use a stereo camera, a lidar system and / or a radar system, etc., to determine positional information associated with the 3D position of the reference marker 138 (and / or the syringe 103) relative to the image capturing device to generate a 3D depth or point cloud of the syringe 103.

[0104] like Figure 3 As shown, in step 308, process 300 includes: generating a color threshold for each image of the syringe. For example, for each image of the syringe 103 in at least two images (e.g., for an initial image of the syringe 103, for a current image of the syringe 103, etc.), the user device 102 (and / or management system 104) may generate the color threshold for the image. As an example, the color threshold may include one or more colors (e.g., a color range, multiple color ranges, one or more color values, etc.) of a color space (e.g., RGB color space, HSV color space, HSL color space, etc.).

[0105] In some non-limiting embodiments or aspects, for each image of syringe 103 in at least two images of syringe 103 (e.g., for an initial image of syringe 103, for a current image of syringe 103, etc.), user device 102 (and / or management system 104) may generate a color threshold for that image based on a predetermined color threshold associated with syringe 103. For example, user device 102 (and / or management system 104) may determine a predetermined color threshold associated with syringe 103 from a plurality of predetermined color thresholds (e.g., from a plurality of predetermined color thresholds associated with a plurality of syringe types and / or volumes stored in a database) based on a reference marker 138 in at least one image of syringe 103 in at least two images of syringe 103.

[0106] In some non-limiting embodiments or aspects, the syringe 103 includes a color calibration mark 139 that includes at least one color, and movable parts of the syringe 103 (e.g., plunger 134, stopper 136, etc.) include that at least one color. For example, for each of at least two images of the syringe 103 (e.g., for an initial image of the syringe 103, for a current image of the syringe 103, etc.), the user device 102 (and / or management system 104) may generate a color threshold for the image based on the at least one color of the color calibration mark 139 in the image. As an example, for each of at least two images of the syringe 103 (e.g., for an initial image of the syringe 103, for a current image of the syringe 103, etc.), the user device 102 (and / or management system 104) may generate a color threshold for the image that is a color range surrounding the color of the color calibration mark 139 in the image.

[0107] In some non-limiting embodiments or aspects, the color calibration mark 139 includes at least two color calibration marks, including a first color calibration mark 139a and a second color calibration mark 139b. The first color calibration mark 139a includes a first color, and the second color calibration mark 139b includes a second color different from the first color. A first portion of the movable part of the syringe (e.g., a first portion of the plunger 134, a first portion 136a of the stopper 136, etc.) includes the first color, and a second portion of the movable part of the syringe (e.g., a second portion of the plunger 134, a second portion 136b of the stopper 136, etc.) that is different from the first portion includes the second color. For example, for each image of the syringe 103 in at least two images of the syringe 103 (e.g., for an initial image of the syringe 103, for a current image of the syringe 103, etc.), the user device 102 (and / or the management system 104) may generate a color threshold for the image based on the first color of the first color calibration mark in the image and the second color of the second color calibration mark in the image. As an example, for each of at least two images of syringe 103 (e.g., for an initial image of syringe 103, for a current image of syringe 103, etc.), user device 102 (and / or management system 104) may generate a color threshold for the image, which is a first color range around the color of a first color calibration mark 139a in the image and a second color range around the color of a second color calibration mark 139b in the image.

[0108] In some non-limiting embodiments or aspects, at least two images of the syringe (e.g., an initial image of syringe 103, a current image of syringe 103, etc.) include a first color space (e.g., RGB color space, etc.), and for each image of syringe 103 in the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), user device 102 (and / or management system 104) may (e.g., using an open-source computer vision library (OpenCV), etc.) convert the image from including the first color space (e.g., RGB color space, etc.) to including a second color space different from the first color space (e.g., HSV color space, etc.). For example, a color threshold for the image may be generated based on the color of color calibration marker 139 in the converted image (e.g., based on the color of the first color calibration marker 139a and the color of the second color calibration marker 139b, etc. in the image), and the region of interest in the converted image may be segmented based on the color threshold to generate a mask image, as described in more detail herein. As an example, user device 102 (and / or management system 104) can convert input image frames from the RGB color space to the HSV color space, which allows the system to more accurately understand the colors or values ​​of the input image frames. In such an example, user device 102 (and / or management system 104) can generate an HSV color threshold for each image to detect and segment the color of the plug 136 in the converted image, as described in more detail herein.

[0109] In this manner, non-limiting embodiments or aspects of this disclosure can continuously calibrate the color threshold of each image or frame on a series of images captured over a period of time to improve the detection of the color of the plug and / or plunger and to track the real-time volume of the syringe under lighting conditions that can vary over that period of time.

[0110] like Figure 3As shown, in step 310, process 300 includes: for each image of the syringe, segmenting the region of interest in the image based on a color threshold to generate a mask image. For example, for each image of syringe 103 in at least two images of syringe 103 (e.g., for an initial image of syringe 103, for a current image of syringe 103, etc.), user device 102 (and / or management system 104) may segment the region of interest in the image based on a color threshold to generate a mask image. As an example, for each image of syringe 103 in at least two images of syringe 103 (e.g., for an initial image of syringe 103, for a current image of syringe 103, etc.), user device 102 (and / or management system 104) may generate a mask image as a binary mask (and / or a dual binary mask for a syringe including two color calibration marks), which maps or defines the regions of stopper 136 and / or plunger 134 that are within the color threshold of the image. In such an example, for each of at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), user device 102 (and / or management system 104) may (e.g., using OpenCV, etc.) apply one or more morphological operations to a binary mask to generate a contour and / or connect groups or blocks of pixels in the mask image.

[0111] In some non-limiting embodiments or aspects, for each of at least two images of syringe 103 (e.g., for an initial image of syringe 103, for a current image of syringe 103, etc.), user equipment 102 (and / or management system 104) may control a display to show the image of the syringe on the display, the image having a mask image generated from that image overlaid thereon. For example, as Figures 4 to 6 As shown, implementations 400, 500, and 600 of a non-limiting embodiment or aspect of an image of a syringe overlaid with a corresponding mask image are illustrated. The mask image, generated by segmenting the region of interest of the image of syringe 103, can be overlaid on the image of syringe 103 to visualize the tracking of the stopper 136 and / or plunger 134 in real time as the stopper 136 and / or plunger 134 moves relative to syringe barrel 132 over time in a series of images of syringe 103 while fluid is being dispensed from syringe 103.

[0112] like Figure 3As shown, in step 312, process 300 includes: determining at least one of the following based on the pixel-to-volume mapping associated with the syringe and a mask image generated from each image of the syringe: the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof. For example, user device 102 (and / or management system 104) may determine the following based on the pixel-to-volume mapping associated with syringe 103: a mask image generated from an initial image of syringe 103; a mask image generated from a current image of syringe 103; at least one of the fluid volume allocated from syringe 103, the remaining fluid volume in syringe 103, or any combination thereof.

[0113] As an example, user device 102 (and / or management system 104) can determine at least one of the following: the fluid volume dispensed from syringe 103, the remaining fluid volume in syringe 103, or any combination thereof, by determining, for each of at least two images of syringe 103, at least one centroid pixel of at least a group of pixels in a mask image that satisfies a color threshold of that image (e.g., the centroid pixel of the region of plug 136 and / or plunger 134 within the color threshold of that image, the centroid pixel of a blob in the mask image, etc.). In such an example, user device 102 (and / or management system 104) can determine at least one of the following: the fluid volume dispensed from syringe, the remaining fluid volume in syringe, or any combination thereof, based on a pixel-to-volume mapping associated with syringe, at least one centroid pixel of a mask image generated from an initial image of syringe, and at least one centroid pixel of a mask image generated from a current image of syringe.

[0114] For example, user equipment 102 (and / or management system 104) may provide at least one centroid pixel of the mask image as input to the pixel-to-volume mapping and receive at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof as output from the pixel-to-volume mapping. As an example, through pixel-to-volume mapping, centroid pixels associated with the initial image of syringe 103 in at least two images of syringe 103 (e.g., earlier and / or first images of syringe 103 in an image time series of syringe 103, etc.) can be assigned or determined to correspond to the initial or full syringe volume (e.g., the initial or full syringe volume position of stopper 136 and / or plunger 134 in its movement path, etc.), and subsequent centroid pixels associated with subsequent images of syringe 103 in at least two images of syringe 103 (e.g., the current image of syringe 103 in an image time series of syringe 103, etc.) input to pixel-to-volume mapping can provide changes in syringe volume from the initial or full syringe volume or from another earlier or previous volume determined from another earlier or previous image in the image time series (e.g., changes in the position of stopper 136 and / or plunger 134 in its movement path from the initial or full volume position or from another earlier or previous position of stopper 136 and / or plunger 134 in its movement path, etc.).

[0115] In some non-limiting embodiments or aspects, for each image of syringe 103 in at least two images of syringe 103, user equipment 102 (and / or management system 104) determines at least two centroid pixels in a mask image that satisfy a color threshold of at least two groups of pixels. The at least two centroid pixels include a first centroid pixel of a first plurality of pixels that satisfy a first color range threshold of the color threshold of the image (e.g., a first centroid pixel of a first region of plug 136 and / or plunger 134 within the first color threshold of the image, a first centroid pixel of a first block in the mask image, etc.) and a second centroid pixel of a second plurality of pixels that satisfy a second color range threshold of the image (e.g., a second centroid pixel of a second region of plug 136 and / or plunger 134 within the second color threshold of the image, a second centroid pixel of a second block in the mask image, etc.). For example, syringe 103 may include at least two color calibration marks, including a first color calibration mark 139a and a second color calibration mark 139b, wherein the first color calibration mark 139a includes a first color and the second color calibration mark 139b includes a second color different from the first color. For example, user equipment 102 (and / or management system 104) may determine at least one of the following based on pixel-to-volume mapping associated with syringe 103, a first centroid pixel of a mask image generated from an initial image of syringe 103, a second centroid pixel of a mask image generated from the initial image of syringe 103, a first centroid pixel of a mask image generated from a current image of syringe 103, and a second centroid pixel of a mask image generated from the current image of syringe 103: the fluid volume dispensed from syringe 103, the remaining fluid volume in syringe 103, or any combination thereof.

[0116] In this way, the centroid pixel or the continuous mapping of pixels to syringe volume for each image or frame can enable real-time tracking of the fluid volume dispensed from the syringe and / or remaining in the syringe.

[0117] In some non-limiting embodiments or aspects, the pixel-to-volume mapping includes a hard-coded or predetermined range of centroid pixels mapped to the volume level of syringe 103. For example, the range of centroid pixels can be generated from an experimental setup in which syringe 103 is held in a static position and / or orientation relative to an image capture device capturing a series of images of syringe 103 as stopper 136 and / or plunger 134 move from their initial or full syringe volume position to their final or fully allocated syringe volume position, and the corresponding volume of syringe 103 is measured (e.g., by a flow sensor, etc.). As an example, the range of centroid pixels of the mask image of each image in the series of images can be mapped to the corresponding volume of syringe 103 measured when that image in the series of images is captured to generate a hard-coded or predetermined pixel-to-volume mapping. In such examples, the hard-coded or predetermined pixel-to-volume mapping can be stored (e.g., in a database, etc.) in association with syringe 103 (e.g., with syringe type and / or volume of syringe 103, etc.) and / or position and / or orientation of syringe 103 relative to the image capture device. Therefore, user equipment 102 (and / or management system 104) can retrieve a stored, hard-coded, or predetermined pixel-to-volume map associated with syringe 103 from a plurality of stored, hard-coded, or predetermined pixel-to-volume maps based on syringe data determined from a reference marker 138 associated with syringe 103 and / or the 2D or 3D position and / or orientation of syringe 103 determined from one or more images of syringe 103 in at least two images of syringe 103.

[0118] In some non-limiting embodiments or aspects, the pixel-to-volume mapping includes and / or is generated by a machine learning model configured to generate at least one of the following from inputs including a mask image generated from an initial image of the syringe and a mask image generated from a current image of the syringe (e.g., one or more centroid pixels associated with the initial image of syringe 103 and one or more centroid pixels associated with the current image of syringe 103, etc.): the volume of fluid allocated from the syringe, the volume of fluid remaining in the syringe, or any combination thereof. For example, user equipment 102 (and / or management system 104) may utilize this machine learning model to process images of syringe 103, regions of interest or bounding boxes of the images, syringe data, and / or the position and / or orientation of syringe 103 determined for each image to generate predicted volumes allocated from syringe 103 and / or remaining in syringe 103. As an example, user device 102 (and / or management system 104) may use machine learning techniques to generate predictive models (e.g., estimators, classifiers, predictive models, detector models, etc.). These machine learning techniques include, for example, supervised and / or unsupervised techniques such as decision trees (e.g., gradient boosting decision trees, random forests, etc.), logistic regression, artificial neural networks (e.g., convolutional neural networks, etc.), Bayesian statistics, learning automata, hidden Markov modeling, linear classifiers, quadratic classifiers, and / or association rule learning, etc. The predictive machine learning model may be trained to provide an output that includes a prediction of at least one of the fluid volume dispensed from syringe 103, the remaining fluid volume in syringe 103, or any combination thereof (e.g., pixel-to-volume mapping at the pixel level, etc., where each pixel may correspond to different amounts of change in syringe volume). In such an example, the prediction may include a probability (e.g., likelihood, etc.) of at least one of the fluid volume dispensed from syringe 103, the remaining fluid volume in syringe 103, or any combination thereof.

[0119] User equipment 102 (and / or management system 104) can generate a predictive model based on training data determined from an experimental setup in which, as stopper 136 and / or plunger 134 move from their initial or full syringe volume position to their final or fully dispensed syringe volume position, syringe 103 can be held in a static and / or changing position and / or orientation relative to an image capture device that captures a series of images of syringe 103, and the corresponding volume of syringe 103 is measured (e.g., by a flow sensor, etc.). In some implementations, the prediction model is designed to receive a mask image generated from an initial image of the syringe (e.g., centroid pixels associated with the initial image of syringe 103, etc.), a mask image generated from a current image of the syringe (e.g., centroid pixels associated with the current image of syringe 103, etc.), syringe data (e.g., syringe type and / or initial or full volume of syringe 103, etc.), and / or the position and / or orientation of the syringe associated with each image (e.g., relative to the image capture device and / or user equipment 102, etc.) as input, and to provide a prediction (e.g., probability, likelihood, binary output, yes-no output, score, predicted score, classification, etc.) of at least one of the fluid volume dispensed from syringe 103, the remaining fluid volume in syringe 103, or any combination thereof, as output. In this implementation, the measured volume of syringe 103 can be used as a label for the corresponding input. In some non-limiting embodiments or aspects, user equipment 102 stores the prediction model (e.g., stores the model for later use). In some non-limiting embodiments or aspects, user equipment 102 (and / or management system 104) stores the initial prediction model in a data structure (e.g., a database, linked list, tree, etc.). In some non-limiting embodiments, the data structure is located within user equipment 102 (and / or management system 104) or outside user equipment 102 (and / or management system 104) (e.g., within a database system, etc.).

[0120] In some non-limiting embodiments or aspects, the prediction model associated with syringe 103 may be stored in association with syringe 103 (e.g., with syringe type and / or volume, etc.) and / or with the position and / or orientation of syringe 103 relative to the image capture device. Therefore, user equipment 102 (and / or management system 104) may retrieve the stored prediction model associated with syringe 103 from a plurality of stored prediction models based on syringe data determined from a reference marker 138 associated with syringe 103 and / or the 2D or 3D position and / or orientation of syringe 103 determined from one or more images of syringe 103 in at least two images of syringe 103.

[0121] In some non-limiting embodiments or aspects, the predictive model may be associated with multiple different syringe types and / or volumes and / or syringe positions and / or orientations relative to image capture devices. For example, the predictive model may be trained based on training data determined from multiple experimental settings in which multiple syringes of different syringe types and / or volumes may remain in static and / or varying positions and / or orientations relative to multiple image capture devices capturing multiple series of images of the multiple syringes as their stoppers 136 and / or plungers 134 move from their initial or full syringe volume positions to their final or fully allocated syringe volume positions, and the corresponding volumes of the multiple syringes are measured (e.g., by flow sensors, etc.).

[0122] Therefore, pixel-to-volume mapping can be configured to track volume levels in a syringe by computing pixel information using hard-coded or predefined ranges and / or machine learning techniques to generate an automated pixel-to-volume mapping model for tracking real-time syringe volume.

[0123] like Figure 3 As shown, in step 314, process 300 includes providing at least one of the following: a fluid volume dispensed from the syringe, a fluid volume remaining in the syringe, or any combination thereof. For example, user equipment 102 (and / or management system 104) can provide at least one of a fluid volume dispensed from syringe 103, a fluid volume remaining in syringe 103, or any combination thereof. As an example, user equipment 102 (and / or management system 104) can control a display (e.g., output component 212, etc.) to display in real time at least one of a fluid volume dispensed from syringe 103, a fluid volume remaining in syringe 103, or any combination thereof. For example, as... Figures 4 to 6 As shown, when fluid is dispensed from syringe 103, at least one of the following can be displayed in real time in a series of images of syringe 103 as the stopper 136 and / or plunger 134 moves relative to syringe barrel 132 over time: the volume of fluid dispensed from syringe 103, the volume of fluid remaining in syringe 103, or any combination thereof (e.g., simultaneously displaying a mask image generated by segmenting the region of interest of the image of syringe 103 overlaid on the image of syringe 103 to visualize the tracking of stopper 136 and / or plunger 134 in real time).

[0124] In some non-limiting embodiments or aspects, user equipment 102 (and / or management system 104) may compare at least one of the following: the volume of fluid dispensed from syringe 103, the remaining volume of fluid in syringe 103, or any combination thereof (and / or the fluid delivery rate determined therefrom) with one or more thresholds, and if the volume of fluid dispensed from syringe 103, the remaining volume of fluid in syringe 103, or any combination thereof (and / or the fluid delivery rate determined therefrom) violates one or more thresholds, then (e.g., via user equipment 102, etc.) an associated alarm is provided.

[0125] In some non-limiting embodiments or aspects, user equipment 102 (and / or management system 104) may aggregate at least one of the following from a multiple injection or fluid delivery procedure: the volume of fluid dispensed from syringe 103, the volume of fluid remaining in syringe 103, or any combination thereof (and / or the fluid delivery rate determined therefrom) to determine one or more metrics associated with syringe type and / or the volume of syringe 103 and / or the user performing multiple injections.

[0126] Although embodiments or aspects have been described in detail for purposes of illustration and description, it will be understood that such detail is for that purpose only, and that embodiments or aspects are not limited to the disclosed embodiments or aspects, but rather are intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it will be understood that this disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment or aspect. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or not specifically disclosed in the specification. Although each dependent claim listed below may be directly dependent on only one claim, the disclosed possible implementations include combinations of each dependent claim with each other claim in the group of claims.

Claims

1. A system comprising: At least one processor, said at least one processor being coupled to memory and configured to: Acquire at least two images of a syringe, including a reference marker, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the reference marker encapsulates syringe data associated with the syringe; Based on the reference marker in at least one image of the syringe from the at least two images of the syringe, a pixel-to-volume mapping associated with the syringe is determined; For each of the at least two images of the syringe: Based on the reference markers in the image, the region of interest in the image is determined; and Based on the color threshold of the image, the region of interest in the image is segmented to generate a mask image; Based on the pixel-to-volume mapping associated with the syringe, a mask image generated from the initial image of the syringe, and a mask image generated from the current image of the syringe, at least one of the following is determined: the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof; and Provide at least one of the following: the fluid volume dispensed from the syringe, the fluid volume remaining in the syringe, or any combination thereof.

2. The system according to claim 1, wherein, The syringe includes at least one color calibration mark, the at least one color calibration mark includes at least one color, wherein a movable part of the syringe includes the at least one color, and wherein the at least one processor is further configured to: For each of the at least two images of the syringe: The color threshold of the image is generated based on the at least one color of the at least one color calibration mark in the image.

3. The system according to claim 2, wherein, The at least one color calibration mark includes at least two color calibration marks, the at least two color calibration marks including a first color calibration mark and a second color calibration mark, the first color calibration mark including a first color, the second color calibration mark including a second color different from the first color, wherein a first portion of the movable part of the syringe includes the first color, wherein a second portion of the movable part of the syringe, different from the first portion, includes the second color, and wherein the at least one processor is further configured to: For each of the at least two images of the syringe: The color threshold of the image is generated based on the first color of the first color calibration mark in the image and the second color of the second color calibration mark in the image.

4. The system according to claim 2, wherein, The at least two images of the syringe include a first color space, and wherein the at least one processor is further configured to: For each of the at least two images of the syringe: The image is converted from including the first color space to including a second color space different from the first color space, wherein the color threshold of the image is generated based on at least one color of at least one color calibration mark in the converted image, and wherein the region of interest in the converted image is segmented based on the color threshold of the image to generate the mask image.

5. The system according to claim 1, wherein, The at least one processor is further configured to determine at least one of the following: the fluid volume dispensed from the syringe, the fluid volume remaining in the syringe, or any combination thereof: For each of the at least two images of the syringe: Determine at least one centroid pixel of at least one group of pixels in the masked image that satisfies the color threshold of the image. The fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, at least one of these is determined based on the pixel-to-volume mapping associated with the syringe, at least one centroid pixel of a mask image generated from the initial image of the syringe, and at least one centroid pixel of a mask image generated from the current image of the syringe.

6. The system according to claim 5, wherein, The at least one processor is further configured to: for each of the at least two images of the syringe, determine at least two centroid pixels in the mask image that satisfy at least two groups of pixels with a color threshold of the image, wherein the at least two centroid pixels include a first centroid pixel of a first plurality of pixels satisfying a first color range threshold of the color threshold of the image and a second centroid pixel of a second plurality of pixels satisfying a second color range threshold of the image. The fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, at least one of these is determined based on the pixel-to-volume mapping associated with the syringe, the first centroid pixel of the mask image generated from the initial image of the syringe, the second centroid pixel of the mask image generated from the initial image of the syringe, the first centroid pixel of the mask image generated from the current image of the syringe, and the second centroid pixel of the mask image generated from the current image of the syringe.

7. The system according to claim 1, wherein, The pixel-to-volume mapping includes hard-coded or pre-defined pixel-to-volume mapping.

8. The system according to claim 1, wherein, The pixel-to-volume mapping includes a machine learning model configured to generate at least one of the following based on a mask image generated from the initial image of the syringe and a mask image generated from the current image of the syringe: the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof.

9. The system according to claim 1, wherein, The at least one processor is further configured to: For each of the at least two images of the syringe: The control display shows the image of the syringe on the display, the image having the mask image generated from the image overlaid thereon.

10. The system according to claim 1, wherein, The at least one processor is further configured to: For each of the at least two images of the syringe: Based on the reference markers in the image, positional information associated with the three-dimensional (3D) position of the syringe relative to the image capture device that captured the image is determined, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

11. A method comprising: At least two images of a syringe, including a reference marker, are acquired using at least one processor, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the reference marker encapsulates syringe data associated with the syringe; Using the at least one processor, a pixel-to-volume mapping associated with the syringe is determined based on the reference marker in at least one image of the syringe from at least two images of the syringe; For each of the at least two images of the syringe: Using the at least one processor, the region of interest in the image is determined based on reference markers in the image; and Using the at least one processor, the region of interest in the image is segmented based on a color threshold to generate a mask image; Using the at least one processor, based on the pixel-to-volume mapping associated with the syringe, a mask image generated from the initial image of the syringe, and a mask image generated from the current image of the syringe, at least one of the following is determined: the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof; and Using the at least one processor, at least one of the following is provided: a fluid volume dispensed from the syringe, a fluid volume remaining in the syringe, or any combination thereof.

12. The method according to claim 11, wherein, The syringe includes at least one color calibration mark, the at least one color calibration mark includes at least one color, wherein a movable part of the syringe includes the at least one color, and wherein the method further includes: For each of the at least two images of the syringe: Using the at least one processor, the color threshold of the image is generated based on the at least one color of the at least one color calibration mark in the image.

13. The method according to claim 12, wherein, The at least one color calibration mark includes at least two color calibration marks, the at least two color calibration marks including a first color calibration mark and a second color calibration mark, the first color calibration mark including a first color, the second color calibration mark including a second color different from the first color, wherein a first portion of the movable part of the syringe includes the first color, wherein a second portion of the movable part of the syringe, different from the first portion, includes the second color, and wherein the method further includes: For each of the at least two images of the syringe: Using the at least one processor, the color threshold of the image is generated based on the first color of the first color calibration mark in the image and the second color of the second color calibration mark in the image.

14. The method according to claim 12, wherein, The at least two images of the syringe include a first color space, and the method further includes: For each of the at least two images of the syringe: Using the at least one processor, the image is converted from including the first color space to including a second color space different from the first color space, wherein the color threshold of the image is generated based on at least one color of at least one color calibration mark in the converted image, and wherein the region of interest in the converted image is segmented based on the color threshold of the image to generate the mask image.

15. The method according to claim 11, wherein, Determine at least one of the following: the volume of fluid dispensed from the syringe, the volume of fluid remaining in the syringe, or any combination thereof, further comprising: For each of the at least two images of the syringe: Using the at least one processor, at least one centroid pixel of at least one group of pixels in the masked image that satisfies the color threshold of the image is determined. The fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, at least one of these is determined based on the pixel-to-volume mapping associated with the syringe, at least one centroid pixel of a mask image generated from the initial image of the syringe, and at least one centroid pixel of a mask image generated from the current image of the syringe.

16. The method according to claim 15, wherein, Determine at least one of the following: the volume of fluid dispensed from the syringe, the volume of fluid remaining in the syringe, or any combination thereof, further including: For each of the at least two images of the syringe: Using the at least one processor, at least two centroid pixels of at least two groups of pixels in the mask image that satisfy the color threshold of the image are determined. The at least two centroid pixels include a first centroid pixel of a first group of pixels satisfying a first color range threshold of the color threshold of the image, and a second centroid pixel of a second group of pixels satisfying a second color range threshold of the image. The fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, at least one of these is determined based on the pixel-to-volume mapping associated with the syringe, the first centroid pixel of the mask image generated from the initial image of the syringe, the second centroid pixel of the mask image generated from the initial image of the syringe, the first centroid pixel of the mask image generated from the current image of the syringe, and the second centroid pixel of the mask image generated from the current image of the syringe.

17. The method according to claim 11, wherein, The pixel-to-volume mapping includes hard-coded or pre-defined pixel-to-volume mapping.

18. The method according to claim 11, wherein, The pixel-to-volume mapping includes a machine learning model configured to generate at least one of the following based on a mask image generated from the initial image of the syringe and a mask image generated from the current image of the syringe: the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof.

19. The method of claim 11, further comprising: For each of the at least two images of the syringe: Using the at least one processor, a display is controlled to show the image of the syringe on the display, the image having a mask image generated from the image overlaid thereon.

20. The method of claim 11, further comprising: For each of the at least two images of the syringe: Using the at least one processor, positional information associated with the three-dimensional (3D) position of the syringe relative to the image capture device that captured the image is determined based on the reference marker in the image, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

21. A computer program product comprising at least one non-transitory computer-readable medium, said at least one non-transitory computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: Obtain at least two images of the syringe, including reference markers, wherein, The at least two images of the syringe include an initial image of the syringe and a current image of the syringe, wherein the reference marker encapsulates syringe data associated with the syringe; Based on the reference marker in at least one image of the syringe from the at least two images of the syringe, a pixel-to-volume mapping associated with the syringe is determined; For each of the at least two images of the syringe: Based on the reference markers in the image, the region of interest in the image is determined; and Based on the color threshold of the image, the region of interest in the image is segmented to generate a mask image; Based on the pixel-to-volume mapping associated with the syringe, a mask image generated from the initial image of the syringe, and a mask image generated from the current image of the syringe, at least one of the following is determined: the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof; and Provide at least one of the following: the fluid volume dispensed from the syringe, the fluid volume remaining in the syringe, or any combination thereof.

22. The computer program product according to claim 21, wherein, The syringe includes at least one color calibration mark, the at least one color calibration mark includes at least one color, wherein a movable part of the syringe includes the at least one color, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: For each of the at least two images of the syringe: The color threshold of the image is generated based on the at least one color of the at least one color calibration mark in the image.

23. The computer program product according to claim 22, wherein, The at least one color calibration mark includes at least two color calibration marks, the at least two color calibration marks including a first color calibration mark and a second color calibration mark, the first color calibration mark including a first color, the second color calibration mark including a second color different from the first color, wherein a first portion of the movable part of the syringe includes the first color, wherein a second portion of the movable part of the syringe, different from the first portion, includes the second color, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: For each of the at least two images of the syringe: The color threshold of the image is generated based on the first color of the first color calibration mark in the image and the second color of the second color calibration mark in the image.

24. The computer program product according to claim 22, wherein, The at least two images of the syringe include a first color space, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: For each of the at least two images of the syringe: The image is converted from including the first color space to including a second color space different from the first color space, wherein the color threshold of the image is generated based on at least one color of at least one color calibration mark in the converted image, and wherein the region of interest in the converted image is segmented based on the color threshold of the image to generate the mask image.

25. The computer program product according to claim 21, wherein, When executed by the at least one processor, the program instructions further cause the at least one processor to determine at least one or any combination of the fluid volume dispensed from the syringe, the fluid volume remaining in the syringe, and so on: For each of the at least two images of the syringe: Determine at least one centroid pixel of at least one group of pixels in the masked image that satisfies the color threshold of the image. The fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, at least one of these is determined based on the pixel-to-volume mapping associated with the syringe, at least one centroid pixel of a mask image generated from the initial image of the syringe, and at least one centroid pixel of a mask image generated from the current image of the syringe.

26. The computer program product according to claim 25, wherein, When executed by the at least one processor, the program instructions further cause the at least one processor to: for each of the at least two images of the syringe, determine at least two centroid pixels in the mask image that satisfy the color threshold of the image, wherein the at least two centroid pixels include a first centroid pixel of a first plurality of pixels satisfying a first color range threshold of the color threshold of the image and a second centroid pixel of a second plurality of pixels satisfying a second color range threshold of the image. The fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof, at least one of these is determined based on the pixel-to-volume mapping associated with the syringe, the first centroid pixel of the mask image generated from the initial image of the syringe, the second centroid pixel of the mask image generated from the initial image of the syringe, the first centroid pixel of the mask image generated from the current image of the syringe, and the second centroid pixel of the mask image generated from the current image of the syringe.

27. The computer program product according to claim 21, wherein, The pixel-to-volume mapping includes hard-coded or pre-defined pixel-to-volume mapping.

28. The computer program product according to claim 21, wherein, The pixel-to-volume mapping includes a machine learning model configured to generate at least one of the following based on a mask image generated from the initial image of the syringe and a mask image generated from the current image of the syringe: the fluid volume allocated from the syringe, the remaining fluid volume in the syringe, or any combination thereof.

29. The computer program product according to claim 21, wherein, When the program instructions are executed by the at least one processor, the at least one processor also causes the at least one processor to: For each of the at least two images of the syringe: The control display shows the image of the syringe on the display, the image having the mask image generated from the image overlaid thereon.

30. The computer program product according to claim 21, wherein, When the program instructions are executed by the at least one processor, the at least one processor also causes the at least one processor to: For each of the at least two images of the syringe: Based on the reference markers in the image, positional information associated with the three-dimensional (3D) position of the syringe relative to the image capture device that captured the image is determined, wherein at least one of the fluid volume dispensed from the syringe, the remaining fluid volume in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

Images