Systems and methods for estimating object distance and / or angle from an image capture device

Abstract

Systems and methods for estimating the distance and / or angle of an object from an image capture device may acquire an image including a fiducial marker captured by an image capture device, extract the pixel width and / or pixel height of the fiducial marker in the image space of the image, determine the distance of the fiducial marker from the image capture device based on the pixel width and / or pixel height, the known width and / or known height, and one or more calibration parameters, estimate the yaw angle of the fiducial marker relative to the image capture device based on the pixel width, pixel height, known width, and known height, determine the roll angle of the fiducial marker relative to the image capture device based on the slope of the position vector of the fiducial marker, and / or provide the distance, yaw angle, and / or roll angle.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 63 / 441,031, entitled "System and Method for Estimating Object Distance and / or Angle from an Image Capture Device," filed January 25, 2023, and this application also claims priority to Indian Provisional Patent Application No. 202211042859, entitled "System and Method for Vascular Access Management," filed July 26, 2022, the entire disclosures of each of which are incorporated herein by reference in their entirety.

[0002] SUMMARY OF THE INVENTION Improved systems, devices, products, apparatus, and / or methods are provided for estimating the distance and / or angle of an object from an image capture device. [Background technology]

[0003] The goal of computer vision may be to find attributes of identified objects such as shape, color, distance from the viewing plane, and orientation angle.

[0004] A dynamic monocular camera used in a handheld manner for image capture may result in some loss of region of interest (ROI) due to the distance between the camera and an object of interest within the camera's field of view (FOV). For example, distance estimation may be used in certain dynamic environments to identify the correct FOV for capturing image information from the camera.

[0005] An object has three-dimensional rotational motions of pitch, yaw, and roll, which result from object and / or camera motion. Existing systems for calculating the object's yaw and roll angles consume significant processing resources and time for the calculations. Summary of the Invention

[0006] Thus, improved systems, devices, articles of manufacture, apparatus, and / or methods for estimating the distance and / or angle of an object from an image capture device are provided.

[0007] According to certain non-limiting embodiments or aspects, a method includes: acquiring a calibration image, coupled to a memory, including a calibration marker, wherein the calibration image including the calibration marker is captured by an image capture device with the calibration marker located at a known distance from the image capture device, the calibration marker having known dimensions; extracting one or more features associated with the calibration marker in an image space of the calibration image; and determining one or more calibration parameters based on the one or more features associated with the calibration marker in the image space of the calibration image, the known distance of the calibration marker from the image capture device, and the known dimensions of the calibration marker; A system is provided that includes acquiring an image including an object, where the image including the object is captured by an image capture device; extracting at least one feature associated with the object in an image space of the image; acquiring dimensions of the object; determining a distance of the object from the image capture device based on the at least one feature associated with the object in the image space of the image, the dimensions of the object, and one or more calibration parameters; and providing the distance of the object from the image capture device.

[0008] In certain non-limiting embodiments or aspects, the one or more features associated with the calibration marker in the image space of the calibration image include pixel dimensions of the calibration marker in the image space of the calibration image.

[0009] In certain non-limiting embodiments or aspects, the one or more calibration parameters include an effective focal length associated with the image capture device, the effective focal length being determined by the following formula: F=(P×D) / W is determined in accordance with where F is the effective focal length, P is the pixel dimension of the calibration marker in the image space of the calibration image, D is the known distance of the calibration marker from the image capture device, and W is the known dimension of the calibration marker.

[0010] In certain non-limiting embodiments or aspects, the at least one feature associated with the object in the image space of the image includes a pixel dimension of the object in the image space of the image.

[0011] In certain non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to obtain the dimensions of the object by storing in memory a plurality of dimensions associated with a plurality of types of object, determining the type of object based on the image, and determining the dimensions of the object from the plurality of dimensions associated with the plurality of types of object based on the type of object.

[0012] In some non-limiting embodiments or aspects, the at least one processor determines a pixel size based on a number of pixels associated with the object in the image, a known dimension of the at least one other object, and a number of pixels associated with the at least one other object in the image. 、 The device is programmed and / or configured to obtain the dimensions of the object by estimating the dimensions of the object.

[0013] In some non-limiting embodiments or aspects, at least one processor is configured to calculate a value for a signal that satisfies the following formula: D'=(W'×F) / P' and (c) determining a distance of the object from the image capture device according to the method of claim 1; where D' is the distance of the object from the image capture device, W' is the size of the object, F is the effective focal length, and P' is the pixel size of the object in the image space of the image.

[0014] In certain non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to provide the distance of the object from the image capture device by displaying, on the display, an image including the object concurrently with the distance of the object from the image capture device.

[0015] In certain non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to provide the distance of the object from the image capture device by using the distance of the object from the image capture device to determine whether the object is connected to another object in the image.

[0016] In certain non-limiting embodiments or aspects, at least one processor is programmed and / or configured to acquire an image including an object by acquiring a first image including the object captured by an image capture device at a first angle relative to the object, and acquiring a second image including the object captured by the image capture device at a second angle relative to the object different from the first angle, wherein the at least one processor is programmed and / or configured to: use an image feature detector algorithm to detect a plurality of first interest points associated with the object in the first image and a plurality of second interest points associated with the object in the second image; and use an image feature descriptor algorithm to generate a plurality of first descriptor vectors associated with the plurality of first interest points and a plurality of second interest points. The computer program is programmed and / or configured to extract at least one feature associated with the object in an image space of the image by: generating a plurality of associated second descriptor vectors; matching at least one first interest point of the plurality of first interest points to at least one second interest point of the plurality of second interest points based on the plurality of first descriptor vectors associated with the plurality of first interest points and the plurality of second descriptor vectors associated with the plurality of second interest points using a feature matching algorithm; and determining position information associated with a three-dimensional (3D) position of the object relative to the image capture device based on the at least one interest point of the plurality of first interest points matched to the at least one interest point of the plurality of second interest points.

[0017] According to some non-limiting embodiments or aspects, using an image capture device to capture a calibration image including the calibration markers with known dimensions located at a known distance from the image capture device; A method is provided that includes calibrating an image capture device by extracting, with at least one processor, one or more features associated with a calibration marker in image space of a calibration image; and determining, with the at least one processor, one or more calibration parameters based on the one or more features associated with the calibration marker in image space of the calibration image, a known distance of the calibration marker from the image capture device, and known dimensions of the calibration marker; capturing an image including an object with the image capture device; extracting, with the at least one processor, at least one feature associated with the object in image space of the image; obtaining, with the at least one processor, dimensions of the object; determining, with the at least one processor, a distance of the object from the image capture device based on the at least one feature associated with the object in image space of the image, the dimensions of the object, and the one or more calibration parameters; and providing, with the at least one processor, the distance of the object from the image capture device.

[0018] In certain non-limiting embodiments or aspects, the one or more features associated with the calibration marker in the image space of the calibration image include pixel dimensions of the calibration marker in the image space of the calibration image.

[0019] In certain non-limiting embodiments or aspects, the one or more calibration parameters include an effective focal length associated with the image capture device, the effective focal length being determined according to the following equation: F=(P×D) / W where F is the effective focal length, P is the pixel dimension of the calibration marker in image space of the calibration image, D is the known distance of the calibration marker from the image capture device, and W is the known dimension of the calibration marker.

[0020] In certain non-limiting embodiments or aspects, the at least one feature associated with the object in the image space of the image includes a pixel dimension of the object in the image space of the image.

[0021] In some non-limiting embodiments or aspects, obtaining the dimensions of the object includes storing in a memory a plurality of dimensions associated with a plurality of types of objects, determining the type of object based on the image, and determining the dimensions of the object from the plurality of dimensions associated with the plurality of types of objects based on the type of object.

[0022] In some non-limiting embodiments or aspects, obtaining the dimensions of the object includes estimating the dimensions of the object based on a number of pixels associated with the object in the image, known dimensions of at least one other object, and a number of pixels associated with at least one other object in the image.

[0023] In some non-limiting embodiments or aspects, the distance of the object from the image capture device is determined according to the following formula: D'=(W'×F) / P' where D' is the distance of the object from the image capture device, W' is the size of the object, F is the effective focal length, and P' is the pixel size of the object in the image space of the image.

[0024] In some non-limiting embodiments or aspects, providing the distance of the object from the image capture device includes at least one of displaying, on the display, the distance of the object from the image capture device simultaneously with an image including the object, using the distance of the object from the image capture device to determine whether the object is connected to another object in the image, or any combination thereof.

[0025] In certain non-limiting embodiments or aspects, capturing an image including the object using an image capture device includes capturing a first image including the object at a first angle relative to the object using the image capture device, and capturing a second image including the object at a second angle relative to the object that is different from the first angle using the image capture device; and extracting at least one feature associated with the object in image space of the image using at least one processor includes detecting a plurality of first interest points associated with the object in the first image and a plurality of second interest points associated with the object in the second image using an image feature detector algorithm; and extracting at least one feature associated with the object in image space of the image using an image feature descriptor algorithm. generating a plurality of first descriptor vectors associated with a number of first interest points and a plurality of second descriptor vectors associated with a number of second interest points; matching at least one first interest point of the plurality of first interest points to at least one second interest point of the plurality of second interest points based on the plurality of first descriptor vectors associated with the plurality of first interest points and the plurality of second descriptor vectors associated with the plurality of second interest points using a feature matching algorithm; and determining position information associated with a three-dimensional (3D) position of the object relative to the image capture device based on the at least one first interest point of the plurality of first interest points matched to the at least one second interest point of the plurality of second interest points.

[0026] According to certain non-limiting embodiments or aspects, a computer program product is provided that includes at least one non-transitory computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: acquire a calibration image including a calibration marker, the calibration image including the calibration marker being captured by an image capture device using a calibration marker positioned a known distance from the image capture device, the calibration marker having known dimensions; extract one or more features associated with the calibration marker in an image space of the calibration image; determine one or more calibration parameters based on the one or more features associated with the calibration marker in the image space of the calibration image, the known distance of the calibration marker from the image capture device, and the known dimensions of the calibration marker; acquire an image including an object, the image including the object being captured by the image capture device; extract at least one feature associated with the object in an image space of the image; obtain dimensions of the object; determine a distance of the object from the image capture device based on the at least one feature associated with the object in the image space of the image, the dimensions of the object, and the one or more calibration parameters; and provide the distance of the object from the image capture device.

[0027] According to certain non-limiting embodiments or aspects, a system is provided that includes at least one processor coupled to a memory and programmed and / or configured to: acquire an image including a fiducial marker captured by an image capture device, the fiducial marker having a known width and a known height; extract a pixel width and a pixel height of the fiducial marker in image space of the image; estimate a yaw angle of the fiducial marker relative to the image capture device based on the pixel width, pixel height, and known width and height; and provide a yaw angle of the fiducial marker relative to the image capture device.

[0028] In certain non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to estimate the yaw angle of the fiducial marker relative to the image capture device according to the following equation: Yaw angle = α*β where β is a multiplication factor and α is determined according to the following formula: α=((w k / h k )-(w p / h p ))*90 During the ceremony, w k is the known width of the fiducial marker, and h k is the known height of the fiducial marker, and w p is the pixel width of the fiducial marker in the image space of the image, and h p is the pixel height of the fiducial marker in image space of the image.

[0029] In some non-limiting embodiments or aspects, the multiplication factor β is p and pixel height h p is determined from a plurality of pre-stored multiplication coefficients according to a ratio of

[0030] In some non-limiting embodiments or aspects, the known width w of the fiducial marker k and the known height of the fiducial marker, h k The ratio is 1:1.

[0031] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag.

[0032] According to some non-limiting embodiments or aspects, a method is provided that includes: acquiring, with at least one processor, an image including a fiducial marker captured by an image capture device, the fiducial marker having a known width and a known height; extracting, with the at least one processor, a pixel width and a pixel height of the fiducial marker in image space of the image; estimating, with the at least one processor, a yaw angle of the fiducial marker relative to the image capture device based on the pixel width, pixel height, and known width and known height; and providing, with the at least one processor, a yaw angle of the fiducial marker relative to the image capture device.

[0033] In certain non-limiting embodiments or aspects, the at least one processor estimates the yaw angle of the fiducial marker relative to the image capture device according to the following equation: Yaw angle = α*β where β is a multiplication factor and α is determined according to the following formula: α=((w k / h k )-(w p / h p ))*90 During the ceremony, w k is the known width of the fiducial marker, and h k is the known height of the fiducial marker, and w p is the pixel width of the fiducial marker in the image space of the image, and hp is the pixel height of the fiducial marker in image space of the image.

[0034] In some non-limiting embodiments or aspects, the multiplication factor β is p and pixel height h p is determined from a plurality of pre-stored multiplication coefficients according to a ratio of

[0035] In some non-limiting embodiments or aspects, the fiducial markers k The known width and known height of the fiducial marker, h k The ratio is 1:1.

[0036] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag.

[0037] According to some non-limiting embodiments or aspects, a system is provided that includes at least one processor coupled to a memory and programmed and / or configured to: acquire an image including a fiducial marker captured by an image capture device, the fiducial marker having a known width and a known height; extract a centroid of the fiducial marker in image space of the image; determine a position vector of the fiducial marker based on the centroid, the known width, and the known height; determine a tilt of the position vector of the fiducial marker relative to a coordinate base axis of the image; determine a roll angle of the fiducial marker relative to the image capture device based on the tilt of the position vector of the fiducial marker relative to the coordinate base axis of the image; and provide the roll angle of the fiducial marker relative to the image capture device.

[0038] In certain non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to estimate the tilt of the position vector of the fiducial marker relative to the coordinate base axes of the image according to the following equation: Slope = (y2-y1) / (x2-x1) where x1, y1 are the coordinates of the first point at the first end of the position vector, and x2, y2 are the coordinates of the second point at the second end of the position vector.

[0039] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to determine the roll angle of the fiducial marker relative to the image capture device based on the tilt of the position vector of the fiducial marker relative to the coordinate base axes of the image according to the following formula: Roll angle = tan -1 (Tilt).

[0040] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag.

[0041] In some non-limiting embodiments or aspects, the centroid of the fiducial marker in the image space of the image comprises the average of each point at each corner of AprilTag, and the position vector comprises the sum of the differences of the points on the axes of the image.

[0042] According to some non-limiting embodiments or aspects, a method is provided that includes: acquiring, using at least one processor, an image including a fiducial marker captured by an image capture device, wherein the fiducial marker has a known width and a known height; extracting, using at least one processor, a centroid of the fiducial marker in image space of the image; determining, using at least one processor, a position vector of the fiducial marker based on the centroid, the known width, and the known height; determining a tilt of the position vector of the fiducial marker relative to a coordinate base axis of the image; determining, using at least one processor, a roll angle of the fiducial marker relative to the image capture device based on the tilt of the position vector of the fiducial marker relative to the coordinate base axis of the image; and providing, using the at least one processor, the roll angle of the fiducial marker.

[0043] In certain non-limiting embodiments or aspects, the at least one processor estimates the tilt of the position vector of the fiducial marker relative to the coordinate base axes of the image according to the following equation: Slope = (y2-y1) / (x2-x1) where x1, y1 are the coordinates of the first point at the first end of the position vector, and x2, y2 are the coordinates of the second point at the second end of the position vector.

[0044] In some non-limiting embodiments or aspects, at least one processor determines the roll angle of the fiducial marker relative to the image capture device based on the tilt of the position vector of the fiducial marker relative to the coordinate base axes of the image according to the following formula: Roll angle = tan -1 (tilt)

[0045] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag.

[0046] In some non-limiting embodiments or aspects, the centroid of the fiducial marker in the image space of the image comprises the average of each point at each corner of AprilTag, and the position vector comprises the sum of the differences of the points on the axes of the image.

[0047] According to some non-limiting embodiments or aspects, a method includes: acquiring an image including a fiducial marker coupled to a memory and captured by an image capture device, the fiducial marker having a known width and a known height; extracting a pixel width and a pixel height of the fiducial marker in an image space of the image; determining a distance of the fiducial marker from the image capture device based on the pixel width and / or pixel height, the known width and / or known height, and one or more calibration parameters; estimating a yaw angle of the fiducial marker relative to the image capture device based on the pixel width, pixel height, known width, and known height; and extracting a pixel width and a pixel height of the fiducial marker in an image space of the image. a system including at least one processor programmed and / or configured to: extract a centroid of a fiducial marker; determine a position vector of the fiducial marker based on the centroid, a known width, and a known height; determine a slope of the position vector of the fiducial marker relative to a coordinate base axis of the image; determine a roll angle of the fiducial marker relative to an image capture device based on the slope of the position vector of the fiducial marker relative to the coordinate base axis of the image; and provide a distance of the fiducial marker from the image capture device, a yaw angle of the fiducial marker relative to the image capture device, and a roll angle of the fiducial marker relative to the image capture device.

[0048] In certain non-limiting embodiments or aspects, the at least one processor is further programmed and / or configured to: acquire a calibration image including the calibration marker, wherein the calibration image including the calibration marker is captured by an image capture device using the calibration marker positioned a known distance from the image capture device, the calibration marker having known dimensions; extract one or more features associated with the calibration marker in image space of the calibration image; and determine one or more calibration parameters based on the one or more features associated with the calibration marker in image space of the calibration image, the known distance of the calibration marker from the image capture device, and the known dimensions of the calibration marker.

[0049] In some non-limiting embodiments or aspects, at least one processor is programmed and / or configured to provide the distance of the fiducial marker from the image capture device, the yaw angle of the fiducial marker relative to the image capture device, and the roll angle of the fiducial marker relative to the image capture device by displaying the distance of the fiducial marker from the image capture device, the yaw angle of the fiducial marker relative to the image capture device, and the roll angle of the fiducial marker relative to the image capture device on a display simultaneously with an image including the fiducial marker.

[0050] In some non-limiting embodiments or aspects, at least one processor is programmed and / or configured to provide the distance of the fiducial marker from the image capture device, the yaw angle of the fiducial marker relative to the image capture device, and the roll angle of the fiducial marker relative to the image capture device by determining whether a medical device associated with the fiducial marker is connected to another medical device in the image using the distance of the fiducial marker from the image capture device, the yaw angle of the fiducial marker relative to the image capture device, and the roll angle of the fiducial marker relative to the image capture device.

[0051] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag.

[0052] According to some non-limiting embodiments or aspects, a method includes: acquiring, with at least one processor, an image including a fiducial marker captured by an image capture device, the fiducial marker having a known width and a known height; extracting, with at least one processor, a pixel width and a pixel height of the fiducial marker in image space of the image; determining, with at least one processor, a distance of the fiducial marker from the image capture device based on the pixel width and / or pixel height, the known width and / or known height, and one or more calibration parameters; estimating, with at least one processor, a yaw angle of the fiducial marker relative to the image capture device based on the pixel width, pixel height, known width, and known height; and, with at least one processor, using at least one processor to extract a centroid of the fiducial marker in image space of the image; using at least one processor to determine a position vector of the fiducial marker based on the centroid, the known width, and the known height; using at least one processor to determine a slope of the position vector of the fiducial marker with respect to coordinate cardinal axes of the image; using at least one processor to determine a roll angle of the fiducial marker relative to an image capture device based on the slope of the position vector of the fiducial marker with respect to the coordinate cardinal axes of the image; and using at least one processor to provide a distance of the fiducial marker from the image capture device, a yaw angle of the fiducial marker with respect to the image capture device, and a roll angle of the fiducial marker with respect to the image capture device.

[0053] In certain non-limiting embodiments or aspects, the method further includes: acquiring, with at least one processor, a calibration image including the calibration marker, wherein the calibration image including the calibration marker is captured by an image capture device with the calibration marker located at a known distance from the image capture device, and the calibration marker has known dimensions; extracting, with the at least one processor, one or more features associated with the calibration marker in image space of the calibration image; and determining, with the at least one processor, one or more calibration parameters based on the one or more features associated with the calibration marker in image space of the calibration image, the known distance of the calibration marker from the image capture device, and the known dimensions of the calibration marker.

[0054] In some non-limiting embodiments or aspects, at least one processor provides the distance of the fiducial marker from the image capture device, the yaw angle of the fiducial marker relative to the image capture device, and the roll angle of the fiducial marker relative to the image capture device by displaying the distance of the fiducial marker from the image capture device, the yaw angle of the fiducial marker relative to the image capture device, and the roll angle of the fiducial marker relative to the image capture device on a display simultaneously with an image including the fiducial marker.

[0055] In some non-limiting embodiments or aspects, at least one processor provides the distance of the fiducial marker from the image capture device, the yaw angle of the fiducial marker relative to the image capture device, and the roll angle of the fiducial marker relative to the image capture device by determining whether a medical device associated with the fiducial marker in the image is connected to another medical device using the distance of the fiducial marker from the image capture device, the yaw angle of the fiducial marker relative to the image capture device, and the roll angle of the fiducial marker relative to the image capture device.

[0056] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag. [Brief explanation of the drawings]

[0057] Additional advantages and details are explained in more detail below with reference to exemplary embodiments shown in the accompanying schematic drawings.

[0058] [Figure 1A] FIG. 1 is a diagram of a non-limiting embodiment or aspect of an environment in which the systems, devices, products, apparatus, and / or methods described herein may be implemented. [Figure 1B] FIG. 1 is a diagram of a non-limiting embodiment or aspect of an environment in which the systems, devices, products, apparatus, and / or methods described herein may be implemented. [Figure 2] FIG. 1C is a diagram of a non-limiting embodiment or aspect of one or more devices and / or components of one or more systems of FIGS. 1A and 1B. [Figure 3A] FIG. 1 is a perspective view of an implementation of a non-limiting embodiment or aspect of a medical device. [Figure 3B] 3B is a perspective view of an implementation of a non-limiting embodiment or aspect of the medical device of FIG. 3A connected together. [Figure 4]1 is a flowchart of a non-limiting embodiment or aspect of a process for vascular access management. [Figure 5] 1 shows an exemplary image of a catheter insertion site containing multiple medical devices. [Figure 6] 1 illustrates an implementation of a non-limiting embodiment or aspect of a fiducial marker. [Figure 7] 1 is a perspective view of an exemplary image of a catheter insertion site on a patient. [Figure 8] 1 illustrates exemplary parameters used in implementing non-limiting embodiments or aspects of a process for vascular access management. [Figure 9] 1 is a chart of exemplary unlikely but possible connections between medical devices. [Figure 10] 1 illustrates an implementation of a non-limiting embodiment or aspect of an IV line representation. [Figure 11A] 1 illustrates an exemplary catheter tree construction sequence of a process for vascular access management, according to a non-limiting embodiment or aspect. [Figure 11B] 1 illustrates an exemplary catheter tree. [Figure 12A] 1 is a flowchart of a non-limiting embodiment or aspect of a process for estimating object distance from an image capture device. [Figure 12B] 1 is a flowchart of a non-limiting embodiment or aspect of a process for estimating object distance from an image capture device. [Figure 13] 1 is an annotated image of an exemplary AprilTag. [Figure 14] 10 illustrates an implementation of a non-limiting embodiment or aspect of a display that includes the distance of a medical device from an image capture device. [Figure 15] 1 is a flowchart of a non-limiting embodiment or aspect of a process for estimating object distance from an image capture device. [Figure 16] 1 illustrates an implementation of a non-limiting embodiment or aspect of a multi-view camera setup. [Figure 17]1 is a flowchart of a non-limiting embodiment or aspect of a process for estimating the yaw angle of an object from an image capture device. [Figure 18] 1 is a flowchart of a non-limiting embodiment or aspect of a process for estimating object roll angle from an image capture device. [Figure 19A] Annotated image of squares or rectangles representing the areas between the corners of AprilTag ordered clockwise. [Figure 19B] 19B is a graph of the points forming a square or rectangle representing the area between the corners of AprilTag of FIG. 19A. [Figure 19C] 19C is a graph of the orientation axis of AprilTag of FIGS. 19A and 19B. [Figure 20A] Indicates the coordinate base of the image containing AprilTags. [Figure 20B] 10 shows exemplary x and y vectors of AprilTag relative to the coordinate axes of the image. DETAILED DESCRIPTION OF THE INVENTION

[0059] It should be understood that the present disclosure may contemplate various alternative modifications and step sequences unless expressly specified to the contrary. It should also be understood that the specific devices and processes illustrated in the accompanying drawings, and described in the following specification, are merely exemplary and non-limiting embodiments or aspects. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered limiting.

[0060] For purposes of the following description, the terms "end," "upper," "lower," "right," "left," "vertical," "horizontal," "top," "bottom," "lateral," "longitudinal," and derivatives thereof, refer to the embodiments or aspects as they are oriented in the drawings. However, it should be understood that the embodiments or aspects may assume various alternative variations and step sequences unless expressly specified to the contrary. It should also be understood that the specific devices and processes illustrated in the accompanying drawings and described in the following specification are merely exemplary and non-limiting embodiments or aspects. Accordingly, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein should not be considered limiting unless otherwise indicated.

[0061] As used herein, aspects, components, elements, structures, acts, steps, functions, instructions, and / or the like should not be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more" and "at least one." Furthermore, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with "one or more" or "at least one." Where only one item is intended, the term "one" or similar language is used. Also, as used herein, the terms "has," "have," "having," etc. are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly specified otherwise.

[0062] As used herein, the terms “communication” and “communicate” may refer to the reception, receipt, transmission, transfer, provision, etc. of information (e.g., data, signals, messages, instructions, commands, and / or the like). One unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) communicating with another unit means that the one unit can directly or indirectly receive information from and / or transmit information to the other unit. This may refer to a direct or indirect connection that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even if the information being transmitted is modified, processed, relayed, and / or routed between the first and second units. For example, a first unit may communicate with a second unit even if the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may communicate with a second unit if at least one intermediary unit (e.g., a third unit located between the first and second units) processes information received from the first unit and communicates the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet and / or the like) containing data. It will be understood that numerous other arrangements are possible.

[0063] As used herein, the term “computing device” may refer to one or more electronic devices configured to communicate directly or indirectly with one or more networks. A computing device may be a mobile device, desktop computer, server, or the like. Furthermore, the term “computer” may refer to any computing device that includes the components necessary to receive, process, and output data, typically including a display, processor, memory, input devices, and a network interface. A “computing system” may include one or more computing devices or computers. An “application” or “application program interface” (API) refers to computer code or other data sorted on a computer-readable medium that can be executed by a processor to facilitate interaction between software components for receiving data from a client, such as a client-side front-end and / or a server-side back-end. An “interface” refers to one or more generated displays, such as a graphical user interface (GUI), with which a user can interact directly or indirectly (e.g., via a keyboard, mouse, touchscreen, etc.). Furthermore, two or more computers, such as, for example, servers, or other computerized devices communicating directly or indirectly within a network environment may constitute a "system" or a "computing system."

[0064] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, software, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limiting of the implementation. Thus, the operation and workings of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0065] Some non-limiting embodiments or aspects are described herein in relation to thresholds. As used herein, meeting a threshold may mean that a value is greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, less than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0066] 1A, which is a diagram of an example environment 100 in which the devices, systems, methods, and / or products described herein may be implemented. As shown in FIG. 1A, environment 100 includes user devices 102, a management system 104, and / or a communication network 106. The systems and / or devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0067] 1B, which is a diagram of a non-limiting embodiment or aspect of an implementation of an environment 100 in which the systems, devices, products, apparatus, and / or methods described herein may be implemented. For example, as shown in FIG. 1B, the environment 100 may include a hospital room including a patient, one or more medical devices 108, one or more fiducial markers 110 associated with the one or more medical devices 108, and / or a caregiver (e.g., a nurse, etc.).

[0068] The user device 102 may include one or more devices capable of receiving information and / or data from the management system 104 (e.g., via the communications network 106, etc.) and / or communicating information and / or data to the management system 104 (e.g., via the communications network 106, etc.). For example, the user device 102 may include one or more computing systems including one or more processors (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, one or more tablet computers, etc.). In some non-limiting embodiments or aspects, the user device 102 may include a tablet computer or a mobile computing device, such as an Apple® iPad, an Apple® iPhone, an Android® tablet, an Android® phone, etc.

[0069] The user device 102 may include one or more image capture devices (e.g., one or more cameras, one or more sensors, etc.) configured to capture one or more images of an environment surrounding the one or more image capture devices (e.g., environment 100, etc.). For example, the user device 102 may include one or more image capture devices configured to capture one or more images of one or more medical devices 108, one or more fiducial markers 110 associated with the one or more medical devices 108, and / or a patient. As an example, the user device 102 may include a monocular camera.

[0070] The management system 104 may include one or more devices capable of receiving information and / or data from the user devices 102 (e.g., via the communications network 106, etc.) and / or communicating information and / or data to the user devices 102 (e.g., via the communications network 106, etc.). For example, the management system 104 may include one or more computing systems including one or more processors (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, etc.). In some non-limiting embodiments or aspects, the management system 104 includes and / or is accessible via a hospital nursing station or terminal. For example, the management system 104 may provide bedside nurse support, nursing station manager support, retrospective reporting for nursing management, etc.

[0071] The communication network 106 may include one or more wired and / or wireless networks. For example, the communication network 106 may include a cellular network (e.g., a long-term evolution (LTE) network, a third-generation (3G) network, a fourth-generation (4G) network, a fifth-generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-optic-based network, a cloud computing network, and / or the like, and / or a combination of these or other types of networks.

[0072] The number and arrangement of systems and devices shown in Figures 1A and 1B are provided as an example. There may be additional, fewer, different, or differently arranged systems and / or devices. Furthermore, two or more systems or devices shown in Figures 1A and 1B may be implemented within a single system or device, or a single system or device shown in Figures 1A and 1B may be implemented as multiple distributed systems or devices. Additionally or alternatively, a set of systems or devices (e.g., one or more systems, one or more devices, etc.) of environment 100 may perform one or more functions described as being performed by another set of systems or another set of devices of environment 100.

[0073] 2, which is a diagram of example components of a device 200. The device 200 may correspond to a user device 102 (e.g., one or more devices of a system of user devices 102, etc.) and / or one or more devices of the management system 104. In some non-limiting embodiments or aspects, the user device 102 (e.g., one or more devices of a system of user devices 102, etc.) and / or one or more devices of the management system 104 may include at least one device 200 and / or at least one component of the device 200. As shown in FIG. 2, the device 200 may include a bus 202, a processor 204, a memory 206, a storage component 208, an input component 210, an output component 212, and a communication interface 214.

[0074] Bus 202 may include components that enable communication between components of device 200. In some non-limiting embodiments or aspects, processor 204 may be implemented in hardware, software, or a combination of hardware and software. For example, processor 204 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component (e.g., a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.) that can be programmed to perform functions. Memory 206 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 204.

[0075] Storage component 208 may store information and / or software related to the operation and use of device 200. For example, storage component 208 may include a hard disk (e.g., a magnetic disk, optical disk, magneto-optical disk, solid-state disk, etc.), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of computer-readable medium, along with a corresponding drive.

[0076] Input components 210 may include components that enable device 200 to receive information, for example, via user input (e.g., a touchscreen display, a keyboard, a keypad, a mouse, buttons, switches, a microphone, etc.). Additionally or alternatively, input components 210 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, an image capture device, etc.). Output components 212 may include components that provide output information from device 200 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.).

[0077] Communications interface 214 may include transceiver-like components (e.g., a walkie-talkie, a separate receiver and transmitter, etc.) that enable device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communications interface 214 may enable device 200 to receive information from other devices and / or provide information to another device. For example, communications interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, and / or the like.

[0078] The device 200 may perform one or more processes described herein. The device 200 may perform these processes based on the processor 204 executing software instructions stored by a computer-readable medium, such as the memory 206 and / or the storage component 208. A computer-readable medium (e.g., a non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A memory device includes a memory space located within a single physical storage device or a memory space spanning multiple physical storage devices.

[0079] Software instructions may be loaded into memory 206 and / or storage component 208 from another computer-readable medium or from another device via communications interface 214. When executed, the software instructions stored in memory 206 and / or storage component 208 may cause processor 204 to perform one or more processes described herein. Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments or aspects described herein are not limited to any specific combination of hardware circuitry and software.

[0080] The memory 206 and / or the storage component 208 may include a data storage or one or more data structures (e.g., a database, etc.). The device 200 may be able to receive information from, store information in, transmit information to, or retrieve information stored in the data storage or one or more data structures in the memory 206 and / or the storage component 208.

[0081] The number and arrangement of components shown in Figure 2 are provided as an example. In some non-limiting embodiments or aspects, device 200 may include additional, fewer, different, or differently arranged components than those shown in Figure 2. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0082] Referring now to Figures 3A and 3B, Figure 3A is a perspective view of an implementation of a non-limiting embodiment or aspect of a medical device, and Figure 3B is a perspective view of an implementation of a non-limiting embodiment or aspect of the medical device of Figure 3A connected together.

[0083] The medical device 108 may include at least one of a peripheral IV catheter (PIVC), a peripherally inserted central catheter (PICC), a midline catheter, a central venous catheter (CVC), a needleless connector, a catheter dressing, a catheter stabilizing device, an antiseptic cap, an antiseptic swab or wipe, an IV tubing set, an extension set, a Y-connector, a stopcock, an infusion pump, a flush syringe, a medication delivery syringe, an IV fluid bag, a lumen adapter (e.g., the number of lumen adapters associated with a catheter may indicate the number of lumens included in the catheter, etc.), or any combination thereof.

[0084] Fiducial markers 110 (e.g., tags, labels, codes, etc.) may be associated with (e.g., removably attached, permanently attached, integrated, implemented into) medical devices 108. In some non-limiting embodiments or aspects, each medical device 108 in environment 100 may be associated with a fiducial marker 110. In some non-limiting embodiments or aspects, only a portion of the medical devices 108 in environment 100 may be associated with fiducial markers 110. In some non-limiting embodiments or aspects, none of the medical devices 108 in environment 100 may be associated with a fiducial marker 110.

[0085] A fiducial marker 110 may encapsulate an identifier associated with the type of medical device 108 associated with the fiducial marker 110 and / or may uniquely identify the medical device 108 associated with the fiducial marker 110 from other medical devices. For example, a fiducial marker 110 may encapsulate an identifier associated with at least one of the following types of medical devices: peripheral IV catheter (PIVC), peripherally inserted central catheter (PICC), midline catheter, central venous catheter (CVC), needleless connector, antiseptic cap, antiseptic swab or wipe, IV tubing set, extension set, Y-connector, stopcock, infusion pump, flush syringe, medication delivery syringe, IV fluid bag, or any combination thereof, and / or may uniquely identify a medical device 108 (e.g., a first needleless connector, etc.) from other medical devices (e.g., a second needleless connector, etc.) that include an identifier associated with the same type of medical device.

[0086] Fiducial marker 110 may encapsulate pose information associated with the 3D position of fiducial marker 110. For example, fiducial marker 110 may include markings that, when captured in an image, allow for calculating the precise 3D position of the fiducial marker relative to the image capture device that captured the image (e.g., the x, y, z coordinate position of the fiducial marker, etc.) and / or the precise 2D position of the fiducial marker in the image itself (e.g., the x, y coordinate position of the fiducial marker within the image, etc.).

[0087] In some non-limiting embodiments or aspects, the fiducial marker 110 may include an AprilTag. For example, the fiducial marker 110 may include a custom tag 48h12 type AprilTag V3, which allows AprilTag V3 detection to be used to determine a unique ID, which may indicate the type of fiducial marker 110 (e.g., leading digits, etc.) and / or the unique serial number (e.g., trailing digits, etc.) of that particular medical device 108 within the field of view (FOV) of the image capture device, and / or the location of the fiducial marker 110 (e.g., x, y, and z coordinates, Z-axis, Y-axis, and X-axis direction vectors, etc.). However, non-limiting embodiments or aspects are not so limited, and the fiducial marker 110 may include a QR code, a barcode (e.g., a 1D barcode, a 2D barcode, etc.), an Aztec code, a Data Matrix code, an ArUco marker, a colored pattern, a reflective pattern, a fluorescent pattern, a predetermined shape and / or color (e.g., a red pentagon, a blue hexagon, etc.), an LED pattern, a hologram, etc., encapsulating an identifier associated with the type of medical device 108 associated with the fiducial marker 110, uniquely identifying the medical device 108 associated with the fiducial marker 110 from other medical devices, and / or encapsulating pose information associated with the 3D position of the fiducial marker 110.

[0088] In some non-limiting embodiments or aspects, the fiducial marker 110 may include a color calibration region located adjacent to the variable color region to calibrate color over a wider range of lighting conditions. For example, in the case of a 2×2 grid, cell (1,1) in the upper left corner of the guard may include a predetermined and / or standard calibration color region (e.g., neutral gray, etc.), and the user device 102 and / or management system 104 may use the predetermined and / or standard calibration color region to calibrate colors in images used to detect or determine the fiducial marker 110 in those images and / or to detect or determine color changes in patient tissue (e.g., patient tissue adjacent an insertion site, etc.) in those images. In such an example, the user device 102 and / or management system 104 may use predetermined and / or standard calibration color regions to orient the fiducial marker 110 to appropriately rotate and decode the colors within the fiducial marker 110 to decode the identifier encapsulated by the fiducial marker 110 and / or determine how to track the fiducial marker 110 within the environment 100.

[0089] 3A and 3B, the fiducial markers 110 may be symmetrically arranged in a ring about the axis of the medical device 108 associated with them, which may allow at least one fiducial marker 110 to be presented in the FOV of the image capture device regardless of the orientation of the medical device 108. The fiducial marker 110 may be clocked so that the orientation of the marker (e.g., as indicated by its pose information, etc.) is aligned with the proximal or distal direction of fluid flow through the medical device 108 associated with that fiducial marker. The fiducial marker 110 may be rigidly fixed to the medical device 108 (e.g., rigidly fixed to a rigid portion of the medical device 108 and / or a catheter tree including the medical device 108, etc.) so that the fiducial marker 110 cannot translate and / or rotate along the medical device 108, which may reduce movement and / or change in distance of the fiducial marker relative to other fiducial markers and / or medical devices.

[0090] The fiducial markers 110 may be located at or directly adjacent to the connection points or ports of each medical device 108 such that the fiducial markers 110 on the connected medical devices 108 are collinear (e.g., parallel, etc.). For example, the fiducial markers 110 on connected medical devices 108 that are collinear (e.g., parallel, etc.) in this manner may also be adjacent or may be separated by a known distance.

[0091] A single medical device 108 may include one or more sets of fiducial markers 110. For example, as shown in Figures 3A and 3B, the cap on the far right in each of these figures includes a single set of fiducial markers 110 (e.g., each fiducial marker 110 in the set is identical and / or encapsulates the same information, etc.), while the tube to the left of the cap includes two sets of fiducial markers 110 at respective connection points of the tube, separated by the tube. In such an example, collinearity between the fiducial markers 110 at each end of the tube may not be guaranteed, and the connection between the fiducial markers 110 at each end of the tube may be established via a predefined scheme (e.g., each fiducial marker 110 in each set of fiducial markers 110 on the same medical device 108 has the same value or different but adjacent values, etc.). Note that the spacing between connected medical devices 108 may vary (e.g., as shown on the left side of FIG. 3B); however, this spacing may be deterministic and known by the user device 102 and / or management system 104 for each possible connection between the medical devices.

[0092] 4, which is a flowchart of a non-limiting embodiment or aspect of a process 400 for vascular access management. In some non-limiting embodiments or aspects, one or more of the steps of process 400 are performed (e.g., completely, partially, etc.) by user device 102 (e.g., one or more devices of a system of user device 102, etc.). In some non-limiting embodiments or aspects, one or more of the steps of process 400 may be performed (e.g., completely, partially, etc.) by a device or group of devices separate from user device 102, such as management system 104 (e.g., one or more devices of management system 104, etc.).

[0093] 4, in step 402, process 400 includes acquiring an image. For example, the user device 102 may acquire images (e.g., a single image, multiple images, a series of images, etc.) of multiple medical devices 108 captured by an image capture device. As an example, the image capture device of the user device 102 may capture images of multiple medical devices 108. In such an example, a nurse may use the user device 102 to acquire one or more images of a patient's catheter site and / or an infusion pump connected to the catheter site. For example, FIG. 5 shows an exemplary image of a catheter insertion site including multiple medical devices.

[0094] In some non-limiting embodiments or aspects, the image may include a series of images. For example, the image capture device of the user device 102 may capture a series of images using a burst capture technique (e.g., “burst mode,” continuous shooting mode, etc.), which may enable the user device 102 to create a second layer of likelihood for determining the probability that pairs of medical devices 108 are connected as described in more detail herein, thereby refining motion artifact, angle, distance, and / or missed fiducial marker detection. As an example, the image capture device of the user device 102 may capture a series of images as a live video feed to identify pairs of medical devices 108 that are connected to each other (e.g., identify catheter tree components, generate a catheter tree, etc.) as the live video feed is captured.

[0095] 4, in step 404, process 400 includes determining position information. For example, the user device 102 may determine, based on the image, position information associated with the 3D positions of the multiple medical devices 108 relative to the image capture device and / or the 2D positions of the inner devices 108 in the image itself. In such an example, determining the position information associated with the 3D positions of the multiple medical devices 108 relative to the image capture device and / or the 2D positions of the multiple medical devices 108 in the image itself may further include determining, based on the image, the type of each medical device of the multiple medical devices 108.

[0096] In some non-limiting embodiments or aspects, a first group of medical devices of the plurality of medical devices 108 are associated with a plurality of fiducial markers 110. The plurality of fiducial markers 110 encapsulate a plurality of identifiers associated with the first group of medical devices and may take pose information associated with the 3D and / or 2D positions of the plurality of fiducial markers 110. In such examples, the user device 102 may determine, based on the image, position information associated with the 3D positions of the first group of medical devices relative to the image capture device and / or the 2D positions of the plurality of medical devices 108 in the image itself by determining or identifying the plurality of fiducial markers 110 associated with the first group of medical devices and the pose information associated with the 3D and / or 2D positions of the plurality of fiducial markers 110. For example, referring again to FIG. 6, the multiple fiducial markers 110 may include multiple AprilTags, and the user device 102 may process the image using AprilTag detection software to determine the type of medical device 108 associated with the fiducial markers 110 and / or the unique serial number of the particular medical device 108, calculate the precise 3D position, orientation, and / or identity of the multiple fiducial markers 110 relative to the image capture device that captured the image, and / or calculate the precise 2D position of the multiple fiducial markers 110 in the image itself. As an example, for each medical device 108 in the first group of medical devices, position information associated with the 3D position of that medical device 108 relative to the image capture device may be determined as the 3D position of the fiducial marker 110 associated with that medical device 108 relative to the image capture device, and / or position information associated with the 2D position of that medical device 108 in the image itself may be determined as the 2D position of the fiducial marker 110 associated with that medical device 108. In such an example, for each medical device 108 in the first group of medical devices, the 3D position of the fiducial marker 110 associated with that medical device relative to the image capture device may include the x, y, and z coordinates of the fiducial marker 110 and / or the direction vector of the Z-axis, Y-axis, and X-axis of the fiducial marker 110. In such an example, for each medical device 108 in the first group of medical devices, the 2D position of the fiducial marker 110 in the image itself may include the x, y coordinates of the fiducial marker 110 in the image and / or the y-axis and x-axis direction vectors of the fiducial marker.

[0097] In some non-limiting embodiments or aspects, the medical devices of the second group of the plurality of medical devices are not associated with a fiducial marker. In such examples, the user device 102 may determine or identify, based on the image, for each medical device of the second medical devices, the type of medical device, the 3D position of the medical device relative to the image capture device, and / or the 2D position of the medical device in the image itself, thereby determining position information associated with the 3D position of the second medical device relative to the image capture device and / or the 2D position of the second medical device in the image itself using one or more existing object detection technologies. For example, a medical device 108 without a fiducial marker or identifier tag can be identified by the user device 102 processing the image using one or more object detection techniques (e.g., deep learning techniques, image processing techniques, image segmentation techniques, etc.) to identify or determine location information associated with the image and the 3D position of the identified medical device 108 relative to the image capture device (e.g., including the x-, y-, and z-coordinates of the medical device and the direction vector of the Z-, Y-, and X-axis of the medical device 108, etc.) and / or the 2D position of the identified medical device in the image itself. For example, deep learning techniques may include bounding box techniques that generate a box label for an object of interest (e.g., such as the medical device 108) in the image, image masking techniques (e.g., masked FRCNN (RCNN or CNN)) that capture the specific shape of an object (e.g., such as the medical device 108) in the image, trained neural networks that identify objects (e.g., such as the medical device 108) in the image, classifiers that classify the identified object into an object class or type, etc. By way of example, image processing techniques may include cross-correlation image processing techniques, image contrast techniques, binary or color filtering techniques, etc. By way of example, different catheter lumens may include unique colors that may be used by image processing to identify the type of catheter.

[0098] In some non-limiting embodiments or aspects, the user device 102 may process image data using stereoscopic imaging and / or shadow distance techniques to determine object data including distances from the image capture device to and / or between detected objects, and / or the user device 102 may use multiple cameras, laser focus technology, LiDAR sensors, and / or camera physical zoom-in functionality to determine object data including distances from the image capture device to and / or between detected objects. In some non-limiting embodiments or aspects, the user device 102 may use a 3D optical profiler to obtain image data and / or object data including 3D profiles of objects.

[0099] For example, the image capture device may include a stereo camera, and / or position information associated with the 3D positions of the multiple medical devices relative to the image capture device may be determined using a structure from motion (SfM) algorithm. As an example, the user device 102 may include a stereo camera setup available on many mobile devices, such as an Apple® iPad, an Apple® iPhone, an Android® tablet, an Android® phone, etc. The user device 102 may process images from the stereo camera using an SfM algorithm to extract 3D information, which may enhance object feature recognition of the fiducial markers 110 of the first group of medical devices and / or the fiducial markers 110 of the second group of medical devices without a fiducial marker. As one example, SfM processing may improve the extraction of 3D features from a second group of medical devices without fiducial markers, which may improve the likelihood of image feature accumulation, for example, by using a burst image capture / video mode that captures images at the appropriate orientation / alignment (e.g., pan / tilt, etc.) based on the setting or position of the medical device 108 and / or its anatomical location on the patient's body, and a catheter tree is constructed by starting from the dressing tag and connecting the center point or centroid of the medical device 108 detected using object recognition techniques to other medical devices 108 with or without fiducial markers 110. As another example, these 3D features extracted using SfM processing may be utilized in calculating the coplanarity of the medical device 108 (e.g., generating a catheter tree), which may provide additional benefits to multi-lumen and multi-tube catheter setups by reducing misconnections that can occur in 2D, single-plane space due to false collinearity.

[0100] For example, the image capture device may include a LiDAR system, and the image may include a LiDAR point cloud. As an example, the user device 102 may include a mini LiDAR system available on many mobile devices, such as an Apple® iPad, an Apple® iPhone, an Android® tablet, or an Android® phone. In such an example, the use of LiDAR imagery may improve the accuracy of 3D feature detection because the LiDAR imagery directly provides 3D world information as a point cloud, which may accelerate 3D data collection with reduced or minimal protocol compared to a stereo setup. For example, the user device 102 may use existing image registration and / or transformation techniques to overlap the 3D LiDAR point cloud with 2D object information from camera images, such as color, texture, etc., to detect 3D features for improved catheter tree generation and connectivity accuracy, which may also improve augmented reality generation and environment reconstruction, as well as guide catheter tree generation and connectivity decisions.

[0101] 7, which is a perspective view of an exemplary catheter insertion site on a patient, due to the positioning of the AprilTag and different tubing, the fiducial markers 110 and / or medical devices 108 appear to be co-linear in 2D due to their proximity, creating a high likelihood of tree generation mismatches (e.g., erroneous determination of connections between medical devices, etc.). The user device 102 may use the stereo image / SfM and / or LiDAR image-based approaches described above to determine 3D features of the fiducial markers 110 and / or medical devices 108 to improve co-planarity information and / or provide more accurate catheter tree generation (e.g., more accurate determination of connections between devices, etc.).

[0102] Further details regarding non-limiting embodiments or aspects of step 404 of process 400 are provided below with respect to FIGS. 12A and 12B, 15, 17, and 18.

[0103] 4, in step 406, the process 400 includes determining pairs of medical devices that are connected to each other. For example, the user device 102 may determine pairs of medical devices of the plurality of medical devices 108 that are connected to each other based on location information and / or the type of the plurality of medical devices 108. As an example, for each pair of medical devices of the plurality of medical devices 108, the user device 102 may determine a probability that the pair of medical devices is connected to each other based on 3D location information associated with the pair of medical devices, 2D location information associated with the pair of medical devices, and / or the type of the pair of medical devices. In such an example, 、 For each medical device of the plurality of medical devices 108, it may be determined that the medical device is connected to another medical device in a pair of medical devices that includes the medical device associated with the highest probability of the pair of medical devices that includes the medical device.

[0104] For example, for each pair of medical devices of the plurality of medical devices, the user device 102 may determine the following parameters based on the location information associated with the pair of medical devices: the distance between the center points associated with the pair of medical devices, the angular difference between the orientations of the pair of medical devices, and / or the angle from collinearity of the pair of medical devices, and the probability that the pair of medical devices are connected may be determined based on the distance between the center points associated with the pair of medical devices, the angular difference between the orientations of the pair of medical devices, and / or the angle from collinearity of the pair of medical devices. As an example, and referring to FIG. 8, for each pair of fiducial markers of the plurality of fiducial markers 110a, 110b, and 110c, the user device 102 may determine the following parameters based on the 3D position information and / or 2D position information associated with that pair of fiducial markers: the distance between the center points of that pair of fiducial markers (e.g., proximity, etc.), the angular difference between the orientations of that pair of fiducial markers (e.g., orientation, etc.), and / or the angle from collinearity of that pair of fiducial markers (e.g., collinearity, etc.), and the probability that the pair of medical devices associated with that pair of fiducial markers are connected may be determined based on the determined proximity, orientation, and / or collinearity (e.g., angular difference in orientation, angle created by tag orientation and connection vector, etc.). While shown in FIG. 8 as being determined between pairs of fiducial markers 110, non-limiting embodiments or aspects are not so limited, and proximity, orientation, and / or collinearity between medical devices may be determined between pairs of medical devices without fiducial markers, and / or between pairs of medical devices including a single medical device associated with a fiducial marker and a single medical device without a fiducial marker.

[0105] In some non-limiting embodiments or aspects, the user device 102 may use probability-based tree-building logic or rule sets (e.g., predetermined rule sets, etc.) to determine pairs of medical devices 108 that are connected to each other. As an example, the user device 102 may use the proximity, orientation, and / or collinearity parameters described above (and / or one or more additional parameters, such as the X-axis and Y-axis distance between the pair of medical devices, the depth difference from the image capture device between the pair of medical devices, etc.) to determine the probability of each medical device 108 relative to each of the other medical devices, and when applying the probability-based tree-building logic or rule sets to determine pairs of medical devices 108 that are connected to each other, the user device 102 may assign a predetermined weight to each parameter toward a total probability that the pair of medical devices are connected to each other (e.g., the sum of the weights of each parameter may be 1, etc.). In such an example, the user device 102 may process pairs of medical devices 108 by using dressing tags and / or lumen adapters as starting points or anchors for generating a catheter tree or representing an IV line, and for each medical device, it may be determined that the medical device is connected to the other medical device in the pair of medical devices with which the medical device has the highest connection probability.

[0106] In such an example, the user device 102 may identify a set of medical devices of the plurality of medical devices 108 associated with a preferred IV line architecture based on the type of each medical device of the plurality of medical devices. For pairs of medical devices included in the set of medical devices associated with a preferred IV line architecture, the user device 102 may adjust the weighting used to determine whether the pair of medical devices is connected to one another (e.g., a known preferred architecture may receive a higher weighting in the connection determination logic, etc.). As an example, if the user device 102 identifies each of the medical devices or disposables necessary to create a preferred architecture connection for each IV line in the image, the user device 102 may determine the connection between the pair of medical devices by giving a higher probability of a preferred connection including the pair of medical devices connected in the preferred architecture (e.g., a lumen adapter connected to IV tubing, etc.). For example, when the medical devices are determined to be within a threshold proximity or distance of each other, the proximity or distance parameters of the pair of preferred architecture pieces may be weighted more strongly compared to other parameters (e.g., orientation, collinearity, etc.). In some non-limiting embodiments or aspects, in response to determining an unlikely parameter associated with a pair of medical devices (e.g., a bandage that is a threshold distance, such as 30 cm, from a luminal adapter), the user device 102 may prompt the user to re-acquire the image and / or notify the user of the unlikely parameter (e.g., asking whether multiple catheter components are present in a single image).

[0107] In some non-limiting embodiments or aspects, the user device 102 may adjust the weights used to determine whether a pair of medical devices is connected to one another based on a determination that a medical device of the pair of medical devices is connected to another medical device (e.g., another medical device of a preferred IV line architecture that includes the pair of medical devices). For example, for a pair of medical devices that includes a first medical device and a second medical device, if the user device 102 determines that the first medical device is already connected to a third medical device of a preferred IV line architecture that includes the first medical device, the user device 102 may increase the probability that the first medical device and the second medical device are connected because such a connection completes a preferred IV line architecture.

[0108] In such an example, if the user device 102 identifies only some of the medical devices or disposables necessary to create a preferred architecture connection for each IV line in the image, the user device 102 may determine connections between pairs of medical devices by giving a higher probability of a preferred connection that includes pairs of medical devices that are connected in the preferred architecture and are of a predetermined type of medical device (e.g., a device that is not a cap, etc.). For example, if a triple-lumen catheter is expected for the preferred architecture and only a single IV tubing is identified in the image, the user device 102 may automatically expect a cap on the other lumen or lumen adapter; if the user device 102 does not identify a cap in the image, the user device 102 may prompt the user to reacquire the image and / or provide a notification to the user requesting clarification.

[0109] In such an example, the user device 102 may determine, based on the type of each medical device of the plurality of medical devices, that a medical device of the plurality of medical devices 108 is not associated with a preferred IV line architecture. For example, in response to determining that a medical device of the plurality of medical devices 108 is not associated with a preferred IV line architecture, the user device 102 may prompt the user to acquire another image.

[0110] In such an example, the user device 102 may determine, based on the type of each medical device in the plurality of medical devices, that the number of medical devices in the plurality of medical devices 108 is greater than the number of medical devices associated with the preferred IV line architecture. For example, if the number of identified medical devices is greater than the number required to create a preferred architecture connection for each lumen or IV line, it means that there may be a daisy chain or multiple tagged components in the image. As an example, if the user device 102 identifies more IV tubing sets than catheter lumens, the user device 102 may identify the IV tubing sets in the probability-based tree-building logic. Y More weight may be given to multiple tubes to port connections. As another example, if the user device 102 identifies a threshold number of caps, the user device 102 may give more weight to a proximity parameter for detecting the distance of the medical device from an open port.

[0111] Referring now to FIG. 9, FIG. 9 is a chart of exemplary unlikely but possible connections between medical devices. For example, the probability of a connection between medical devices of the same class or type may be zero (tubing may be an exception, but tubing may have a secondary tubing Y-port tag). For example, an unlikely connection between medical devices 108 may be a connection between medical devices that is unexpected and / or serves no purpose. However, such a connection may not be erroneous and / or potentially initiated by some user. Loose dressings and / or fiducial markers 110 for dressings may be left around a catheter insertion site and appear near a tubing set or other identified medical device; the user device 102 may adjust the proximity or distance from the catheter lumen adapter and the tag vector angle from each other, or give priority to the dressing tags, in its probability-based tree-building logic to determine that these loose fiducial markers or tags are not attached to any of the medical devices. A loose cap may float above a patient near a bed or other tagged object, and the user device 102 may automatically determine that the cap is not connected to another medical device if the distance between the cap and the other medical device meets a predetermined threshold distance. A loose tubing set on a bed or patient may be handled in the same or similar manner as a loose cap. While it is unlikely that lumen adapters can be connected to each other, the user device 102 may determine whether lumen adapters are connected to each other based on the distance and / or collinearity of the two adapters.

[0112] In some non-limiting embodiments or aspects, the user device 102 may process the location information and / or the types of the multiple medical devices 108 with a machine learning model to determine the probability that a pair of medical devices is connected. For example, the user device 102 may generate a predictive model (e.g., an estimator, classifier, predictive model, detector model, etc.) using machine learning techniques, including supervised and / or unsupervised techniques such as, for example, decision trees (e.g., gradient-boosted decision trees, random forests, etc.), logistic regression, artificial neural networks (e.g., convolutional neural networks, etc.), Bayesian statistics, learning automata, hidden Markov modeling, linear classifiers, quadratic classification, association rule learning, and / or the like. The predictive machine learning model may be trained to provide an output including a prediction of whether a pair of medical devices is connected to one another. In such an example, the prediction may include a probability (e.g., likelihood, etc.) that the pair of medical devices is connected to one another.

[0113] The user device 102 may generate a predictive model based on location information associated with each medical device and / or each type of medical device (e.g., training data, etc.). In some implementations, the predictive model is designed to receive as input location information associated with each medical device in a pair of medical devices (e.g., proximity between devices, orientation between devices, collinearity between devices, etc.) and provide as output a prediction (e.g., probability, likelihood, binary output, yes / no output, score, predicted score, classification, etc.) regarding whether the pair of medical devices are connected to each other. In some non-limiting embodiments or aspects, the user device 102 stores the predictive model (e.g., stores the model for later use). In some non-limiting embodiments or aspects, the user device 102 stores the initial predictive model in a data structure (e.g., a database, a linked list, a tree, etc.). In some non-limiting embodiments, the data structure is located within the user device 102 or external (e.g., remote) within the user device 102 (e.g., in the management system 104, etc.).

[0114] As shown in FIG. 4 , in step 408, process 400 includes generating a representation of at least one IV line. For example, user device 102 may generate a representation of at least one IV line including pairs of medical devices determined to be connected to each other based on pairs of medical devices determined to be connected to each other. By way of example, and referring to FIG. 10 , which illustrates an implementation of a non-limiting embodiment or aspect of a representation of an IV line (e.g., a catheter tree, etc.), user device 102 may automatically draw lines connecting medical devices (e.g., catheter tree components, etc.) individually for each IV line and / or display identification information associated with each IV line and / or individual medical devices 108 within each IV line. In such an example, user device 102 may automatically draw and display lines on an image and / or in a series of images, such as a live video feed of a medical device / catheter insertion site. For example, user device 102 may generate a digital representation of each IV line including pairs of medical devices within each IV line according to pairs of medical devices determined to be connected to each other. In such an example, the user device 102 may associate each IV line with a fluid source or pump of an infusion pump and monitor the flow of fluid in the IV line based at least in part on the representation of the fluid flow path. As an example, the user device 102 may control an audio and / or visual output device to output an audible and / or visual indication indicating the status of the IV line and / or the fluid flowing therethrough. For example, the user device 102 may generate a catheter tree or logical IV branch structure that is mapped onto a physical IV branch structure and includes unique node identifiers for each medical device in the physical IV branch structure, each connector or inlet / outlet point to the fluid flow path formed by the medical device, and / or each element of the medical device (e.g., a valve of the medical device) associated with an action that can affect the fluid flow path.

[0115] In some non-limiting embodiments or aspects, the image capture device of the user device 102 may capture multiple images from multiple different fields of view or positions. For example, and referring to FIG. 11A illustrating an exemplary catheter tree construction sequence, the image capture device of the user device 102 may use a burst capture technique to capture a series of images from multiple different fields of view or positions (e.g., a first field of view or position including a patient's insertion site, a second field of view or position including an infusion pump, etc.). The series of images may include tagged and / or untagged medical devices 108. The user device 102 may continuously integrate and / or combine position information associated with the medical devices 108 determined from each image in the series of images and from each series of images captured from each of the fields of view or positions, and may use the integrated position information to determine pairs of medical devices 108 that are connected to each other and to construct and display a catheter tree that includes IV lines formed by pairs of medical devices determined to be connected to each other (e.g., a catheter tree including medical devices from the insertion site and / or dressing tag to the infusion pump or module, etc.). For example, FIG. 11B shows an exemplary catheter tree that may be generated according to the catheter tree construction sequence shown in FIG. 11A.

[0116] In some non-limiting embodiments or aspects, the user device 102 may compare a drug administration, drug dosage, drug delivery route or IV line, and / or drug delivery time associated with an IV line with an approved patient, approved drug, approved drug dosage, approved drug delivery route or IV line, and / or approved drug delivery time associated with a patient identifier and / or drug identifier to reduce drug administration errors. The user device 102 may issue an alert and / or infusion pump to stop fluid flow and / or adjust fluid flow based on the current representation of at least one IV line (e.g., based on the current state of a catheter tree, etc.). For example, if the medication scheduled or loaded into the infusion pump at the entry point of the fluid flow path is determined to be an inappropriate medication for the patient, an inappropriate dosage for the patient and / or medication, an inappropriate medication delivery route for the patient and / or medication (e.g., an inappropriate entry point into the fluid flow path), and / or an inappropriate medication delivery time for the patient and / or medication, the user device 102 may alert and / or control the infusion pump to stop or prevent fluid flow.

[0117] In some non-limiting embodiments or aspects, the user device 102 may determine the dwell time of the medical device 108 (e.g., in the environment 100, etc.) and / or their connection (e.g., the amount or duration of time the medical device is connected to another medical device and / or patient, etc.). For example, the user device 102 may determine the time the medical device 108 enters the environment 100 and / or is connected to another medical device and / or patient based on the probability that the pair of medical devices are connected. As an example, the user device 102 may automatically determine and / or record the time the medical device 108 is connected to another medical device and / or patient that have been determined to be connected to each other (e.g., over a period of time, over a series of images, over a live video feed, etc.), the period from the connection time to the current time the medical device 108 is connected, and / or the time the medical device is disconnected from another medical device and / or patient.

[0118] In some non-limiting embodiments or aspects, the user device 102 may automatically determine and / or record the frequency with which the medical device 108 is connected to another medical device or a particular type of medical device based on pairs of medical devices determined to be connected to each other (e.g., over a period of time, over a series of images, over a live video feed, etc.). For example, the user device 102 may determine how often one or more antiseptic caps are connected to an IV access port, luer tip, etc., and / or the duration for which each cap is connected thereto.

[0119] In some non-limiting embodiments or aspects, the user device 102 may compare the dwell time and / or connection frequency associated with the medical device to a dwell time threshold and / or frequency threshold associated with the medical device and / or connections including the medical device, and may provide an alert associated therewith (e.g., via the user device 102, etc.) if the dwell time and / or connection frequency meets the dwell time threshold and / or frequency threshold. For example, the user device 102 may provide an alert indicating that it is time to replace a medical device in the catheter tree with a new medical device and / or that the medical device should be disinfected and / or flushed.

[0120] 12A and 12B, which are flowcharts of non-limiting embodiments or aspects of a process 1200 for estimating object distance from an image capture device. In some non-limiting embodiments or aspects, one or more of the steps of process 1200 may be performed (e.g., completely, partially, etc.) by user device 102 (e.g., one or more devices of a system of user device 102, etc.). In some non-limiting embodiments or aspects, one or more of the steps of process 1200 are performed (e.g., completely, partially, etc.) by another device or group of devices separate from or including user device 102, such as management system 104 (e.g., one or more devices of management system 104, etc.).

[0121] 12A , in step 1202, process 1200 includes acquiring a calibration image including calibration markers. For example, user device 102 may acquire the calibration image including calibration markers from an image capture device (e.g., the image capture device of user device 102, a monocular digital camera of user device 102, etc.), where the calibration markers have known dimensions and are positioned at a known distance from the image capture device. For example, user device 102 (e.g., the image capture device of user device 102, a monocular digital camera of user device 102, etc.) may capture the calibration image including the calibration markers using calibration markers with known dimensions located at a known distance from the image capture device of user device 102 (e.g., a known distance from the center of the calibration marker to the center of the image capture device, etc.). As an example, the known dimensions of the calibration markers may include the width of the calibration marker, the height of the calibration marker, the area of the calibration marker, etc. In such examples, the calibration image may include a digital image including a predetermined number of pixels (e.g., a predetermined pixel height and a predetermined pixel width, a predetermined image resolution, etc.), and / or individual pixels in the calibration image may be associated with predetermined locations or values in two-dimensional image space (e.g., predetermined x, y coordinates of the pixel, etc.).

[0122] The calibration markers may include objects associated with one or more known dimensions (e.g., known width, known height, known area, etc.). For example, the calibration markers may include medical devices 108, fiducial markers 110 (e.g., AprilTag, etc.), etc. The calibration markers may be positioned parallel to the plane of the image capture device to capture a calibration image.

[0123] 12A, in step 1204, process 1200 includes extracting one or more features associated with the calibration marker in the image space of the calibration image. For example, user device 102 may extract one or more features associated with the calibration marker in the image space of the calibration image. As an example, the one or more features associated with the calibration marker in the image space of the calibration image may include one or more pixel dimensions (e.g., pixel width, pixel height, etc.) of the calibration marker in the image space of the calibration image.

[0124] The user device 102 may use object recognition techniques to detect a region of interest (ROI) including the calibration marker in the image and extract one or more features associated with the calibration marker in the image space of the calibration image by calculating the number of pixels associated with the pixel dimension P of the calibration marker in the image space of the calibration image. For example, the user device 102 may determine the number of pixels between the sides, edges, and / or corners of the calibration marker in the image space of the calibration image as the pixel dimension P (e.g., pixel width, pixel height, etc.) of the calibration marker. As an example, and referring to FIG. 13 , which is an exemplary annotated image of AprilTag, the user device 102 may use object recognition techniques (e.g., OpenCV, etc.) to detect AprilTag as a calibration marker in an ROI in the calibration image that includes the upper right corner pixel C1 of AprilTag, the upper left corner pixel C2 of AprilTag, the lower left corner pixel C3 of AprilTag, and / or the lower right corner pixel C4 of AprilTag. For example, marker detection from the AprilTag library may provide the vertices of the detected marker as C1-C4 and a tag ID, which may be used to calculate features such as a bounding box, pixel distance to the marker, and centroid of the marker. The user device 102 may determine the right side R of the calibration marker as C1-C4 (e.g., R = C1-C4, etc.) and / or the left side L of the calibration marker as C2-C3 (e.g., L = C2-C3, etc.). The user device 102 may determine the pixel dimension P of the calibration marker (pixel width P in this exemplary case of FIG. 13 ) as the sum of the right side R of the calibration marker and the left side L of the calibration marker divided by 2 (e.g., P = (L + R) / 2, etc.).

[0125] 12A , in step 1206, process 1200 includes determining one or more calibration parameters based on one or more features associated with the calibration marker in the image space of the calibration image, the known distance of the calibration marker from the image capture device, and / or the known dimensions of the calibration marker. For example, user device 102 may determine the one or more calibration parameters based on one or more features associated with the calibration marker in the image space of the calibration image, the known distance of the calibration marker from the image capture device, and / or the known dimensions of the calibration marker. As an example, the one or more calibration parameters may be for a distance estimation technique associated with and / or configured for the image capture device, a predetermined resolution of the image capture device used to capture the calibration image, and / or one or more adjustable camera settings used to capture the calibration image (e.g., a focal length of the image capture device, if adjustable, etc.). In such an example, user device 102 may store the one or more calibration parameters (e.g., in association with a distance estimation technique, etc.). In some non-limiting embodiments, one or more calibration parameters may be stored within the user device 102 (e.g., in the memory 206, in the storage component 208, etc.) or externally (e.g., remotely) within the user device 102 (e.g., in the management system 104, etc.).

[0126] The user device 102 may use triangle similarity geometry to determine one or more calibration parameters for a distance estimation technique based on one or more features associated with the calibration marker in the image space of the calibration image, the known distance of the calibration marker from the image capture device, and / or the known dimensions of the calibration marker. For example, the user device 102 may determine an effective focal length F associated with the image capture device as the one or more calibration parameters. As an example, the user device 102 may determine the product of the pixel dimension P of the calibration marker (e.g., pixel width P, pixel height P, etc.) and the known distance D of the calibration marker from the image capture device divided by the known dimension W of the calibration marker (e.g., known width, known height, etc.) (e.g., F=(P × For example, for a pixel dimension P of a calibration marker that is 59 pixels wide, a known distance D of the calibration marker from the image capture device of 30 cm, and a known width of the calibration marker of 1 cm, the effective focal length F of the image capture device may be calculated as F=(59×30) / 1=1770 pixels.

[0127] 12A , in step 1208, process 1200 includes acquiring an image including the object. For example, user device 102 may acquire the image including the object from an image capture device (e.g., an image capture device of user device 102, a monocular digital camera of user device 102, etc.). As an example, user device 102 (e.g., an image capture device of user device 102, a monocular digital camera of user device 102, etc.) may capture the image including the object. For example, user device 102 (e.g., an image capture device of user device 102, a monocular digital camera of user device 102, etc.) may capture the image including the object at step 402 of process 400 and / or as part of another process separate from or including process 400, such as a standalone implementation of process 1200 for estimating object distance from the image capture device. In such examples, the image capture device used to capture the image including the object may be the same image capture device used to capture the calibration image, may have the same predetermined resolution as the image capture device used to capture the calibration image (e.g., the image may include a digital image including the same predetermined number of pixels as the calibration image (e.g., the same predetermined pixel height and the same predetermined pixel width, predetermined image resolution, etc.)), and / or may have the same one or more adjustable camera settings used to capture the calibration image (e.g., the focal length of the image capture device, if adjustable, etc.), and / or individual pixels in the image including the object may be associated with the same predetermined positions or values (e.g., predetermined x, y coordinates of pixels, etc.) as individual pixels in the calibration image. In such examples, the image capture device may be positioned at a distance from the object to capture the image including the object that is different from the known distance of the image capture device from the calibration marker for capturing the calibration image.

[0128] 12B, in step 1210, process 1200 includes extracting at least one feature associated with the object in the image space of the image. For example, user device 102 may extract at least one feature associated with the object in the image space of the image. As an example, the at least one feature associated with the object in the image space of the image may include one or more pixel dimensions (e.g., pixel width, pixel height, etc.) of the object in the image space of the image.

[0129] The user device 102 may use object recognition technology to detect a region of interest (ROI) containing an object in the image and extract at least one feature associated with the object in the image space of the image by calculating the number of pixels associated with the pixel dimension P of the object in the image space of the image. For example, the user device 102 may determine the number of pixels between the sides, edges, and / or corners of the object in the image space of the image as the pixel dimension P of the object (e.g., pixel width, pixel height, etc.). As an example, referring again to FIG. 13 , which is an annotated image of an exemplary AprilTag, the user device 102 may use object recognition technology (e.g., OpenCV, etc.) to detect AprilTag as an object within a ROI in the image that includes the upper right corner pixel C1 of AprilTag, the upper left corner pixel C2 of AprilTag, the lower left corner pixel C3 of AprilTag, and / or the lower right corner pixel C4 of AprilTag. The user device 102 may Object the right side R of C1-C4 (e.g., R=C1-C4), and / or Object The user device 102 may determine the left side L of the vector C as C2-C3 (e.g., L=C2-C3). Object Right side R and Object The sum of the left side of L and the right side of P is divided by 2 (for example, P = (L + R) / 2), Object 13. The pixel dimension P (pixel width P in this exemplary case of FIG. 13) of the pixel may be determined.

[0130] 12B, in step 1212, the process 1200 includes acquiring dimensions of the object. For example, the user device 102 may acquire the dimensions of the object. By way of example, the object may include a medical device 108, a fiducial marker 110 (e.g., an AprilTag, etc.), etc.

[0131] In some non-limiting embodiments or aspects, an object may be associated with one or more known dimensions (e.g., a known width, a known height, a known area, etc.). For example, the user device 102 may obtain the dimensions of an object by processing an image including the object using one or more object detection techniques and / or classifiers, such as those described herein with respect to step 404 of FIG. 4, to identify the class or type of object in the image. The user device 102 may store (e.g., in a lookup table, etc.) the dimensions of a known class or type of object (e.g., the dimensions of an AprilTag, the dimensions of a known medical device 108 such as a particular type of needleless connector, etc.). For example, the dimensions of a known class or type of object may be stored within the user device 102 (e.g., in memory 206, in storage component 208, etc.) or externally (e.g., remotely) within the user device 102 (e.g., in management system 104, etc.). In such an example, the user device 102 may obtain the dimensions of the object by obtaining the dimensions of the object based on the determined class or type of the object in the image (e.g., from a lookup table, etc.).

[0132] In some non-limiting embodiments or aspects, if the user device 102 does not detect a fiducial marker in the image and / or a fiducial marker associated with a detected object in the image, the user device 102 may attempt to detect a non-fiducial marker of known size and may use object recognition techniques (e.g., by using a software package such as OpenCV, ZXing, etc.) to estimate the dimensions of the detected object based on a first number of pixels associated with the detected object in the image, the known size of the detected non-fiducial marker, and a second number of pixels associated with the detected non-fiducial marker. In some non-limiting embodiments or aspects, if a non-fiducial marker with a known size is not detected in the image(s), the user device 102 may use object recognition techniques to detect other components (e.g., catheter components, etc.) whose dimensions are known.

[0133] 12B , in step 1214, process 1200 includes determining a distance of the object from the image capture device based on at least one feature associated with the object in the image space of the image, the dimensions of the object, and one or more calibration parameters. For example, user device 102 may determine the distance of the object from the image capture device (e.g., the distance from the center of the object to the center of the image capture device) based on at least one feature associated with the object in the image space of the image, the dimensions of the object, and one or more calibration parameters. As an example, user device 102 may determine the distance D′ of the object from the image capture device by dividing the product of the dimension (e.g., width) of object W′ and the effective focal length F of the image capture device by the pixel dimension P′ (e.g., pixel width) of the object in the image, e.g., D′=(W′×F) / P′. For example, for a pixel dimension P' of a 35.15 pixel wide object in an image (e.g., a left side L of 35 pixels and a right side R of 35.3 pixels), a 1 cm wide object dimension, and an effective focal length F of 1770 pixels for the image capture device, the distance D' of the object from the image capture device can be calculated as (1 x 1770) / 35.15 = 50.35 cm.

[0134] In some non-limiting embodiments or aspects, the object may include a 3D object including a single marker (e.g., a fiducial marker, a non-fiducial marker, etc.) utilized to contrast a major surface of the object that includes the marker with respect to the remaining surfaces of the object that do not include the marker. For example, the remaining surfaces of the object that do not include the marker may include an unmarked region or surface, which may be a colored or uncolored surface, a textured or flat surface, and / or another surface to distinguish the remaining surface or other surfaces from the major surface of the 2D image. As an example, the user device 102 may determine one or more angles at which the image was captured based on the percentage of the remaining surface that is visible in the image that includes the object, and may determine the distance of the object from the image capture device based on the one or more angles and known dimensions of the major surface that includes the marker. In such an example, the user device 102 may utilize a stereo camera to estimate the object distance. For example, the user device 102 may capture images from different perspective angles using a stereo camera, and the user device 102 may process the images captured from the stereo camera at the different perspective angles to determine the angular deviation between objects, and the angular deviation between the objects and the number of pixels associated with the objects may be used to determine distance as described in more detail herein.

[0135] 12B, in step 1216, the process 1200 includes providing the distance of the object from the image capture device. For example, the user device 102 may provide the distance of the object from the image capture device.

[0136] In some non-limiting embodiments or aspects, the user device 102 may provide the distance of an object from the image capture device by displaying the object's distance from the image capture device in real time on the display of the user device 102, which may guide a user holding the user device 102, including the image capture device, to keep the user device 102 within a specified working distance threshold to capture high-quality images. In such an example, the user device 102 may display the real-time distance of an object from the image capture device simultaneously with an image captured by the image capture device (e.g., simultaneously with an image used to estimate the object's distance from the image capture device). For example, referring to FIG. 14 , which also illustrates an implementation of a non-limiting embodiment or aspect of a display 1400 including the distance of a medical device from the image capture device, the user device 102 may provide a display 1400 including a real-time view captured by the image capture device including multiple medical devices 108, where the distance from the image capture device is displayed simultaneously in association with the multiple medical devices in the display 1400. In some non-limiting embodiments or aspects, the user device 102 may simultaneously provide on the display 1400 a representation of at least one IV line including a pair of medical devices determined to be connected to each other, as described herein with respect to step 408 of FIG. 4 .

[0137] In some non-limiting embodiments or aspects, the user device 102 may provide the distance of an object from the image capture device for use in one or more processes and / or calculations described herein with respect to step 406 of FIG. 4 to determine location information associated with three-dimensional (3D) positions of the plurality of medical devices 108 relative to the image capture device and / or to determine pairs of the plurality of medical devices 108 that are connected to one another, for use in one or more other processes and / or calculations described herein with respect to step 404 of FIG. 4 . For example, the user device 102 may use the distance of an object from the image capture device to determine whether the object is connected to another object in the image. As an example, the object may include a medical device of the plurality of medical devices 108 and / or a fiducial marker 110 associated with a medical device of the plurality of medical devices 108. In such an example, the user device 102 may determine the distance of each medical device of the multiple medical devices 108 from the image capture device, as described herein, and based on the distance of the pair of medical devices from the image capture device, may determine the difference in depth from the image capture device between the pair of medical devices, thereby enabling extraction of 3D position information associated with the multiple medical devices 108.

[0138] In some non-limiting embodiments or aspects, capturing an image including an object using an image capture device, as described with respect to step 402 of process 400 of FIG. 4 and / or step 1208 of process 1200 of FIG. 12A, may include capturing multiple images using the same image capture device (e.g., the same digital monocular camera, etc.) at different angles relative to one or more objects in the image. For example, these multiple images may be captured using the same image capture device (e.g., the same digital monocular camera, etc.) as described herein. BIn step 1214 of process 1200, the distance and / or 3D position information of one or more objects may be used to calculate object distance and / or 3D position information of one or more objects, for example, by using a multi-view depth estimation process that uses triangulation to calculate 3D position information of one or more objects.

[0139] 15, which is a flowchart of a non-limiting embodiment or aspect of a process 1500 for estimating object distance from an image capture device. In some non-limiting embodiments or aspects, one or more of the steps of process 400 are performed (e.g., completely, partially, etc.) by user device 102 (e.g., one or more devices of a system of user device 102, etc.). In some non-limiting embodiments or aspects, one or more of the steps of process 1500 are performed (e.g., completely, partially, etc.) by another device or group of devices separate from or including user device 102, such as management system 104 (e.g., one or more devices of management system 104, etc.).

[0140] 15, in step 1502, process 1500 includes capturing, with an image capture device, a first image including an object at a first angle relative to the object. For example, user device 102 (e.g., an image capture device of user device 102, a monocular digital camera of user device 102, etc.) may capture the first image including the object at a first angle relative to the object.

[0141] 15, in step 1504, process 1500 includes capturing, with an image capture device, a second image including the object at a second angle relative to the object that is different from the first angle. For example, user device 102 (e.g., an image capture device of user device 102, a monocular digital camera of user device 102, etc.) may capture the second image including the object at a second angle relative to the object that is different from the first angle. By way of example and with reference to FIG. 16, the image capture device may capture a second image including the object at a position C at which the first image is captured relative to the object, which may be located at point p in 3D space. 0 The second image is captured from position C 1 can be panned up to

[0142] 15, in step 1506, the process 1500 includes detecting a plurality of first interest points associated with the object in the first image and a plurality of second interest points associated with the object in the second image. For example, the user device 102 may use an image feature detector algorithm to detect a plurality of first interest points associated with the object in the first image and a plurality of second interest points associated with the object in the second image. As an example, the user device 102 may use one or more image feature detector algorithms described by Hassaballah, M., A.A. Abdelmgeid, and Hammam A. Alshazly (hereinafter "Hassaballah et al.") in a 2016 paper titled "Image Features Detection, Description and Matching," the entire contents of which are incorporated herein by reference.

[0143] 15, in step 1508, the process 1500 includes generating a plurality of first descriptor vectors associated with a plurality of first points of interest and a plurality of second descriptor vectors associated with a plurality of second points of interest. For example, the user device 102 、a plurality of first descriptor vectors associated with a plurality of first interest points and a plurality of second descriptor vectors associated with a plurality of second interest points; To generate the image feature descriptor, As an example, the user device 102 may use one or more image feature descriptors described by Hassaballah et al.

[0144] 15, in step 1510, process 1500 includes matching at least one first interest point of the plurality of first interest points to at least one second interest point of the plurality of second interest points. For example, user device 102 may use a feature matching algorithm to match at least one first interest point of the plurality of first interest points to at least one second interest point of the plurality of second interest points based on a plurality of first descriptor vectors associated with the plurality of first interest points and a plurality of second descriptor vectors associated with the plurality of second interest points. As an example, user device 102 may use one or more feature matching algorithms described by Hassaballah et al.

[0145] 15, in step 1512, process 1500 includes determining position information associated with a three-dimensional (3D) position of the object relative to the image capture device. For example, user device 102 may determine position information associated with a three-dimensional (3D) position of the object relative to the image capture device based on at least one first interest point of the plurality of first interest points matched to at least one second interest point of the plurality of second interest points. As an example, user device 102 may determine position information associated with a three-dimensional (3D) position of the object relative to the image capture device based on step 404 of process 400 of FIG. 4 and / or step 404 of process 400 of FIG. 12. BOne or more processes or algorithms described herein with respect to step 1214 of process 1200 may be used to calculate object distance and / or 3D position information of one or more objects, for example, by using a multi-view depth estimation process that calculates 3D position information of one or more objects using triangulation of a first point of interest that matches a second point of interest.

[0146] Thus, non-limiting embodiments or aspects of the present disclosure may use pre-defined or known detection marker features and geometric and / or triangulation properties to identify an ROI, estimate the distance between an object such as a fiducial marker and an image capture device, and / or apply image processing and mathematical logic to estimate distances that are computationally inexpensive compared to existing convolutional neural network (CNN)-based approaches, providing real-time calculation, estimation, and output of distances for better usability, as well as using the same single camera setup to mimic a stereo setup to achieve triangulation via a detect-describe-match pipeline, reducing sensor load and avoiding expensive stereo camera calibration calculations.

[0147] 17 , which is a flowchart of a non-limiting embodiment or aspect of a process 1700 for estimating the yaw angle of an object from an image capture device. In some non-limiting embodiments or aspects, one or more of the steps of process 1700 are performed (e.g., completely, partially, etc.) by user device 102 (e.g., one or more devices of a system of user device 102, etc.). In some non-limiting embodiments or aspects, one or more of the steps of process 1700 are performed (e.g., completely, partially, etc.) by another device or group of devices separate from or including user device 102, such as management system 104 (e.g., one or more devices of management system 104, etc.).

[0148] 17, in step 1702, process 1700 includes acquiring an image including a fiducial marker. For example, user device 102 may acquire an image including a fiducial marker 110 (e.g., a fiducial marker 110 attached to a medical device 108) from an image capture device (e.g., an image capture device of user device 102, a monocular digital camera of user device 102, etc.). As an example, user device 102 (e.g., an image capture device of user device 102, a monocular digital camera of user device 102, etc.) may capture an image including a fiducial marker 110 (e.g., a fiducial marker 110 attached to a medical device 108, etc.). For example, user device 102 (e.g., an image capture device of user device 102, a monocular camera of user device 102, etc.) may capture an image including fiducial marker 110 in step 402 of process 400 and / or as part of another process separate from or including process 400, such as a standalone implementation of process 1700 for estimating the yaw angle of an object from an image capture device. In such an example, the fiducial marker may have a known width and a known height. For example, the fiducial marker may include an AprilTag.

[0149] 17, in step 1704, process 1700 includes extracting the pixel width and pixel height of the fiducial marker from the image. For example, user device 102 may extract the pixel width and pixel height of the fiducial marker 110 in the image space of the image.

[0150] The user device 102 may use object recognition technology to detect a region of interest (ROI) including the fiducial marker 110 in the image and extract the pixel width and pixel height of the fiducial marker from the image by calculating the number of pixels associated with the pixel width and pixel height of the fiducial marker 110 in the image space of the image. For example, the user device 102 may determine the number of pixels between the sides, edges, and / or corners of the object in the image space of the image as the pixel dimension P (e.g., pixel width, pixel height, etc.) of the object. As an example, referring again to FIG. 13 , which is an annotated image of an exemplary AprilTag, the user device 102 may use object recognition technology (e.g., OpenCV, etc.) to detect AprilTag as an object within a ROI in the image that includes the upper right corner pixel C1 of AprilTag, the upper left corner pixel C2 of AprilTag, the lower left corner pixel C3 of AprilTag, and / or the lower right corner pixel C4 of AprilTag. The user device 102 may determine the right side R of the fiducial marker 110 as C1-C4 (e.g., R=C1-C4, etc.) and / or the left side L of the fiducial marker 110 as C2-C3 (e.g., L=C2-C3, etc.). The user device 102 may determine the sum of the right side R of the calibration marker and the left side L of the calibration marker divided by two (e.g., P w =(L+R) / 2, etc., to determine the pixel width P of the fiducial marker 110. w The user device 102 may determine the upper side T of the fiducial marker 110 as C2-C1 (e.g., T=C2-C1, etc.) and / or the lower side B of the fiducial marker 110 as C3-C4 (e.g., B=C3-C4, etc.). The user device 102 may determine the sum of the upper side of the fiducial marker 110 and the lower side of the fiducial marker 110 divided by two (e.g., P h =(T+B) / 2, etc., to determine the pixel height P of the fiducial marker 110. h can be determined.

[0151] 17, in step 1706, process 1700 includes determining a yaw angle of the fiducial marker relative to the image capture device. For example, user device 102 may estimate the yaw angle of the fiducial marker relative to the image capture device based on the pixel width, pixel height, and known width and height. As an example, user device 102 may estimate the yaw angle of the fiducial marker relative to the image capture device according to the following equation (1): Yaw angle = α*β (1)

[0152] where β is a multiplication factor and α is determined according to the following equation (2): α=((w k / h k )-(w p / h p ))*90 (2)

[0153] where w k is the known width of the fiducial marker, and h k is the known height of the fiducial marker, and w p is the pixel width of the fiducial marker in the image space of the image, and h p is the pixel height of the fiducial marker in the image space of the image. In some non-limiting embodiments or aspects, the fiducial marker h k Fiducial marker w for known height of k For example, the known height h of an AprilTag used as a fiducial marker is k The ratio of the known width of the fiducial marker wk to is 1:1.

[0154] The multiplication coefficient β (e.g., weighting coefficient) is p and pixel height h p For example, the user device 102 may have a pixel width w p and pixel height hp The value of the multiplication factor β may be obtained from a lookup table containing multiple pre-stored multiplication factors determined according to the ratio of β to β. As an example, Table 1 below contains the ratio of tag width and height for multiplication factor β calculated for tags at different angles and distances. GT in Table 1 refers to ground truth.

[0155] [Table 1]

[0156] 17, in step 1708, process 1700 includes providing a yaw angle of the fiducial marker relative to the image capture device. For example, user device 102 may provide the yaw angle of the fiducial marker 110 relative to the image capture device.

[0157] In some non-limiting embodiments or aspects, the user device 102 may provide the yaw angle of the fiducial marker 110 from the image capture device by displaying the yaw angle of the fiducial marker 110 from the image capture device in real time on the display of the user device 102, which may guide a user holding the user device 102 including the image capture device to keep the user device 102 within a specified working angle threshold to capture high-quality images. In such an example, the user device 102 may display the real-time yaw angle of the fiducial marker 110 from the image capture device simultaneously with an image captured by the image capture device (e.g., simultaneously with an image used to estimate the distance of the fiducial marker 110 from the image capture device). For example, the user device 102 may provide a display including a real-time view captured by the image capture device, including multiple medical devices 108 with corresponding fiducial markers 110 whose distances and / or yaw angles from the image capture device are displayed simultaneously in relation to the multiple medical devices 108 in the display. In some non-limiting embodiments or aspects, the user device 102 may simultaneously provide on a display a representation of at least one IV line including a pair of medical devices determined to be connected to each other, as described herein with respect to step 408 of FIG. 4 .

[0158] In some non-limiting embodiments or aspects, the user device 102 may provide the yaw angle of the fiducial marker 110 from the image capture device for use in one or more other processes and / or calculations described herein with respect to step 404 of Figure 4 to determine location information associated with three-dimensional (3D) positions of the plurality of medical devices 108 and / or corresponding fiducial markers 110 relative to the image capture device, and / or use in one or more processes and / or calculations described herein with respect to step 406 of Figure 4 to determine medical device pairs of the plurality of medical devices 108 that are connected to each other. For example, the user device 102 may use the yaw angle of the fiducial marker 110 from the image capture device and / or relative to another fiducial marker 110 to determine whether a medical device 108 that includes a fiducial marker 110 is connected to another medical device 108 that includes the other fiducial marker 110 in the image. As an example, the fiducial marker 110 may be associated with a medical device of the plurality of medical devices 108, and the user device 102 may determine the yaw angle of each medical device of the plurality of medical devices 108 from an image capture device as described herein, thereby enabling extraction of 3D position information associated with the plurality of medical devices 108.

[0159] 18 , which is a flowchart of a non-limiting embodiment or aspect of a process 1800 for estimating the roll angle of an object from an image capture device. In some non-limiting embodiments or aspects, one or more of the steps of process 1800 are performed (e.g., completely, partially, etc.) by user device 102 (e.g., one or more devices of a system of user device 102, etc.). In some non-limiting embodiments or aspects, one or more of the steps of process 1800 are performed (e.g., completely, partially, etc.) by another device or group of devices separate from or including user device 102, such as management system 104 (e.g., one or more devices of management system 104, etc.).

[0160] 18, in step 1802, process 1800 includes acquiring an image including a fiducial marker. For example, user device 102 may acquire an image including a fiducial marker 110 (e.g., a fiducial marker 110 attached to a medical device 108, etc.) from an image capture device (e.g., an image capture device of user device 102, a monocular digital camera of user device 102, etc.). As an example, user device 102 (e.g., an image capture device of user device 102, a monocular digital camera of user device 102, etc.) may capture an image including a fiducial marker 110 (e.g., a fiducial marker 110 attached to a medical device 108, etc.). For example, user device 102 (e.g., an image capture device of user device 102, a monocular camera of user device 102, etc.) may acquire an image including a fiducial marker 110 (e.g., a fiducial marker 110 attached to a medical device 108, etc.) from an image capture device (e.g., an image capture device of user device 102, a monocular digital camera of user device 102, etc.) in step 402 of process 400 and / or acquire an image of a subject from the image capture device. roll Process for estimating angles 1800An image including fiducial marker 110 may be captured as part of another process separate from or including process 400, such as a standalone implementation of process 400. In such an example, the fiducial marker may have a known width and a known height. For example, the fiducial marker may include an AprilTag.

[0161] 18, in step 1804, process 1800 includes extracting a centroid associated with the fiducial marker in the image space of the image. For example, user device 102 may extract a centroid associated with fiducial marker 110 in the image space of the image.

[0162] The user device 102 uses object recognition technology to identify the object in the image. Fiduciary Detect the region of interest (ROI) containing the marker. 、One or more features associated with the fiducial marker 110 in the image space of the image may be extracted by calculating the number of pixels associated with the pixel dimension P of the fiducial marker 110 in the image space of the image. For example, the user device 102 may determine the number of pixels between the sides, edges, and / or corners of the fiducial marker 110 in the image space of the image. By way of example, and referring to FIG. 19A , which is an annotated image of a square or rectangle representing the area between the corners of AprilTag ordered clockwise, the user device 102 may use object recognition technology (e.g., OpenCV, etc.) to detect AprilTag as a fiducial marker 110 within a ROI in the image that includes the top-left corner pixel C1 of AprilTag, the top-right corner pixel C2 of AprilTag, the bottom-right corner pixel C3 of AprilTag, and / or the bottom-left corner pixel C4 of AprilTag. For example, marker detection from the AprilTag library may provide the vertices of the detected marker as C1-C4 and the tag ID, and these corners C1-C4 may be used to calculate features such as a bounding box, pixel distance to the marker side, and centroid of the marker. The user device 102 may determine the centroid of AprilTag as being half its height and half its base or width. For example, the user device 102 may determine the y centroid coordinate of AprilTag as C1 minus C4 divided by 2 (e.g., Y_C4). bar =H / 2=(C1-C4) / 2, etc.), and / or the x centroid coordinate of AprilTag is C3 minus C4 divided by 2 (e.g., X_ bar =B / 2=(C3-C4) / 2, etc.

[0163] 18, in step 1806, process 1800 includes determining a position vector of the fiducial marker based on a centroid of the fiducial marker in image space of the image. For example, user device 102 may determine the position vector of the fiducial marker 110 based on the centroid of the fiducial marker 110 in image space of the image, a known width of the fiducial marker, and / or a known height of the fiducial marker. As an example, user device 102 may determine the position vector of the fiducial marker 110 based on the centroid and the midpoint of a side of the centroid marker 110 in image space of the image (e.g., X_ bar or W / 2, Y _bar The centroid may be used to calculate the position vectors x, y, and / or z of the fiducial marker 110 by calculating the Euclidean distance between them (such as H / 2 or H / 2), which may provide the pixel length or distance and / or direction of the x, y, and z position vectors.

[0164] Referring again to FIG. 19A, the user device 102 may calculate an orientation axis and center for each of the corners C1-C4. For example, referring to FIG. 19B, which is a graph of points forming a square or rectangle representing the area between the corners of AprilTag in FIG. 19A, for corners C1-C4 ordered clockwise, the points forming the square or rectangle on the x-axis and y-axis may be used to calculate x- and y-position vectors to identify the orientation of AprilTag. As an example, the center or centroid calculation may include the average of each point (C1, C2, C3, C4) corresponding to each corner of AprilTag, which in the example of FIG. 19B may be [(0,1)+(1,1)+(1,0)+(0,0)] / 4=(0.5,0.5). The difference of the points, e.g., C1-C4 and C2-C3, may refer to the projection on the y-axis. For example, the y position vector or yVec may include the sum of the point differences, which in the example of Figure 19B may be [(0,1)-(0,0)]+[(1,1)-(1,0)]=(0,2). Similarly, the point differences, e.g., C2-C1 and C3-C4, may refer to projections on the x-axis. For example, the x position vector or xVec may include the sum of the point differences, which in the example of Figure 19B may be [(1,1)-(0,1)]+[(1,0)-(0,0)]=(2,0).

[0165] Also, with reference to FIG. 19C, which is a graph of the orientation axis of AprilTag of FIGS. 19A and 19B, the user device 102 calculates the center, y position vector or y Vec, and x position vector or xVec of Using np.distack (for example, tagCore array data structure ) N × 2 × 3 arrayand returns a combined array index with [ :,:,0]=center, [ :, :,1]=yVec, and [ :, :,2]=xVec, which can be added to the center point in the image (e.g., cv2.line(im,center+yVec.astype('int'),center,(255,0,0),int(2*im.shape[0] / 1000)), cv2.line(im,center+xVec.astype('int'),centr, (0,0,255), int(2*im.shape[0] / 1000)), etc.). For example, as shown in FIG. 19C, the user device 102 may draw a line between (Center+yVec) and the center in the y direction, for example, between (0.5,2.5) and (0.5,0.5) in the example of FIG. 19C, and / or may draw a line between the center in the x direction and (Center+xVec), for example, between (2.5,0.5) and (0.5,0.5) in the example of FIG. 19C.

[0166] 18, in step 1808, process 1800 includes determining the slope of the position vector of the fiducial marker relative to the coordinate base axes of the image including the fiducial marker. For example, user device 102 may determine the slope of the position vector of fiducial marker 110 relative to the coordinate base axes of the image including fiducial marker 110. As an example, and referring to FIG. 20A showing the coordinate base axes of the image including AprilTag, the x position vector or xVec may be compared to the x axis of the image including fiducial marker 110. In such an example, the slope of the position vector of fiducial marker 110 relative to the coordinate base axes of the image including fiducial marker 110 may be determined according to equation (3) below. Slope = (y2-y1) / (x2-x1) (3)

[0167] where x1, y1 are the coordinates of the first point at the first end of the line or position vector, and x2, y2 are the coordinates of the second point at the second end of the line or position vector (e.g., between the centroid and the y position vector point, between the centroid and the x position vector point, etc.).

[0168] For example, see also Figure 20B, which shows exemplary x and y position vectors of AprilTag relative to the coordinate cardinal axes of the image for an exemplary scenario in which a tilted AprilTag is found with an x position vector of (3.0,2.6), a y position vector of (2.6,2.0), and a centroid of (2.5,2.5). To calculate the tilt, consider the centroid and y position vector such that (2.5,2.5) = (x2,y2) and (2.6,2.0) = (x1,y1), resulting in a tilt = -5.

[0169] 18, in step 1810, process 1800 includes determining a roll angle of the fiducial marker relative to the image capture device based on the tilt of the position vector of the fiducial marker. For example, user device 102 may determine the roll angle of the fiducial marker 110 relative to the image capture device based on the tilt of the position vector of the fiducial marker 110 relative to the coordinate base axes of the image. As an example, the roll angle of the fiducial marker 110 relative to the image capture device may be determined according to the following equation (4): Roll angle = tan -1 (Tilt) (4)

[0170] For example, referring again to FIG. 20B, for an exemplary scenario in which AprilTag is detected tilted with x position vector (3.0, 2.6), y position vector (2.6, 2.0), centroid (2.5, 2.5), and calculated slope = -5, the roll angle may be calculated as roll angle = 78.7 degrees.

[0171] 18, in step 1812, process 1800 includes providing a roll angle of the fiducial marker relative to the image capture device. For example, user device 102 may provide the roll angle of fiducial marker 110 relative to the image capture device.

[0172] In some non-limiting embodiments or aspects, the user device 102 may provide the roll angle of the fiducial marker 110 from the image capture device by displaying the roll angle of the fiducial marker 110 from the image capture device in real time on the display of the user device 102, which may guide a user holding the user device 102 including the image capture device to keep the user device 102 within a specified working angle threshold to capture high-quality images. In such an example, the user device 102 may display the real-time roll angle of the fiducial marker 110 from the image capture device simultaneously with an image captured by the image capture device (e.g., simultaneously with an image used to estimate the distance of the fiducial marker 110 from the image capture device). For example, the user device 102 may provide a display including a real-time view captured by the image capture device, including multiple medical devices 108 with corresponding fiducial markers 110, where the distance, yaw angle, and / or roll angle from the image capture device are simultaneously displayed in association with the multiple medical devices 108 in the display. In some non-limiting embodiments or aspects, the user device 102 may simultaneously provide on a display a representation of at least one IV line including a pair of medical devices determined to be connected to each other, as described herein with respect to step 408 of FIG. 4 .

[0173] In some non-limiting embodiments or aspects, the user device 102 may provide the roll angle of the fiducial markers 110 from the image capture device for use in one or more other processes and / or calculations described herein with respect to step 404 of Figure 4 to determine position information associated with three-dimensional (3D) positions of the plurality of medical devices 108 and / or corresponding fiducial markers 110 relative to the image capture device, and / or use in one or more processes and / or calculations described herein with respect to step 406 of Figure 4 to determine medical device pairs of the plurality of medical devices 108 that are connected to each other. For example, the user device 102 may use the roll angle of the fiducial marker 110 from the image capture device and / or relative to another fiducial marker 110 to determine whether a medical device 108 that includes a fiducial marker 110 is connected to another medical device 108 that includes the other fiducial marker 110 in the image. As an example, the fiducial markers 110 may be associated with medical devices of the plurality of medical devices 108, and the user device 102 may determine the roll angle of each medical device of the plurality of medical devices 108 from an image capture device as described herein, thereby enabling extraction of 3D position information associated with the plurality of medical devices 108.

[0174] In some non-limiting embodiments or aspects, user device 102 may determine and / or provide the distance of fiducial marker 110 from the image capture device, the yaw angle of fiducial marker 110 relative to the image capture device, and / or the roll angle of fiducial marker 110 relative to the image capture device. For example, user device 102 may provide the distance of fiducial marker 110 from the image capture device, the yaw angle of fiducial marker 110 relative to the image capture device, and / or the roll angle of fiducial marker 110 relative to the image capture device by displaying on a display an image including fiducial marker 110, simultaneously with the distance from fiducial marker 110, the yaw angle of fiducial marker 110 relative to the image capture device, and / or the roll angle of fiducial marker 110 relative to the image capture device. As an example, the user device 102 uses the distance of the fiducial marker 110 from the image capture device, the yaw angle of the fiducial marker 110 relative to the image capture device, and the roll angle of the fiducial marker 110 relative to the image capture device to provide the distance of the fiducial marker 110 from the image capture device, the yaw angle of the fiducial marker 110 relative to the image capture device, and the roll angle of the fiducial marker 110 relative to the image capture device to determine whether the medical device associated with the fiducial marker 110 is connected to another medical device (which may be associated with another fiducial marker) in the image.

[0175] While embodiments or aspects have been described in detail for purposes of illustration and description, it should be understood that such detail is for that purpose only and that the embodiments or aspects are not limited to the disclosed embodiments or aspects, but rather are intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it should be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or disclosed herein. While each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the set of claims.

Claims

1. coupled to the memory, acquiring a calibration image including a calibration marker, the calibration image including the calibration marker being captured by the image capture device using the calibration marker positioned a known distance from the image capture device, the calibration marker having known dimensions; extracting one or more features associated with the calibration marker in image space of the calibration image; determining one or more calibration parameters based on the one or more features associated with the calibration marker in the image space of the calibration image, the known distance of the calibration marker from the image capture device, and the known dimensions of the calibration marker; capturing an image including an object, the image including the object being captured by the image capture device; extracting at least one feature associated with the object in image space of the image; obtaining dimensions of the object; determining the distance of the object from the image capture device based on the at least one feature associated with the object in the image space of the image, the dimensions of the object, and the one or more calibration parameters; providing the distance of the object from the image capture device; 1. A system comprising: at least one processor programmed and / or configured for:

2. The system of claim 1 , wherein the one or more features associated with the calibration marker in the image space of the calibration image include pixel dimensions of the calibration marker in the image space of the calibration image.

3. The one or more calibration parameters include an effective focal length associated with the image capture device, the effective focal length being determined by the following formula: F = (P x D) / W is determined in accordance with 3. The system of claim 2, wherein F is the effective focal length, P is the pixel dimension of the calibration marker in the image space of the calibration image, D is the known distance of the calibration marker from the image capture device, and W is the known dimension of the calibration marker.

4. The system of claim 3 , wherein the at least one feature associated with the object in the image space of the image includes a pixel dimension of the object in the image space of the image.

5. The at least one processor storing in a memory a plurality of dimensions associated with a plurality of types of objects; determining a type of the object based on the image; determining the dimensions of the object based on the type of the object from the plurality of dimensions associated with the plurality of types of the object; The system of claim 4 , programmed and / or configured to obtain the dimensions of the object by:

6. 5. The system of claim 4, wherein the at least one processor is programmed and / or configured to obtain the dimensions of the object by estimating the dimensions of the object based on a number of pixels associated with the object in the image, known dimensions of at least one other object, and a number of pixels associated with the at least one other object in the image.

7. The at least one processor may be configured to: D' = (W' x F) / P' and (b) determining the distance of the object from the image capture device according to 2. The system of claim 1, wherein D′ is the distance of the object from an image capture device, W′ is the dimension of the object, F is an effective focal length, and P′ is a pixel dimension of the object in the image space of the image.

8. 2. The system of claim 1, wherein the at least one processor is programmed and / or configured to provide the distance of the object from the image capture device by displaying the distance of the object from the image capture device on a display simultaneously with the image including the object.

9. 10. The system of claim 1, wherein the at least one processor is programmed and / or configured to provide the distance of the object from the image capture device by using the distance of the object from the image capture device to determine whether the object is connected to another object in the image.

10. The at least one processor obtaining a first image including the object captured by the image capture device at a first angle relative to the object; obtaining a second image including the object captured by the image capture device at a second angle relative to the object that is different from the first angle; and (b) acquiring the image including the object by: The at least one processor using an image feature detector algorithm to detect a plurality of first interest points associated with the object in the first image and a plurality of second interest points associated with the object in the second image; generating a plurality of first descriptor vectors associated with the plurality of first interest points and a plurality of second descriptor vectors associated with the plurality of second interest points using an image feature descriptor algorithm; using a feature matching algorithm to match at least one first interest point of the plurality of first interest points to at least one second interest point of the plurality of second interest points based on the plurality of first descriptor vectors associated with the plurality of first interest points and the plurality of second descriptor vectors associated with the plurality of second interest points; determining position information associated with a three-dimensional (3D) position of the object relative to the image capture device based on the at least one first interest point of the plurality of first interest points matched to at least one second interest point of the plurality of second interest points; The system of claim 1 , programmed and / or configured to extract the at least one feature associated with the object in the image space of the image by:

11. using an image capture device to capture a calibration image including a calibration marker having known dimensions located at a known distance from the image capture device; extracting, with at least one processor, one or more features associated with the calibration marker in image space of the calibration image; determining, with the at least one processor, one or more calibration parameters based on the one or more features associated with the calibration marker in the image space of the calibration image, the known distance of the calibration marker from the image capture device, and the known dimensions of the calibration marker; The distance of the object from the image capture device capturing an image including the object with the image capture device; extracting, with the at least one processor, at least one feature associated with the object in image space of the image; obtaining dimensions of the object using the at least one processor; determining, with the at least one processor, the distance of the object from the image capture device based on the at least one feature associated with the object in the image space of the image, the dimensions of the object, and the one or more calibration parameters; providing, with the at least one processor, the distance of the object from the image capture device; and estimating by calibrating an image capture device by

12. The method of claim 11 , wherein the one or more features associated with the calibration marker in the image space of the calibration image include pixel dimensions of the calibration marker in the image space of the calibration image.

13. The one or more calibration parameters include an effective focal length associated with the image capture device, the effective focal length being determined by the following formula: F = (P x D) / W is determined in accordance with 13. The method of claim 12, wherein F is the effective focal length, P is the pixel dimension of the calibration marker in the image space of the calibration image, D is the known distance of the calibration marker from the image capture device, and W is the known dimension of the calibration marker.

14. The method of claim 13 , wherein the at least one feature associated with the object in the image space of the image includes a pixel dimension of the object in the image space of the image.

15. Obtaining the dimensions of the object includes: storing in a memory a plurality of dimensions associated with a plurality of types of objects; determining a type of the object based on the image; determining the dimension of the object from the plurality of dimensions associated with the plurality of types of objects based on the type of the object; 15. The method of claim 14, comprising:

16. 15. The method of claim 14, wherein obtaining the dimensions of the object comprises estimating the dimensions of the object based on a number of pixels associated with the object in the image, known dimensions of at least one other object, and a number of pixels associated with the at least one other object in the image.

17. The distance of the object from the image capture device is calculated using the following formula: D' = (W' x F) / P' is determined in accordance with 12. The method of claim 11 , wherein D′ is the distance of the object from an image capture device, W′ is the dimension of the object, F is an effective focal length, and P′ is a pixel dimension of the object in the image space of the image.

18. Providing the distance of the object from the image capture device includes: displaying on a display the image including the object and the distance of the object from the image capture device; using the distance of the object from the image capture device to determine whether the object is connected to another object in the image; or any combination thereof, The method of claim 11 , comprising at least one of:

19. Capturing the image including the object with the image capture device includes: capturing a first image including the object at a first angle relative to the object with the image capture device; capturing a second image with the image capture device that includes the object at a second angle relative to the object that is different from the first angle; Including, Extracting, with the at least one processor, the at least one feature associated with the object in the image space of the image includes: using an image feature detector algorithm to detect a plurality of first interest points associated with the object in the first image and a plurality of second interest points associated with the object in the second image; generating a plurality of first descriptor vectors associated with the plurality of first interest points and a plurality of second descriptor vectors associated with the plurality of second interest points using an image feature descriptor algorithm; using a feature matching algorithm to match at least one first interest point of the plurality of first interest points to at least one second interest point of the plurality of second interest points based on the plurality of first descriptor vectors associated with the plurality of first interest points and the plurality of second descriptor vectors associated with the plurality of second interest points; determining position information associated with a three-dimensional (3D) position of the object relative to the image capture device based on the at least one first interest point of the plurality of first interest points matched to at least one second interest point of the plurality of second interest points; The method of claim 11 , comprising:

20. When executed by at least one processor, the method causes the at least one processor to: acquiring a calibration image including a calibration marker, the calibration image including the calibration marker being captured by an image capture device with the calibration marker positioned a known distance from the image capture device, the calibration marker having a known dimension; extracting one or more features associated with the calibration marker in image space of the calibration image; determining one or more calibration parameters based on the one or more features associated with the calibration marker in the image space of the calibration image, the known distance of the calibration marker from the image capture device, and the known dimensions of the calibration marker; acquiring an image including an object, the image including the object being captured by the image capture device; extracting at least one feature associated with the object in image space of the image; obtaining dimensions of the object; determining the distance of the object from the image capture device based on the at least one feature associated with the object in the image space of the image, the dimensions of the object, and the one or more calibration parameters; providing the distance of the object from the image capture device; 1. A computer program product comprising at least one non-transitory computer-readable medium containing program instructions to cause a

Images