System and method for estimating the distance of an object from an image capturing device and / or the angle of an object off

Abstract

A system and method for estimating a distance of an object from an image acquisition device and / or an angle of the object away from the image acquisition device may: acquire an image including a fiducial marker, the image acquired by the image acquisition device; extracting the pixel width and / or the pixel height of the reference mark in the image space of the image; determining a distance of the fiducial marker from the image acquisition device based on the pixel width and / or pixel height, the known width and / or known height, and one or more calibration parameters; estimating a yaw angle of the fiducial marker relative to the image acquisition device based on the pixel width, the pixel height, the known width, and the known height; determining a roll angle of the reference mark relative to the image acquisition equipment based on the slope of the position vector of the reference mark; and / or providing the distance, yaw angle and / or roll angle.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 441,011, filed on January 25, 2023, entitled “System and Method for Estimating Object Distance and / or Angle from an Image Capture Device,” and also claims priority to Indian Provisional Application No. 202211042859, filed on July 26, 2022, entitled “System and Method for Vascular Access Management,” and the entire disclosure of each of these applications is incorporated by reference in its entirety. Background Art

[0003] The goal of computer vision can be to find properties of recognized objects, such as shape, color, distance from the viewing plane, and / or orientation, etc.

[0004] A dynamic monocular camera used for image acquisition in a handheld manner may result in some loss of region of interest (ROI) due to the distance between the camera and objects of interest in the camera's field-of-view (FOV). For example, distance estimation can be used in certain dynamic environments to identify the accurate FOV for acquiring image information from the camera.

[0005] Objects have three dimensions of rotational motion: pitch, yaw, and roll, which are generated by the motion of the object and / or the camera. Existing systems for calculating the yaw and roll angles of objects consume a lot of processing resources and computing time. Summary of the invention

[0006] Thus, improved systems, devices, products, apparatus and / or methods for estimating the distance of an object from an image acquisition device and / or the angle at which an object deviates from an image acquisition device are provided.

[0007] According to some non-limiting embodiments or aspects, a system is provided, the system comprising: at least one processor, the at least one processor coupled to a memory and programmed and / or configured to: acquire a calibration image including a calibration mark, wherein the calibration image including the calibration mark is acquired by an image acquisition device, the calibration mark is located at a known distance from the image acquisition device, and wherein the calibration mark has a known size; extract one or more features associated with the calibration mark in an image space of the calibration image; determine one or more calibration parameters based on the one or more features associated with the calibration mark in the image space of the calibration image, the known distance of the calibration mark from the image acquisition device, and the known size of the calibration mark; acquire an image including an object, wherein the image including the object is acquired by an image acquisition device; extract at least one feature associated with the object in the image space of the image; acquire the size of the object; determine the distance of the object from the image acquisition device based on the at least one feature associated with the object in the image space of the image, the size of the object, and the one or more calibration parameters; and provide the distance of the object from the image acquisition device.

[0008] In some non-limiting embodiments or aspects, the one or more features associated with the calibration mark in the image space of the calibration image include a pixel size of the calibration mark in the image space of the calibration image.

[0009] In some non-limiting embodiments or aspects, the one or more calibration parameters include an effective focal length associated with the image acquisition device, wherein the effective focal length is determined according to the following equation:

[0010] F=(P×D) / W

[0011] Where F is the effective focal length, P is the pixel size of the calibration mark in the image space of the calibration image, D is the known known distance of the calibration mark from the image acquisition device, and W is the known size of the calibration mark.

[0012] In some non-limiting embodiments or aspects, the at least one feature associated with the object in the image space of the image comprises a pixel size of the object in the image space of the image.

[0013] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to obtain the size of the object by: storing multiple sizes associated with multiple types of objects in a memory; determining the type of the object based on the image; and determining the size of the object from the multiple sizes associated with the multiple types of objects based on the type of the object.

[0014] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to obtain the size of the object by estimating the size of the object based on the number of pixels associated with the object in the image, the known size of at least one other object, and the number of pixels associated with the at least one other object in the image.

[0015] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to determine the distance of the object from the image acquisition device according to the following equation:

[0016] D'=(W'×F) / P'

[0017] Where D' is the distance of the object from the image acquisition device, W' is the size of the object, F is the effective focal length, and P' is the pixel size of the object in the image space of the image.

[0018] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to provide the distance of the object from the image acquisition device by displaying the distance of the object from the image acquisition device on a display simultaneously with an image including the object.

[0019] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to provide the distance of the object from the image acquisition device by using the distance of the object from the image acquisition device to determine whether the object is connected to another object in the image.

[0020] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to acquire an image including an object by: acquiring a first image including the object, the first image being acquired by an image acquisition device at a first angle relative to the object; acquiring a second image including the object, the second image being acquired by the image acquisition device at a second angle relative to the object, the second angle being different from the first angle, wherein the at least one processor is programmed and / or configured to extract the at least one feature associated with the object in the image space of the image by: detecting a plurality of first interest points associated with the object in the first image and a plurality of second interest points associated with the object in the second image using an image feature detector algorithm; and extracting the at least one feature associated with the object in the image space of the image by: detecting a plurality of first interest points associated with the object in the first image and a plurality of second interest points associated with the object in the second image using an image feature descriptor. The algorithm generates a plurality of first descriptor vectors associated with the plurality of first interest points and a plurality of second descriptor vectors associated with the plurality of second interest points; matches at least one first interest point among the plurality of first interest points with at least one second interest point among the plurality of second interest points based on the plurality of first descriptor vectors associated with the plurality of first interest points and the plurality of second descriptor vectors associated with the plurality of second interest points using a feature matching algorithm; and determines position information associated with a three-dimensional (3D) position of an object relative to an image acquisition device based on the at least one first interest point among the plurality of first interest points that matches the at least one second interest point among the plurality of second interest points.

[0021] According to some non-limiting embodiments or aspects, a method is provided, the method comprising: calibrating an image acquisition device by: using the image acquisition device to acquire a calibration image including a calibration mark, the calibration mark having a known size and being located at a known distance from the image acquisition device; using at least one processor to extract one or more features associated with the calibration mark in an image space of the calibration image; using the at least one processor to determine one or more calibration parameters based on the one or more features associated with the calibration mark in the image space of the calibration image, the known distance of the calibration mark from the image acquisition device, and the known size of the calibration mark; and estimating the distance of an object from the image acquisition device by: using the image acquisition device to acquire an image including an object; using the at least one processor to extract at least one feature associated with the object in the image space of the image; using the at least one processor to obtain the size of the object; using the at least one processor to determine the distance of the object from the image acquisition device based on the at least one feature associated with the object in the image space of the image, the size of the object, and the one or more calibration parameters; and using the at least one processor to provide the distance of the object from the image acquisition device.

[0022] In some non-limiting embodiments or aspects, the one or more features associated with the calibration mark in the image space of the calibration image include a pixel size of the calibration mark in the image space of the calibration image.

[0023] In some non-limiting embodiments or aspects, the one or more calibration parameters include an effective focal length associated with the image acquisition device, wherein the effective focal length is determined according to the following equation:

[0024] F=(P×D) / W

[0025] Where F is the effective focal length, P is the pixel size of the calibration mark in the image space of the calibration image, D is the known known distance of the calibration mark from the image acquisition device, and W is the known size of the calibration mark.

[0026] In some non-limiting embodiments or aspects, the at least one feature associated with the object in the image space of the image comprises a pixel size of the object in the image space of the image.

[0027] In some non-limiting embodiments or aspects, obtaining the size of the object includes: storing multiple sizes associated with multiple types of objects in a memory; determining the type of the object based on the image; and determining the size of the object from the multiple sizes associated with the multiple types of objects based on the type of the object.

[0028] In some non-limiting embodiments or aspects, obtaining the size of the object includes estimating the size of the object based on the number of pixels associated with the object in the image, the known size of at least one other object, and the number of pixels associated with the at least one other object in the image.

[0029] In some non-limiting embodiments or aspects, the distance of the object from the image acquisition device is determined according to the following equation:

[0030] D'=(W'×F) / P'

[0031] Where D' is the distance of the object from the image acquisition device, W' is the size of the object, F is the effective focal length, and P' is the pixel size of the object in the image space of the image.

[0032] In some non-limiting embodiments or aspects, providing the distance of an object from an image acquisition device includes at least one of: displaying the distance of the object from the image acquisition device on a display simultaneously with an image including the object; using the distance of the object from the image acquisition device to determine whether the object is connected to another object in the image; or any combination of the above.

[0033] In some non-limiting embodiments or aspects, acquiring an image including an object using an image acquisition device includes: acquiring a first image including the object at a first angle relative to the object using the image acquisition device; acquiring a second image including the object at a second angle relative to the object using the image acquisition device, the second angle being different from the first angle, wherein extracting the at least one feature associated with the object in the image space of the image using the at least one processor includes: detecting a plurality of first interest points associated with the object in the first image and a plurality of second interest points associated with the object in the second image using an image feature detector algorithm; generating a plurality of first descriptor vectors associated with the plurality of first interest points and a plurality of second descriptor vectors associated with the plurality of second interest points using an image feature descriptor algorithm; matching at least one first interest point of the plurality of first interest points with at least one second interest point of the plurality of second interest points based on the plurality of first descriptor vectors associated with the plurality of first interest points and the plurality of second descriptor vectors associated with the plurality of second interest points using a feature matching algorithm; and determining position information associated with a three-dimensional (3D) position of the object relative to the image acquisition device based on the at least one first interest point of the plurality of first interest points that matches the at least one second interest point of the plurality of second interest points.

[0034] According to some non-limiting embodiments or aspects, a computer program product is provided, the computer program product comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising program instructions, the program instructions, when executed by at least one processor, causing the at least one processor to: acquire a calibration image comprising a calibration mark, wherein the calibration image comprising the calibration mark is acquired by an image acquisition device, the calibration mark is located at a known distance from the image acquisition device, and wherein the calibration mark has a known size; extract one or more features associated with the calibration mark in an image space of the calibration image; determine one or more calibration parameters based on the one or more features associated with the calibration mark in the image space of the calibration image, the known distance of the calibration mark from the image acquisition device, and the known size of the calibration mark; acquire an image comprising an object, wherein the image comprising the object is acquired by an image acquisition device; extract at least one feature associated with the object in the image space of the image; acquire the size of the object; determine the distance of the object from the image acquisition device based on the at least one feature associated with the object in the image space of the image, the size of the object, and the one or more calibration parameters; and provide the distance of the object from the image acquisition device.

[0035] According to some non-limiting embodiments or aspects, a system is provided, comprising: at least one processor coupled to a memory and programmed and / or configured to: acquire an image including a reference marker, the image being acquired by an image acquisition device, wherein the reference marker has a known width and a known height; extract the pixel width and pixel height of the reference marker in the image space of the image; estimate the yaw angle of the reference marker relative to the image acquisition device based on the pixel width, pixel height, and the known width and known height; and provide the yaw angle of the reference marker relative to the image acquisition device.

[0036] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to estimate the yaw angle of the fiducial marker relative to the image acquisition device according to the following equation:

[0037] Yaw angle = α*β

[0038] Where β is the multiplication factor and α is determined according to the following equation:

[0039] α=((w k / h k )–(w p / h p ))*90

[0040] where w k is the known width of the fiducial mark, h k is the known height of the fiducial marker, w p is the pixel width of the fiducial marker in the image space of the image, and h p is the pixel height of the fiducial marker in the image space of the image.

[0041] In some non-limiting embodiments or aspects, the multiplication factor β is based on the pixel width w p With pixel height h p The ratio is determined from multiple pre-stored multiplication factors.

[0042] In some non-limiting embodiments or aspects, the known width w of the fiducial mark is k The known height h from the reference mark k The ratio is one to one.

[0043] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag.

[0044] According to some non-limiting embodiments or aspects, a method is provided, comprising: using at least one processor to acquire an image including a reference marker, the image being acquired by an image acquisition device, wherein the reference marker has a known width and a known height; using at least one processor to extract the pixel width and pixel height of the reference marker in the image space of the image; using at least one processor to estimate a yaw angle of the reference marker relative to the image acquisition device based on the pixel width, pixel height, and the known width and known height; and using the at least one processor to provide the yaw angle of the reference marker relative to the image acquisition device.

[0045] In some non-limiting embodiments or aspects, the at least one processor estimates the yaw angle of the fiducial marker relative to the image acquisition device according to the following equation:

[0046] Yaw angle = α*β

[0047] Where β is the multiplication factor and α is determined according to the following equation:

[0048] α=((w k / h k )–(w p / h p ))*90

[0049] where w k is the known width of the fiducial mark, h k is the known height of the fiducial marker, w p is the pixel width of the fiducial marker in the image space of the image, and h p is the pixel height of the fiducial marker in the image space of the image.

[0050] In some non-limiting embodiments or aspects, the multiplication factor β is based on the pixel width w p The ratio to the pixel height hp is determined from a plurality of pre-stored multiplication factors.

[0051] In some non-limiting embodiments or aspects, the known width w of the fiducial mark is k The known height h from the reference mark k The ratio is one to one.

[0052] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag.

[0053] According to some non-limiting embodiments or aspects, a system is provided, comprising: at least one processor coupled to a memory and programmed and / or configured to: acquire an image including a reference marker, the image being acquired by an image acquisition device, wherein the reference marker has a known width and a known height; extract the center of mass of the reference marker in the image space of the image; determine a position vector of the reference marker based on the center of mass, the known width and the known height; determine a slope of the position vector of the reference marker relative to a coordinate base axis of the image; determine a roll angle of the reference marker relative to the image acquisition device based on the slope of the position vector of the reference marker relative to the coordinate base axis of the image; and provide a roll angle of the reference marker relative to the image acquisition device.

[0054] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to estimate the slope of the position vector of the fiducial marker relative to the coordinate base axis of the image according to the following equation:

[0055] Slope = (y2–y1) / (x2–x1)

[0056] Where x1, y1 are the coordinates of a first point at a first end of the position vector, and x2, y2 are the coordinates of a second point at a second end of the position vector.

[0057] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to determine a roll angle of the fiducial marker relative to the image acquisition device based on a slope of a position vector of the fiducial marker relative to a coordinate base axis of the image according to the following equation:

[0058] Roll angle = tan-1 (slope)

[0059] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag.

[0060] In some non-limiting embodiments or aspects, the centroid of the fiducial marker in image space of the image comprises the average of points at the corners of the AprilTag, wherein the position vector comprises the sum of differences of multiple points on the axis of the image.

[0061] According to some non-limiting embodiments or aspects, a method is provided, the method comprising: using at least one processor to acquire an image including a reference marker, the image being acquired by an image acquisition device, wherein the reference marker has a known width and a known height; using the at least one processor to extract the center of mass of the reference marker in the image space of the image; using the at least one processor to determine a position vector of the reference marker based on the center of mass, the known width and the known height; determining a slope of the position vector of the reference marker relative to a coordinate base axis of the image; using the at least one processor to determine a roll angle of the reference marker relative to the image acquisition device based on the slope of the position vector of the reference marker relative to the coordinate base axis of the image; and using the at least one processor to provide a roll angle of the reference marker relative to the image acquisition device.

[0062] In some non-limiting embodiments or aspects, the at least one processor estimates a slope of a position vector of the fiducial marker relative to a coordinate base axis of the image according to the following equation:

[0063] Slope = (y2-y1) / (x2-x1)

[0064] Where x1, y1 are the coordinates of a first point at a first end of the position vector, and x2, y2 are the coordinates of a second point at a second end of the position vector.

[0065] In some non-limiting embodiments or aspects, the at least one processor determines the roll angle of the fiducial marker relative to the image acquisition device based on the slope of the position vector of the fiducial marker relative to the coordinate base axis of the image according to the following equation:

[0066] Roll angle = tan-1 (slope)

[0067] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag.

[0068] In some non-limiting embodiments or aspects, the centroid of the fiducial marker in image space of the image comprises the average of points at the corners of the AprilTag, wherein the position vector comprises the sum of differences of multiple points on the axis of the image.

[0069] According to some non-limiting embodiments or aspects, a system is provided, the system comprising: at least one processor, the at least one processor coupled to a memory and programmed and / or configured to: acquire an image including a fiducial marker, the image being acquired by an image acquisition device, wherein the fiducial marker has a known width and a known height; extract a pixel width and a pixel height of the fiducial marker in an image space of the image; determine a distance of the fiducial marker from the image acquisition device based on the pixel width and / or pixel height, the known width and / or the known height, and one or more calibration parameters; estimate a yaw angle of the fiducial marker relative to the image acquisition device based on the pixel width, pixel height, the known width and the known height; extract a center of mass of the fiducial marker in the image space of the image; determine a position vector of the fiducial marker based on the center of mass, the known width and the known height; determine a slope of the position vector of the fiducial marker relative to a coordinate base axis of the image; determine a roll angle of the fiducial marker relative to the image acquisition device based on the slope of the position vector of the fiducial marker relative to the coordinate base axis of the image; and provide the distance of the fiducial marker from the image acquisition device, the yaw angle of the fiducial marker relative to the image acquisition device, and the roll angle of the fiducial marker relative to the image acquisition device.

[0070] In some non-limiting embodiments or aspects, the at least one processor is also programmed and / or configured to: acquire a calibration image including a calibration mark, wherein the calibration image including the calibration mark is acquired by an image acquisition device, wherein the calibration mark is located at a known distance from the image acquisition device, and wherein the calibration mark has a known size; extract one or more features associated with the calibration mark in the image space of the calibration image; and determine one or more calibration parameters based on the one or more features associated with the calibration mark in the image space of the calibration image, the known distance of the calibration mark from the image acquisition device, and the known size of the calibration mark.

[0071] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to provide the distance of the reference marker from the image acquisition device, the yaw angle of the reference marker relative to the image acquisition device, and the roll angle of the reference marker relative to the image acquisition device by displaying the distance of the reference marker from the image acquisition device, the yaw angle of the reference marker relative to the image acquisition device, and the roll angle of the reference marker relative to the image acquisition device on a display simultaneously with an image including the reference marker.

[0072] In some non-limiting embodiments or aspects, the at least one processor is programmed and / or configured to provide a distance of the fiducial marker from the image acquisition device, a yaw angle of the fiducial marker relative to the image acquisition device, and a roll angle of the fiducial marker relative to the image acquisition device in the following manner: using the distance of the fiducial marker from the image acquisition device, the yaw angle of the fiducial marker relative to the image acquisition device, and the roll angle of the fiducial marker relative to the image acquisition device to determine whether a medical device associated with the fiducial marker is connected to another medical device in the image.

[0073] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag.

[0074] According to some non-limiting embodiments or aspects, a method is provided, the method comprising: acquiring an image including a fiducial marker using at least one processor, the image being acquired by an image acquisition device, wherein the fiducial marker has a known width and a known height; extracting the pixel width and pixel height of the fiducial marker in an image space of the image using the at least one processor; determining the distance of the fiducial marker from the image acquisition device based on the pixel width and / or pixel height, the known width and / or the known height, and one or more calibration parameters using the at least one processor; estimating the yaw angle of the fiducial marker relative to the image acquisition device using the at least one processor based on the pixel width, pixel height, the known width and the known height; extracting the centroid of the fiducial marker in the image space of the image using the at least one processor; determining the position vector of the fiducial marker based on the centroid, the known width and the known height using the at least one processor; determining the slope of the position vector of the fiducial marker relative to a coordinate base axis of the image using the at least one processor; determining the roll angle of the fiducial marker relative to the image acquisition device based on the slope of the position vector of the fiducial marker relative to the coordinate base axis of the image using the at least one processor; and providing the distance of the fiducial marker from the image acquisition device, the yaw angle of the fiducial marker relative to the image acquisition device, and the roll angle of the fiducial marker relative to the image acquisition device using the at least one processor.

[0075] In some non-limiting embodiments or aspects, the method further includes: acquiring a calibration image including a calibration mark using the at least one processor, wherein the calibration image including the calibration mark is acquired by an image acquisition device, wherein the calibration mark is located at a known distance from the image acquisition device, and wherein the calibration mark has a known size; extracting one or more features associated with the calibration mark in the image space of the calibration image using the at least one processor; and determining one or more calibration parameters using the at least one processor based on the one or more features associated with the calibration mark in the image space of the calibration image, the known distance of the calibration mark from the image acquisition device, and the known size of the calibration mark.

[0076] In some non-limiting embodiments or aspects, the at least one processor provides the distance of the reference marker from the image acquisition device, the yaw angle of the reference marker relative to the image acquisition device, and the roll angle of the reference marker relative to the image acquisition device by displaying the distance of the reference marker from the image acquisition device, the yaw angle of the reference marker relative to the image acquisition device, and the roll angle of the reference marker relative to the image acquisition device on a display simultaneously with an image including the reference marker.

[0077] In some non-limiting embodiments or aspects, the at least one processor provides the distance of the reference marker from the image acquisition device, the yaw angle of the reference marker relative to the image acquisition device, and the roll angle of the reference marker relative to the image acquisition device in the following manner: using the distance of the reference marker from the image acquisition device, the yaw angle of the reference marker relative to the image acquisition device, and the roll angle of the reference marker relative to the image acquisition device to determine whether the medical device associated with the reference marker is connected to another medical device in the image.

[0078] In some non-limiting embodiments or aspects, the fiducial marker comprises an AprilTag. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Additional advantages and details are explained in more detail below with reference to an exemplary embodiment shown in the attached schematic drawings, in which:

[0080] Figure 1A are schematic diagrams of non-limiting embodiments or aspects of environments in which the systems, devices, products, apparatuses, and / or methods described herein may be implemented;

[0081] Figure 1B are schematic diagrams of non-limiting examples or aspects of implementation of environments in which the systems, devices, products, apparatuses, and / or methods described herein may be implemented;

[0082] Figure 2 yes Figure 1A and Figure 1B Schematic diagrams of non-limiting embodiments or aspects of one or more devices and / or components of one or more systems;

[0083] Figure 3A are perspective views of implementations of non-limiting examples or aspects of a plurality of medical devices;

[0084] Figure 3B yes Figure 3A A perspective view of a non-limiting example or aspect of an embodiment in which multiple medical devices are connected together;

[0085] Figure 4is a flow chart of a non-limiting embodiment or aspect of a process of vascular access management;

[0086] Figure 5 An example image of a catheter insertion site including multiple medical devices is shown.

[0087] Figure 6 Non-limiting examples or aspects of implementations of fiducial markers are shown;

[0088] Figure 7 is a stereogram of an example image of a catheter insertion site on a patient;

[0089] Figure 8 Example parameters used in implementation of a non-limiting embodiment or aspect of a process for vascular access management are shown;

[0090] Fig. 9 is a diagram of examples of atypical but possible connections between medical devices;

[0091] Fig.10 A non-limiting example or aspect embodiment of a representation of an IV line is shown;

[0092] Fig.11A An example catheter tree building sequence of a process for vascular access management according to a non-limiting embodiment or aspect is shown;

[0093] Fig. 11B An example conduit tree is shown;

[0094] Fig. 12A and Fig. 12B is a flow chart of a non-limiting embodiment or aspect of a process for estimating a distance of an object from an image acquisition device;

[0095] Fig.13 is an annotated image of the example AprilTag;

[0096] Fig.14 A non-limiting embodiment or aspect of a display including a plurality of medical device distances from an image acquisition device is shown;

[0097] Fig.15 is a flow chart of a non-limiting embodiment or aspect of a process for estimating a distance of an object from an image acquisition device;

[0098] Fig.16 Implementations of non-limiting embodiments or aspects of a multi-view camera setup are illustrated;

[0099] Fig.17 is a flow chart of a non-limiting embodiment or aspect of a process for estimating a yaw angle of an object from an image acquisition device;

[0100] Fig.18 is a flow chart of a non-limiting embodiment or aspect of a process for estimating a roll angle of an object away from an image acquisition device;

[0101] Fig.19A is an annotated image of a square or rectangle representing the area between the corners of AprilTag in clockwise order;

[0102] Fig.19B is formed Fig.19A A coordinate diagram of the points of a square or rectangle representing the area between the corners of the AprilTag;

[0103] Fig.19C yes Fig.19A and Fig.19B The coordinate diagram of the orientation axis of AprilTag in;

[0104] Fig. 20A showing the coordinate base axes of the image including the AprilTag; and

[0105] Fig. 20B Example x- and y-vectors of AprilTAG relative to the coordinate base axes of the image are shown. DETAILED DESCRIPTION

[0106] It should be understood that, unless expressly provided to the contrary, the present disclosure may employ various alternative variations and step sequences. It should also be understood that the specific devices and processes shown in the accompanying drawings and described in the following specification are merely exemplary and non-limiting embodiments or aspects. Therefore, specific dimensions and other physical characteristics associated with the embodiments or aspects disclosed herein should not be considered limiting.

[0107] For the purposes of description hereinafter, the terms "end", "upper", "lower", "right side", "left side", "vertical", "horizontal", "top", "bottom", "lateral", "longitudinal" and their derivatives shall relate to the various embodiments or aspects as they are oriented in the accompanying drawings. However, it will be understood that various alternative variations and step sequences may be employed in the various embodiments or aspects unless expressly specified to the contrary. It will also be understood that the specific devices and processes shown in the accompanying drawings and described in the following specification are merely non-limiting exemplary embodiments or aspects. Therefore, unless otherwise indicated, specific dimensions and other physical characteristics associated with the embodiments or aspects of the embodiments or aspects disclosed herein should not be considered limiting.

[0108] Aspects, parts, elements, structures, actions, steps, functions and / or instructions etc. used herein should not be interpreted as critical or necessary unless explicitly described as such. In addition, as used herein, the articles "one" and "an" are intended to include one or more items, and can be used interchangeably with "one or more" and "at least one". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related items and unrelated items, etc.), and can be used interchangeably with "one or more" or "at least one". In the case of only one item being meant, the term "one" or similar language is used. In addition, as used herein, the terms "have", "have" or "have" etc. are intended to be open terms. In addition, unless otherwise explicitly stated, the phrase "based on" is intended to mean "based at least in part on".

[0109] As used herein, the terms "communication" and "communicate" may refer to the reception, receipt, transmission, transfer and / or provision of information (e.g., data, signals, messages, instructions and / or commands, etc.). A unit (e.g., a device, a system, a component of a device or system and / or a combination thereof, etc.) communicating with another unit means that the unit is able to directly or indirectly receive information from another unit and / or transmit information to another unit. This may refer to a direct or indirect connection that is essentially wired and / or wireless. In addition, although the transmitted information may be modified, processed, relayed and / or routed between a first unit and a second unit, the two units may communicate with each other. For example, even if the first unit passively receives information and does not actively transmit information to the second unit, the first unit may communicate with the second unit. As another example, if at least one intermediate unit (e.g., a third unit located between the first unit and the second unit) processes the received information from the first unit and transmits the processed information to the second unit, the first unit may communicate with the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet, etc.) comprising data. It will be appreciated that many other arrangements are possible.

[0110] As used herein, the term "computing device" may refer to one or more electronic devices that are configured to communicate directly or indirectly with one or more networks or to communicate through one or more networks. A computing device may be a mobile or portable computing device, a desktop computer and / or a server, etc. In addition, the term "computer" may refer to any computing device that includes the necessary components for receiving, processing and outputting data, and typically includes a display, a processor, a memory, an input device and a network interface. A "computing system" may include one or more computing devices or computers. An "application" or "application program interface (API)" refers to a computer code or other data stored on a computer-readable medium, which may be executed by a processor to facilitate interaction between software components, such as a client-side front end and / or a server-side back end for receiving data from a client. An "interface" refers to a generated display, such as one or more graphical user interfaces (GUIs) with which a user may interact directly or indirectly (e.g., via a keyboard, a mouse, a touch screen, etc.). In addition, a plurality of computers (e.g., servers or other computerized devices) that communicate directly or indirectly in a network environment may constitute a "system" or a "computing system."

[0111] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, software, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited to these embodiments. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0112] Some non-limiting embodiments or aspects are described herein in conjunction with threshold values. As used herein, satisfying a threshold value may refer to a value being greater than a threshold value, more than a threshold value, above a threshold value, greater than or equal to a threshold value, less than a threshold value, less than a threshold value, below a threshold value, less than or equal to a threshold value, equal to a threshold value, etc.

[0113] Reference now Figure 1A , Figure 1A 1 is a schematic diagram of an example environment 100 in which the devices, systems, methods, and / or products described herein may be implemented. Figure 1A As shown, environment 100 includes user devices 102, management system 104, and / or communication network 106. Systems and / or devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0114] Also refer to Figure 1B , Figure 1B is a schematic diagram of a non-limiting example or aspect of an implementation of an environment 100 in which the systems, devices, products, apparatuses, and / or methods described herein may be implemented. Figure 1B As shown, the environment 100 may include a patient room including: a patient, one or more medical devices 108, one or more fiducial markers 110 associated with the one or more medical devices 108, and / or a caregiver (eg, a nurse, etc.).

[0115] The user device 102 may include one or more devices that are capable of receiving information and / or data from the management system 104 (e.g., via the communication network 106, etc.) and / or transmitting information and / or data to the management system 104 (e.g., via the communication network 106, etc.). For example, the user device 102 may include one or more computing systems that include one or more processors (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, one or more tablet computers, etc.). In some non-limiting embodiments or aspects, the user device 102 may include a tablet computer or a mobile computing device, such as Tablet( iPad), cell phone( iPhone), Android tablet ( tablet), cell phone( phone), etc.

[0116] The user device 102 may include one or more image acquisition devices (e.g., one or more cameras, one or more sensors, etc.) configured to acquire one or more images of an environment (e.g., environment 100, etc.) surrounding the one or more image acquisition devices. For example, the user device 102 may include one or more image acquisition devices configured to acquire one or more images of one or more medical devices 108, one or more fiducial markers 110 associated with the one or more medical devices 108, and / or a patient. As an example, the user device 102 may include a monocular camera.

[0117] The management system 104 may include one or more devices that are capable of receiving information and / or data from the user device 102 (e.g., via the communication network 106, etc.) and / or transmitting information and / or data to the user device 102 (e.g., via the communication network 106, etc.). For example, the management system 104 may include one or more computing systems that include one or more processors (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, etc.). In some non-limiting embodiments or aspects, the management system 104 includes a nurse station or terminal in a hospital and / or is accessible via a nurse station or terminal in a hospital. For example, the management system 104 may provide bedside nurse support, nurse station administrator support, and / or retrospective reporting of nursing management, etc.

[0118] The communication network 106 may include one or more wired networks and / or wireless networks. For example, the communication network 106 may include a cellular network (e.g., a long-term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network and / or a cloud computing network, etc., and / or a combination of these or other types of networks.

[0119] Figure 1A and Figure 1B The number and arrangement of the systems and devices shown in are provided as examples. Figure 1A and Figure 1B There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, or differently arranged systems and / or devices compared to the systems and / or devices shown in . Figure 1A and Figure 1B Two or more systems or devices shown in the figure may be implemented in a single system or a single device, or Figure 1A and Figure 1B A single system or a single device shown in the environment 100 may be implemented as multiple distributed systems or devices. Additionally or alternatively, a group of systems or a group of devices (e.g., one or more systems, one or more devices, etc.) of the environment 100 may perform one or more functions described as being performed by another group of systems or another group of devices of the environment 100.

[0120] Reference now Figure 2 , Figure 2 2 is a schematic diagram of example components of device 200. Device 200 may correspond to user device 102 (e.g., one or more devices of a system of user device 102, etc.) and / or one or more devices of management system 104. In some non-limiting embodiments or aspects, user device 102 (e.g., one or more devices of a system of user device 102, etc.) and / or one or more devices of management system 104 may include at least one device 200 and / or at least one component of device 200. Figure 2 As shown, device 200 may include a bus 202 , a processor 204 , a memory 206 , a storage component 208 , an input component 210 , an output component 212 , and a communication interface 214 .

[0121] The bus 202 may include components that allow communication between the components of the device 200. In some non-limiting embodiments or aspects, the processor 204 may be implemented in hardware, software, or a combination of hardware and software. For example, the processor 204 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.). The memory 206 may include a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by the processor 204.

[0122] The storage component 208 may store information and / or software related to the operation and use of the device 200. For example, the storage component 208 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, a solid-state disk, etc.), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of computer-readable medium, and a corresponding drive.

[0123] Input components 210 may include components that allow device 200 to receive information, for example, via a user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally or alternatively, input components 210 may include sensors for sensing information (e.g., global positioning system (GPS) components, accelerometers, gyroscopes, actuators, image acquisition devices, etc.). Output components 212 may include components that provide output information from device 200 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.).

[0124] The communication interface 214 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables the device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 214 may allow the device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, interface and / or cellular network interface, etc.

[0125] The device 200 can perform one or more processes described herein. The device 200 can perform these processes based on the processor 204 executing software instructions stored by a computer-readable medium (e.g., memory 206 and / or storage component 208). Computer-readable media (e.g., non-transitory computer-readable media) are defined herein as non-transitory storage devices. Storage devices include storage space located within a single physical storage device or storage space spread across multiple physical storage devices.

[0126] The software instructions may be read into the memory 206 and / or storage component 208 from another computer-readable medium or from another device via the communication interface 214. The software instructions stored in the memory 206 and / or storage component 208, when executed, may cause the processor 204 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the embodiments or aspects described herein are not limited to any specific combination of hardware circuitry and software.

[0127] The memory 206 and / or storage component 208 may include a data storage device or one or more data structures (e.g., a database, etc.). The device 200 may be capable of receiving information from the data storage device or one or more data structures in the memory 206 and / or storage component 208, storing information in the data storage device or one or more data structures, transmitting information to the data storage device or one or more data structures, or searching for information stored in the data storage device or one or more data structures.

[0128] Figure 2 The number and arrangement of components shown are provided as examples. In some non-limiting embodiments or aspects, Figure 2 Device 200 may include additional components, fewer components, different components, or differently arranged components than those shown in . Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0129] Reference now Figure 3A and Figure 3B , Figure 3A are perspective views of implementations of non-limiting examples or aspects of a plurality of medical devices, and Figure 3B yes Figure 3A A perspective view of an implementation of a non-limiting example or aspect in which multiple medical devices are connected together.

[0130] The medical devices 108 may include at least one of the following types of medical devices: a peripheral IV catheter (PIVC); a peripherally inserted central catheter (PICC); a midline catheter; a central venous catheter (CVC); a needleless connector; a catheter dressing; a catheter stabilization device; a disinfectant cap; a disinfectant swab or wipe; an IV tubing set; an extension set; a Y-connector; a stopcock; an infusion pump; a flush syringe; a drug delivery syringe; an IV fluid bag; an inner lumen adapter (e.g., the number of inner lumen adapters associated with a catheter may indicate the number of lumens included in the catheter, etc.); or any combination thereof.

[0131] A fiducial marker 110 (e.g., a label, tag, code, etc.) can be associated with a medical device 108 (e.g., removably attached to the medical device, permanently attached to the medical device, integrated with the medical device, implemented on the medical device, etc.). In some non-limiting embodiments or aspects, each medical device 108 in the environment 100 can be associated with a fiducial marker 110. In some non-limiting embodiments or aspects, only a portion of the medical devices 108 in the environment 100 can be associated with a fiducial marker 110. In some non-limiting embodiments or aspects, none of the multiple medical devices 108 in the environment 100 can be associated with a fiducial marker 110.

[0132] The fiducial marker 110 may encapsulate an identifier associated with the type of medical device 108 associated with the fiducial marker 110 and / or uniquely identify the medical device 108 associated with the fiducial marker 110 from other medical devices. For example, the fiducial marker 110 may encapsulate an identifier associated with at least one of the following types of medical devices: a peripheral IV catheter (PIVC); a peripherally inserted central catheter (PICC); a midline catheter; a central venous catheter (CVC); a needleless connector; a sterile cap; a sterile swab or wipe; an IV tubing set; an extension set; a Y-connector; a stopcock; an infusion pump; a flushing syringe; a drug delivery syringe; an IV fluid bag; or any combination thereof, and / or the fiducial marker may uniquely identify the medical device 108 (e.g., a first needleless connector, etc.) from other medical devices (e.g., a second needleless connector, etc.) that include an identifier associated with the same type of medical device.

[0133] The fiducial markers 110 may encapsulate pose information associated with the 3D position of the fiducial markers 110. For example, the fiducial markers 110 may include markers that, when captured in an image, enable calculation of the precise 3D position of the fiducial marker relative to an image capture device that captured the image (e.g., the x, y, z coordinate position of the fiducial marker, etc.) and / or the precise two-dimensional (2D) position of the fiducial marker in the image itself (e.g., the x, y coordinate position of the fiducial marker in the image, etc.).

[0134] In some non-limiting embodiments or aspects, the fiducial marker 110 may include an AprilTag. For example, the fiducial marker 110 may include an AprilTag V3 of the custom Tag 48h12 type, which enables the use of AprilTag V3 detection to determine a unique ID and / or position of the fiducial marker 110 in the field of view (FOV) of the image acquisition device (e.g., x, y, and z coordinates, direction vectors of the Z axis, Y axis, and X axis, etc.), the unique ID may indicate (e.g., in leading digits, etc.) the type of medical device 108 associated with the fiducial marker and / or (e.g., in trailing digits, etc.) a unique serial number of the specific medical device 108. However, non-limiting embodiments or aspects are not limited to this, and the reference marker 110 may include a QR code, a barcode (e.g., a one-dimensional (1D) barcode, a 2D barcode, etc.), an Aztec code, a Data Matrix code, an ArUco mark, a color pattern, a reflective pattern, a fluorescent pattern, a predefined shape and / or color (e.g., a red pentagon, a blue hexagon, etc.), an LED pattern and / or a hologram, etc., which encapsulates an identifier associated with the type of medical device 108 associated with the reference marker 110, uniquely identifies the medical device 108 associated with the reference marker 110 from other medical devices, and / or encapsulates posture information associated with the 3D position of the reference marker 110.

[0135] In some non-limiting embodiments or aspects, the fiducial marker 110 may include a color calibration region located adjacent to the variable color region to calibrate the color under a wider range of lighting conditions. For example, for a 2×2 grid, the cell (1,1) at the upper left corner of the grid may include a predefined and / or standard calibration color region (e.g., neutral gray, etc.) that the user device 102 and / or the management system 104 may use to calibrate the colors in the images for detecting or determining the fiducial marker 110 in the images and / or detecting or determining color changes in patient tissue (e.g., patient tissue near an insertion site, etc.) in the images. In such an example, the user device 102 and / or the management system 104 may use the predefined and / or standard calibration color region to orient the fiducial marker 110, to determine how to properly rotate the fiducial marker 110, and to decode the colors in the fiducial marker 110 to decode an identifier encapsulated by the fiducial marker 110 and / or to track the fiducial marker 110 within the environment 100.

[0136] like Figure 3A and Figure 3B As shown, the fiducial markers 110 can be arranged in a ring symmetrically around the axis of the medical device 108 associated with these fiducial markers, which can enable at least one fiducial marker 110 to be presented to the FOV of the image acquisition device regardless of the orientation of the medical device 108. The fiducial marker 110 can be locked so that the direction of the marker (e.g., indicated by its posture information, etc.) is aligned with the proximal direction or distal direction of the fluid flowing through the medical device 108 associated with the fiducial marker. The fiducial marker 110 can be rigidly fixed to the medical device 108 so that the fiducial marker 110 cannot translate along the medical device 108 and / or rotate around the medical device 108 (e.g., rigidly fixed to a rigid portion of the medical device 108 and / or a catheter tree including the medical device 108, etc.), which can reduce the movement and / or distance change of the fiducial marker relative to other fiducial markers and / or medical devices.

[0137] The fiducial markers 110 may be located at or just adjacent to the connection points or ports of the various medical devices 108, such that the fiducial markers 110 on the connected plurality of medical devices 108 are collinear (e.g., parallel, etc.). For example, the fiducial markers 110 on the connected plurality of medical devices 108 that are collinear (e.g., parallel, etc.) in this manner may also be continuous or spaced a known distance apart.

[0138] A single medical device 108 may include one or more sets of fiducial markers 110. For example, Figure 3A and Figure 3BAs shown, the rightmost cap in each of these figures includes a single set of fiducial markers 110 (e.g., where each fiducial marker 110 in the set is identical and / or encapsulates the same information, etc.), and the tube to the immediate left of the cap includes two sets of fiducial markers 110 located at respective connection points of the tube and separated by the tube. In such an example, collinearity between the fiducial markers 110 at each end of the tube may not be guaranteed, and the connection between the fiducial markers 110 at each end of the tube may be established by a predefined scheme (e.g., where each fiducial marker 110 in each set of fiducial markers 110 on the same medical device 108 has the same value or different but adjacent values, etc.). Note that the spacing between multiple connected medical devices 108 may vary (e.g., as shown in FIG. 1 ). Figure 3B ); however, for each possible connection between the medical devices, the interval can be determined and known by the user device 102 and / or the management system 104.

[0139] Reference now Figure 4 , Figure 4 is a flow chart of a non-limiting embodiment or aspect of a process 400 for vascular access management; in some non-limiting embodiments or aspects, one or more of the multiple steps of process 400 may be performed by user device 102 (e.g., one or more devices of a system of user device 102, etc.) (e.g., completely, partially, etc.). In some non-limiting embodiments or aspects, one or more of the multiple steps of process 400 may be performed by another device or group of devices (e.g., management system 104 (e.g., one or more devices of management system 104, etc.)) that is separate from or includes user device 102 (e.g., completely, partially, etc.).

[0140] like Figure 4 As shown, at step 402, process 400 includes acquiring an image. For example, user device 102 may acquire images (e.g., a single image, multiple images, a series of images, etc.) of multiple medical devices 108 acquired by an image acquisition device. As an example, the image acquisition device of user device 102 may acquire images of multiple medical devices 108. In such an example, a nurse may use user device 102 to capture one or more images of a patient's catheter site and / or an infusion pump connected to the catheter site. For example, Figure 5 An example image of a catheter insertion site including multiple medical devices is shown.

[0141] In some non-limiting embodiments or aspects, the image may include a series of images. For example, the image acquisition device of the user device 102 may use a burst acquisition technique (e.g., a "burst mode," a continuous shooting mode, etc.) to acquire a series of images, which may enable the user device 102 to create a second likelihood layer for determining the probability that multiple pairs of medical devices 108 are connected as described in more detail below, thereby improving motion artifacts, angles, distances, and / or missed fiducial marker detection. As an example, the image acquisition device of the user device 102 may acquire a series of images as a live video feed to identify multiple pairs of medical devices 108 that are connected to each other (e.g., identify catheter tree components and generate catheter trees, etc.) when the live video feed is acquired.

[0142] like Figure 4 As shown, at step 404, process 400 includes determining position information. For example, user device 102 may determine position information associated with 3D positions of multiple medical devices 108 relative to the image acquisition device and / or 2D positions of the medical devices 108 in the image itself based on the image. In such an example, determining position information associated with 3D positions of multiple medical devices 108 relative to the image acquisition device and / or 2D positions of multiple medical devices 108 in the image itself may further include determining a type of each medical device in the multiple medical devices 108 based on the image.

[0143] In some non-limiting embodiments or aspects, a first group of medical devices in the plurality of medical devices 108 are associated with a plurality of fiducial markers 110. The plurality of fiducial markers 110 may encapsulate a plurality of identifiers associated with the first group of medical devices and pose information associated with the 3D positions and / or 2D positions of the plurality of fiducial markers 110. In such an example, the user device 102 may determine position information associated with the 3D positions of the first group of medical devices relative to the image acquisition device and / or the 2D positions of the plurality of medical devices 108 in the image itself by determining or identifying the plurality of fiducial markers 110 associated with the first group of medical devices and the pose information associated with the 3D positions and / or 2D positions of the plurality of fiducial markers 110 based on the image. For example, also referring to Figure 6, the plurality of fiducial markers 110 may include a plurality of AprilTags, and the user device 102 may process the image using AprilTag detection software to determine the type of medical device 108 associated with the plurality of fiducial markers 110 and / or a unique serial number of the particular medical device 108, and calculate the precise 3D position, orientation, and / or identity of the plurality of fiducial markers 110 relative to the image acquisition device that acquired the image, and / or the precise 2D position of the plurality of fiducial markers 110 in the image itself. As an example, for each medical device 108 in the first group of medical devices, the position information associated with the 3D position of the medical device 108 relative to the image acquisition device may be determined as the 3D position of the fiducial marker 110 associated with the medical device 108 relative to the image acquisition device, and / or the position information associated with the 2D position of the medical device 108 in the image itself may be determined as the 2D position of the fiducial marker 110 associated with the medical device 108. In such an example, for each medical device 108 in the first group of medical devices, the 3D position of the fiducial marker 110 associated with the medical device relative to the image acquisition device may include: the x, y, and z coordinates of the fiducial marker 110 and / or the direction vectors of the Z axis, Y axis, and X axis of the fiducial marker 110. In such an example, for each medical device 108 in the first group of medical devices, the 2D position of the fiducial marker 110 in the image itself may include: the x, y coordinates of the fiducial marker 110 in the image and / or the direction vectors of the Y axis and X axis of the fiducial marker.

[0144] In some non-limiting embodiments or aspects, a second group of medical instruments in the plurality of medical instruments are not associated with a fiducial marker. In such an example, the user device 102 may determine or identify the type of the medical instrument, the 3D position of the medical instrument relative to the image acquisition device, and / or the 2D position of the medical instrument in the image itself for each medical instrument in the second group of medical instruments based on the image using one or more existing object detection techniques, thereby determining position information associated with the 3D position of the second group of medical instruments relative to the image acquisition device and / or the 2D position of the second group of medical instruments in the image itself. For example, a medical device 108 without a reference mark or identifier tag can be identified by a user device 102, which processes an image using one or more object detection techniques (e.g., deep learning techniques, image processing techniques, image segmentation techniques, etc.) to identify or determine the medical device 108 in the image, as well as position information associated with the 3D position of the identified medical device 108 relative to the image acquisition device (e.g., including the x, y, and z coordinates of the medical device and the direction vectors of the Z, Y, and X axes of the medical device 108, etc.) and / or the 2D position of the identified medical device in the image itself. For example, deep learning techniques may include bounding box techniques for generating box labels for objects of interest in an image (e.g., medical devices 108, etc.), image masking techniques (e.g., masked fast regional convolutional neural networks (FRCNN) (regional convolutional neural networks (RCNN) or convolutional neural networks (CNN))) for capturing objects of specific shapes in an image (e.g., medical devices 108, etc.), trained neural networks for identifying objects in an image (e.g., medical devices 108, etc.), and / or classifiers for classifying the identified objects into multiple categories or types of objects, etc. As an example, image processing techniques may include cross-correlation image processing techniques, image contrast techniques, and / or binary or color filtering techniques, etc. As an example, different catheter lumens may include unique colors, and image processing may use the color to identify the type of catheter.

[0145] In some non-limiting embodiments or aspects, the user device 102 may process the image data using stereo imaging technology and / or shadow distance technology to determine object data including the distance from the image acquisition device to the detected object and / or the distance between each detected object, and / or the user device 102 may use multiple cameras, laser focusing technology, LiDAR sensors and / or camera physical magnification functions to acquire image data to determine object data including the distance from the image acquisition device to the detected object and / or the distance between each detected object. In some non-limiting embodiments or aspects, the user device 102 may use a 3D optical profiler to acquire image data and / or object data including a 3D profile of an object.

[0146] For example, the image acquisition device may include a stereo camera, and / or the position information associated with the 3D position of the plurality of medical devices relative to the image acquisition device may be determined using a structure from motion (SfM) algorithm. As an example, the user device 102 may include a stereo camera setup, which is available in many mobile devices, such as Tablet computer, cell phone, Tablets and / or Mobile phones, etc. The user device 102 can process the images from the stereo camera using an SfM algorithm to extract 3D information that can enhance object feature recognition of the fiducial markers 110 of the first group of medical devices and / or object feature recognition of medical devices in the second group of medical devices without fiducial markers. As an example, the SfM processing can improve the extraction of 3D features of medical devices in the second group of medical devices without fiducial markers, which can improve the chance of image feature accumulation, for example, by using a continuous image acquisition / video mode that acquires images with appropriate orientation / alignment (e.g., pan / tilt, etc.) based on the settings or positions of the medical devices 108 and / or the anatomical locations of these medical devices on the patient, wherein a catheter tree is constructed by starting from a dressing tag and connecting the center point or centroid of the medical device 108 detected using object recognition technology to other medical devices 108 with or without fiducial markers 110. As another example, these 3D features extracted using SfM processing can be used to calculate the coplanarity of the medical device 108 (e.g., for generating catheter trees, etc.), which can provide additional advantages in multi-lumen and multi-tube catheter settings by reducing missed connections that may occur in 2D single-plane space due to pseudo-collinearity.

[0147] For example, the image acquisition device may include a LiDAR system, and the image may include a LiDAR point cloud. As an example, the user device 102 may include a mini-LiDAR system, which is used in many mobile devices (e.g., Tablet computer, cell phone, Tablets and / or In such an example, the use of LiDAR images can improve the accuracy of 3D feature detection because LiDAR images directly provide 3D world information as a point cloud, which can accelerate 3D data collection with reduced or minimal protocols compared to stereo settings. For example, user device 102 can use existing image registration and / or transformation techniques to overlay 2D object information (e.g., color and / or texture, etc.) from camera images with the 3D LiDAR point cloud to detect 3D features for enhancing duct tree generation and connection accuracy, which can also improve augmented reality generation and environment reconstruction, as well as guide duct tree generation and connectivity determination.

[0148] For example, also refer to Figure 7 , Figure 7 is a stereoscopic image of an example catheter insertion site on a patient, due to the positioning of the AprilTag and the different tubes, the likelihood of mismatched tree generation (e.g., incorrect determination of connections between medical devices, etc.) is high because the fiducial markers 110 and / or medical devices 108 appear to be co-linear in 2D due to their proximity. The user device 102 can use the above-described stereo / SfM and / or LiDAR image-based methods for determining 3D features of the fiducial markers 110 and / or medical devices 108 to improve coplanar information and / or provide more accurate catheter tree generation (e.g., more accurate determination of connections between devices, etc.).

[0149] The following references Fig. 12A and Fig. 12B , Fig.15 , Fig.17 as well as Fig.18 Further details regarding non-limiting embodiments or aspects of step 404 of process 400 are provided.

[0150] like Figure 4 As shown, at step 406, the process 400 includes determining a plurality of pairs of medical devices that are connected to each other. For example, the user device 102 may determine a plurality of pairs of medical devices that are connected to each other among the plurality of medical devices 108 based on the position information and / or the type of the plurality of medical devices 108. As an example, for each pair of medical devices among the plurality of medical devices 108, the user device 102 may determine a probability that the pair of medical devices are connected to each other based on the 3D position information associated with the pair of medical devices, the 2D position information associated with the pair of medical devices, and / or the type of the pair of medical devices. In such an example, the user device 102, for each medical device among the plurality of medical devices 108, the medical device may be determined to be connected to another medical device in a pair of medical devices that includes the medical device, and the pair of medical devices is associated with the highest probability of the plurality of pairs of medical devices that include the medical device.

[0151] For example, for each pair of medical devices among the plurality of medical devices, the user device 102 may determine the following parameters based on the location information associated with the pair of medical devices: the distance between the center points associated with the pair of medical devices; the angular difference between the orientations of the pair of medical devices; and / or the angle of deviation of the pair of medical devices from the collinearity, and the probability that the pair of medical devices are connected may be determined based on the distance between the center points of the pair of medical devices, the angular difference between the orientations of the pair of medical devices, and / or the angle of deviation of the pair of medical devices from the collinearity. As an example, also refer to Figure 8 , for each pair of reference markers among the multiple reference markers 110a, 110b and 110c, the user device 102 can determine the following parameters based on the 3D position information and / or 2D position information associated with the pair of reference markers: the distance between the center points of the pair of reference markers (e.g., proximity, etc.); the angular difference between the orientations of the pair of reference markers (e.g., orientation, etc.); and / or the angle of deviation from collinearity of the pair of reference markers (e.g., collinearity, etc.), and the probability that a pair of medical devices associated with the pair of reference markers are connected can be determined based on the determined proximity, orientation and / or collinearity (e.g., the angular difference in orientation and angle generated by the tag orientation and the connection vector, etc.). Although in Figure 8 110, but the non-limiting embodiments or aspects are not limited thereto, and the proximity, orientation and / or colinearity between medical devices may be determined between multiple pairs of medical devices without reference markers, and / or between multiple pairs of medical devices including a single medical device associated with a reference marker and a single medical device without a reference marker.

[0152] In some non-limiting embodiments or aspects, the user device 102 may use probability-based tree building logic or rule sets (e.g., predefined rule sets, etc.) to determine a pair of interconnected medical devices 108. As an example, the user device 102 may use the above parameters: proximity, orientation, and / or colinearity (and / or one or more additional parameters, such as the X-axis distance and the Y-axis distance between the pair of medical devices, and / or the depth difference between the pair of medical devices and the image acquisition device, etc.) to determine the probability of each medical device 108 connecting to each of the other medical devices, and when applying the probability-based tree building logic or rule set to determine the pair of interconnected medical devices 108, the user device 102 may assign a predefined weight to each parameter for the total probability of the pair of medical devices being connected to each other (e.g., the sum of the weights of the various parameters may be 1, etc.). In such an example, the user device 102 can process multiple pairs of medical devices 108 by using the dressing tags and / or intraluminal adapters as starting points or anchors for generating a catheter tree or a representation of one or more IV lines, and for each medical device, the medical device can be determined to be connected to the other medical device in the pair of medical devices that has the highest probability of connection to the medical device.

[0153] In such an example, the user device 102 may identify a group of medical devices in the plurality of medical devices 108 that are associated with a preferred IV line architecture based on the type of each of the plurality of medical devices. For a pair of medical devices included in the group of medical devices associated with the preferred IV line architecture, the user device 102 may adjust the weights used to determine whether the pair of medical devices are connected to each other (e.g., a known preferred architecture may receive a higher weight in the connection determination logic, etc.). As an example, if the user device 102 identifies each of the plurality of medical devices or disposables required to generate a preferred architecture connection for each IV line in an image, the user device 102 may determine a connection between the pair of medical devices by assigning a higher probability to a preferred connection including a plurality of pairs of medical devices connected in a preferred architecture (e.g., an inner lumen adapter connected to an IV tube, etc.). For example, when the medical devices are determined to be within a threshold proximity or distance of each other, a proximity or distance parameter of a pair of preferred architecture components may be assigned a higher weight than other parameters (e.g., orientation, colinearity, etc.). In some non-limiting embodiments or aspects, the user device 102 can prompt the user to retake an image and / or provide the user with a notification of these atypical parameters (e.g., asking whether there are multiple catheter components in a single image, etc.) in response to determining atypical parameters associated with a pair of medical devices (e.g., the distance of the dressing from the inner cavity adapter is a threshold distance, such as 30 centimeters (cm), etc.).

[0154] In some non-limiting embodiments or aspects, the user device 102 may adjust the weight used to determine whether a pair of medical devices are connected to each other based on a determination that one medical device in a pair of medical devices is connected to another medical device (e.g., another medical device in a preferred IV line architecture that includes the pair of medical devices). For example, for a pair of medical devices including a first medical device and a second medical device, if the user device 102 determines that the first medical device has been connected to a third medical device in a preferred IV line architecture (the preferred IV line architecture includes the first medical device, the second medical device, and the third medical device), the user device 102 may assign a higher probability that the first medical device and the second medical device are connected because such a connection implements the preferred IV line architecture.

[0155] In such an example, if the user device 102 identifies in the image only a portion of the multiple medical devices or disposables required to generate a preferred architecture connection for each IV line, the user device 102 may determine a connection between a pair of medical devices by assigning a greater probability to the preferred connections, which include multiple pairs of medical devices connected in the preferred architecture and having medical devices of predefined types (e.g., devices that are not caps, etc.). For example, if a triple lumen catheter is expected for the preferred architecture and only a single IV tube is identified in the image, the user device 102 may automatically expect other lumens or caps on lumen adapters, and if the user device 102 does not identify a cap in the image, the user device 102 may prompt the user to retake the image and / or provide a notification to the user requesting clarification.

[0156] In such an example, the user device 102 can determine that none of the plurality of medical devices 108 is associated with the preferred IV line architecture based on the type of each of the plurality of medical devices. For example, in response to determining that none of the plurality of medical devices 108 is associated with the preferred IV line architecture, the user device 102 can prompt the user to acquire another image.

[0157] In such an example, the user device 102 can determine that the number of medical devices in the plurality of medical devices 108 is greater than the number of medical devices associated with the preferred IV line architecture based on the type of each medical device in the plurality of medical devices. For example, if the number of identified medical devices is greater than the number required to generate the preferred architecture connection for each lumen or IV line, it means that there may be daisy-chaining or multiple marked components in the image. As an example, if the user device 102 identifies more IV tube sets than catheter lumens, the user device 102 can assign a greater weight to multiple tube to IV tube y-port connections in the probability-based tree building logic. As another example, if the user device 102 identifies a threshold number of caps, the user device 102 can assign a greater weight to a proximity parameter used to detect the distance of a medical device to an open port.

[0158] Reference now Fig. 9 , Fig. 9 is a diagram of examples of atypical but possible connections between medical devices. For example, the connection probability of medical devices of the same category or type may be zero (tubes may be an exception - however, tubes may have secondary tube Y-port labels). For example, an atypical connection between medical devices 108 may be an unexpected and / or purposeless connection between medical devices; however, such a connection may not be incorrect and / or may be potentially initiated by some users. Loose dressings and / or dressing fiducial markers 110 may be left around the catheter insertion site and appear to be close to the tube set or other identified medical devices, and the user device 102 may adjust the weight or prioritize the proximity or distance between the dressing label and the catheter lumen adapter, and the label vector angle between each other in the probability-based tree building logic to determine that these loose fiducial markers or labels are not attached to any medical device in multiple medical devices. Loose caps may float around on a bed or patient near other labeled objects, and if the distance between the cap and another medical device meets the predefined threshold distance, the user device 102 may automatically determine that the cap is not connected to the other medical device. Loose tubing on the bed or patient may be handled in the same or similar manner as loose caps. It is atypical, but possible, for the lumen adapters to be connected to each other, and the user device 102 may determine whether the two adapters are connected to each other based on the distance and / or colinearity between the two adapters.

[0159] In some non-limiting embodiments or aspects, the user device 102 may process the location information and / or type of the plurality of medical devices 108 with a machine learning model to determine the probability that the plurality of pairs of medical devices are connected. For example, the user device 102 may generate a prediction model (e.g., an estimator, a classifier, a prediction model, a detector model, etc.) using a machine learning technique including, for example, supervised and / or unsupervised techniques, such as decision trees (e.g., gradient boosted decision trees, random forests, etc.), logistic regression, artificial neural networks (e.g., convolutional neural networks, etc.), Bayesian statistics, learning automata, hidden Markov modeling, linear classifiers, quadratic classifiers, and / or association rule learning, etc. The prediction machine learning model may be trained to provide an output including a prediction of whether a pair of medical devices are connected to each other. In such an example, the prediction may include the probability (e.g., likelihood, etc.) that the pair of medical devices are connected to each other.

[0160] The user device 102 may generate a prediction model based on the location information associated with each medical device and / or the type of each medical device (e.g., training data, etc.). In some embodiments, the prediction model is designed to receive the location information associated with each medical device in a pair of medical devices (e.g., the proximity between each device, the orientation between each device, the collinearity between each device, etc.) as input, and provide a prediction (e.g., probability, possibility, binary output, yes-no output, score, prediction score, classification, etc.) about whether the pair of medical devices are connected to each other as output. In some non-limiting embodiments or aspects, the user device 102 stores the prediction model (e.g., stores the model for subsequent use). In some non-limiting embodiments or aspects, the user device 102 stores the initial prediction model in a data structure (e.g., a database, a linked list, a tree, etc.). In some non-limiting embodiments, the data structure is located inside the user device 102 or outside the user device 102 (e.g., away from the user device 102) (e.g., in the management system 104, etc.).

[0161] like Figure 4 As shown, at step 408, process 400 includes generating a representation of at least one IV line. For example, user device 102 may generate a representation of at least one IV line including the plurality of pairs of medical devices determined to be connected to each other based on the plurality of pairs of medical devices determined to be connected to each other. As an example, also refer to Fig.10 , Fig.10In a non-limiting embodiment or aspect of a representation of IV lines (e.g., a catheter tree, etc.), the user device 102 can automatically draw lines connecting various medical devices (e.g., catheter tree components, etc.) for each IV line, and / or display identification information associated with each IV line and / or various medical devices 108 in each IV line. In such an example, the user device 102 can automatically draw and display lines on an image of a medical device / catheter insertion site and / or within a series of images (e.g., a live video feed, etc.). For example, the user device 102 can generate a digital representation of each IV line including each pair of medical devices in each IV line based on multiple pairs of medical devices determined to be connected to each other. In such an example, the user device 102 can associate each IV line with a fluid source or pump of an infusion pump and monitor fluid flow in one or more IV lines based at least in part on the representation of the fluid flow path. As an example, the user device 102 can control an audio output device and / or a visual output device to output an audible indication and / or a visual indication, wherein the audible indication and / or the visual indication indicates the status of the IV line and / or the fluid flowing therethrough. For example, the user device 102 can generate a catheter tree or a logical IV branching structure that is mapped onto the physical IV branching structure and includes a unique node identifier for each medical device of the physical IV branching structure, each connector or entry / exit point of a fluid flow path formed by the medical device, and / or each element of the medical device associated with an action that can affect the fluid flow path (e.g., a valve in a medical device).

[0162] In some non-limiting embodiments or aspects, the image acquisition device of the user device 102 can acquire multiple images from multiple different fields of view or positions. Fig.11A , Fig.11A An example catheter tree building sequence is shown, in which an image acquisition device of a user device 102 may use continuous capture technology to acquire a series of images from multiple different fields of view or positions (e.g., a first field of view or position including an insertion site on a patient, a second field of view or position including an infusion pump, etc.). The series of images may include marked and / or unmarked medical devices 108. The user device 102 may continuously integrate and / or combine position information associated with the medical device 108 determined based on each image in the series of images and based on each series of images acquired from each field of view or position, and use the integrated position information to determine multiple pairs of medical devices 108 that are connected to each other, and build and display a catheter tree that includes IV lines formed by multiple pairs of medical devices that are determined to be connected to each other (e.g., a catheter tree including medical devices from an insertion site and / or a dressing tag to an infusion pump or module, etc.). For example, Fig. 11B Fig. 11B It shows that according to Fig.11A An example ductal tree generated by the ductal tree building sequence shown.

[0163] In some non-limiting embodiments or aspects, the user device 102 can compare the medication, medication dose, medication delivery route or IV line, and / or medication delivery time associated with the IV line to an approved patient, approved medication, approved medication dose, approved medication delivery route or IV line, and / or approved medication delivery time associated with a patient identifier and / or medication identifier to reduce medication errors. The user device 102 can issue an alarm and / or cause an infusion pump to stop fluid flow and / or adjust fluid flow based on a current representation of at least one IV line (e.g., based on a current state of a catheter tree, etc.). For example, if a drug arranged for or loaded into an infusion pump at an entry point into a fluid flow path is determined to be an inappropriate drug for the patient, an inappropriate dosage for the patient and / or drug, an inappropriate drug delivery route for the patient and / or drug (e.g., an inappropriate entry point into the fluid flow path), and / or an inappropriate drug delivery time for the patient and / or drug, the user device 102 can issue an alarm and / or control the infusion pump to stop or prevent fluid flow.

[0164] In some non-limiting embodiments or aspects, the user device 102 may determine the dwell time of the medical device 108 (e.g., in the environment 100, etc.) and / or the dwell time of its connection (e.g., the amount of time or duration that the medical device is connected to another medical device and / or patient, etc.). For example, the user device 102 may determine the time when the medical device 108 enters the environment 100 and / or is connected to another medical device and / or patient based on the probability that multiple pairs of medical devices are connected. As an example, the user device 102 may automatically determine and / or record the time when the medical device 108 is connected to another medical device and / or patient, the duration from the connection time to the current time to which the medical device 108 has been connected, and / or the time when the medical device is disconnected from another medical device and / or patient based on multiple pairs of medical devices that are determined to be connected to each other (e.g., over a period of time, over a series of images, over a live video feed, etc.).

[0165] In some non-limiting embodiments or aspects, user device 102 may automatically determine and / or record the frequency with which medical device 108 is connected to another medical device or a particular type of medical device based on multiple pairs of medical devices that are determined to be connected to each other (e.g., over a period of time, over a series of images, over a live video feed, etc.). For example, user device 102 may determine the frequency with which one or more sterilization caps are connected to an IV access port and / or luer tip, etc. and / or the duration that each cap is connected to an IV access port and / or luer tip.

[0166] In some non-limiting embodiments or aspects, the user device 102 may compare the dwell time and / or connection frequency associated with the medical device to a dwell time threshold and / or frequency threshold associated with the medical device and / or the connection including the medical device, and if the dwell time and / or connection frequency meets the dwell time threshold and / or frequency threshold, provide an alert related thereto (e.g., via the user device 102, etc.). For example, the user device 102 may provide an alert indicating that it is time to replace the medical device in the catheter tree with a new medical device and / or that the medical device should be sterilized and / or flushed.

[0167] Reference now Fig. 12A and Fig. 12B , Fig. 12A and Fig. 12B 1 is a flow chart of a non-limiting embodiment or aspect of a process 1200 for estimating a distance of an object from an image acquisition device. In some non-limiting embodiments or aspects, one or more of the multiple steps of process 1200 may be performed by a user device 102 (e.g., one or more devices of a system of user device 102, etc.) (e.g., completely, partially, etc.). In some non-limiting embodiments or aspects, one or more of the multiple steps of process 1200 may be performed by another device or a group of devices (e.g., management system 104 (e.g., one or more devices of management system 104, etc.)) that is separate from or includes user device 102 (e.g., completely, partially, etc.).

[0168] like Fig. 12AAs shown, at step 1202, process 1200 includes acquiring a calibration image including a calibration mark. For example, user device 102 may acquire a calibration image including a calibration mark from an image acquisition device (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.), wherein the calibration mark has a known size and is located at a known distance from the image acquisition device. For example, user device 102 (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.) may acquire a calibration image including a calibration mark, wherein the calibration mark has a known size and is located at a known distance from the image acquisition device of user device 102 (e.g., a known distance from the center of the calibration mark to the center of the image acquisition device, etc.). As an example, the known size of the calibration mark may include a width of the calibration mark, a height of the calibration mark, and / or an area of ​​the calibration mark, etc. In such examples, the calibration image may include a digital image comprising a predetermined number of pixels (e.g., a predetermined pixel height and a predetermined pixel width, a predetermined image resolution, etc.), and / or individual pixels in the calibration image may be associated with a predetermined position or predetermined value in a two-dimensional image space (e.g., a predetermined x, y coordinate of the pixel, etc.).

[0169] The calibration mark may include an object associated with one or more known dimensions (e.g., a known width, a known height, a known area, etc.). For example, the calibration mark may include a medical device 108, and / or a fiducial mark 110 (e.g., AprilTag, etc.), etc. The calibration mark may be positioned parallel to the plane of the image acquisition device used to acquire the calibration image.

[0170] like Fig. 12A As shown, at step 1204, process 1200 includes extracting one or more features associated with the calibration mark in the image space of the calibration image. For example, user device 102 may extract one or more features associated with the calibration mark in the image space of the calibration image. As an example, the one or more features associated with the calibration mark in the image space of the calibration image may include one or more pixel dimensions (e.g., pixel width, pixel height, etc.) of the calibration mark in the image space of the calibration image.

[0171] User device 102 may extract the one or more features associated with the calibration mark in the image space of the calibration image by detecting a region of interest (ROI) in the image that includes the calibration mark using object recognition technology, and calculating the number of pixels in the image space of the calibration image that are associated with the pixel size P of the calibration mark. For example, user device 102 may determine the number of pixels between sides, between edges, and / or between corners of the calibration mark in the image space of the calibration image as the pixel size P (e.g., pixel width, pixel height, etc.) of the calibration mark. As an example, also refer to Fig.13 , Fig.13 The annotated image of the example AprilTag, the user device 102 can use object recognition technology (e.g., Open Source Computer Vision Class Library (OpenCV) etc.) to detect the AprilTag as a calibration marker in the ROI in the calibration image, the calibration marker including the upper right pixel C1 of the AprilTag, the upper left pixel C2 of the AprilTag, the lower left pixel C3 of the AprilTag, and / or the lower right pixel C4 of the AprilTag. For example, the marker detection from the AprilTag library can provide the vertices and tag IDs of the detected marker as C1 to C4, and these vertices C1 to C4 can be used to calculate features, such as a bounding box, pixel distances of the sides of the marker, and / or the centroid of the marker, etc. The user device 102 can determine the right side R of the calibration marker as C1 minus C4 (e.g., R=C1-C4, etc.), and / or determine the left side L of the calibration marker as C2 minus C3 (e.g., L=C2-C3, etc.). User device 102 may determine the pixel size P of the calibration mark as the sum of the right side R of the calibration mark and the left side L of the calibration mark divided by 2 (e.g., P=(L+R) / 2, etc.). Fig.13 In this example case, the pixel size P is the pixel width P.

[0172] like Fig. 12A As shown, at step 1206, process 1200 includes determining one or more calibration parameters based on one or more features associated with the calibration mark in the image space of the calibration image, a known distance of the calibration mark from the image acquisition device, and / or a known size of the calibration mark. For example, the user device 102 may determine one or more calibration parameters based on the one or more features associated with the calibration mark in the image space of the calibration image, the known distance of the calibration mark from the image acquisition device, and / or the known size of the calibration mark. As an example, the one or more calibration parameters may be used for a distance estimation technique associated with the image acquisition device and / or a distance estimation technique configured for the image acquisition device, a predetermined resolution of the image acquisition device for acquiring the calibration image, and / or one or more adjustable camera settings for acquiring the calibration image (e.g., the focal length of the image acquisition device (if adjustable), etc.). In such an example, the user device 102 may store the one or more calibration parameters (e.g., in association with the distance estimation technique, etc.). In some non-limiting embodiments, the one or more calibration parameters may be stored within the user device 102 (e.g., stored in the memory 206, the storage component 208, etc.) or outside the user device 102 (e.g., away from the user device 102) (e.g., stored in the management system 104, etc.).

[0173] The user device 102 may determine the one or more calibration parameters of the distance estimation technique using triangular similarity geometry based on the one or more features associated with the calibration mark in the image space of the calibration image, the known distance of the calibration mark from the image acquisition device, and / or the known size of the calibration mark. For example, the user device 102 may determine the effective focal length F associated with the image acquisition device as the one or more calibration parameters. As an example, the user device 102 may determine the effective focal length F of the image acquisition device as the product of the pixel size P of the calibration mark (e.g., pixel width P, pixel height P, etc.) and the known distance D of the calibration mark from the image acquisition device divided by the known size W of the calibration mark (e.g., known width, known height, etc.) (e.g., F=(P×D) / W, etc.). For example, for a case where the pixel size P of the calibration mark is 59 pixels wide, the known distance D of the calibration mark from the image acquisition device is 30 cm, and the known width of the calibration mark is 1 cm, the effective focal length F of the image acquisition device may be calculated as F=(59×30) / 1=1770 pixels.

[0174] like Fig. 12AAs shown, at step 1208, process 1200 includes acquiring an image including an object. For example, user device 102 may acquire an image including an object from an image acquisition device (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.). As an example, user device 102 (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.) may acquire an image including an object. For example, user device 102 (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.) may acquire an image including an object at step 402 of process 400 and / or as part of another process separate from or including process 400 (e.g., an independent implementation of process 1200 for estimating a distance of an object from an image acquisition device). In such an example, the image acquisition device used to acquire the image including the object may be the same image acquisition device used to acquire the calibration image, may have the same predetermined resolution as the image acquisition device used to acquire the calibration image (e.g., the image may include a digital image containing the same predetermined number of pixels (e.g., the same predetermined pixel height and the same predetermined pixel width, predetermined image resolution, etc.) as the calibration image, etc.), and / or may have the same one or more adjustable camera settings as used to acquire the calibration image (e.g., the focal length of the image acquisition device (if adjustable), etc.); and / or each pixel in the image including the object may be associated with a predetermined position or predetermined value in the same two-dimensional image space as each pixel in the calibration image (e.g., a predetermined x, y coordinate of the pixel, etc.). In such an example, the image acquisition device may be located at a distance from the object used to acquire the image including the object that is different from the known distance of the image acquisition device from the calibration mark used to acquire the calibration image.

[0175] like Fig. 12B As shown, at step 1210, process 1200 includes extracting at least one feature associated with the object in the image space of the image. For example, user device 102 may extract at least one feature associated with the object in the image space of the image. As an example, the at least one feature associated with the object in the image space of the image may include one or more pixel dimensions (e.g., pixel width, pixel height, etc.) of the object in the image space of the image.

[0176] The user device 102 may extract the at least one feature associated with the object in the image space of the image by using object recognition technology to detect a region of interest (ROI) in the image that includes the object, and calculating the number of pixels in the image space of the image that are associated with the pixel size P of the object. For example, the user device 102 may determine the number of pixels between sides, between edges, and / or between corners of the object in the image space of the image as the pixel size P (e.g., pixel width, pixel height, etc.) of the object. As an example, again referring to Fig.13 , Fig.13 is an annotated image of an example AprilTag, and the user device 102 may use object recognition technology (e.g., OpenCV, etc.) to detect the AprilTag as an object in the ROI in the image, the object including the upper right pixel C1 of the AprilTag, the upper left pixel C2 of the AprilTag, the lower left pixel C3 of the AprilTag, and / or the lower right pixel C4 of the AprilTag. The user device 102 may determine the right side R of the calibration mark as C1 minus C4 (e.g., R=C1–C4, etc.), and / or determine the left side L of the calibration mark as C2 minus C3 (e.g., L=C2–C3, etc.). The user device 102 may determine the pixel size P of the calibration mark as the sum of the right side R of the calibration mark and the left side L of the calibration mark divided by 2 (e.g., P=(L+R) / 2, etc.), in Fig.13 In this example case, the pixel size P is the pixel width P.

[0177] like Fig. 12B As shown, at step 1212, process 1200 includes obtaining the size of the object. For example, user device 102 may obtain the size of the object. As an example, the object may include a medical device 108, and / or a fiducial marker 110 (e.g., AprilTag, etc.), etc.

[0178] In some non-limiting embodiments or aspects, an object may be associated with one or more known dimensions (e.g., a known width, a known height, a known area, etc.). For example, user device 102 may obtain the dimensions of an object by using a method as described herein. Figure 4The image including the object may be processed using one or more object detection techniques and / or classifiers described in step 404 to identify the category or type of the object in the image. The user device 102 may store (e.g., in a lookup table, etc.) the dimensions of objects of known categories or types (e.g., the dimensions of the AprilTag, and / or the dimensions of known medical devices 108 (e.g., a particular type of needleless connector, etc.), etc.). For example, the dimensions of objects of known categories or types may be stored within the user device 102 (e.g., stored in the memory 206, the storage component 208, etc.) or outside the user device 102 (e.g., away from the user device 102) (e.g., stored in the management system 104, etc.). In such an example, the user device 102 may retrieve the dimensions of the object (e.g., from a lookup table, etc.) based on the determined category or type of the object in the image to obtain the dimensions of the object.

[0179] In some non-limiting embodiments or aspects, if the user device 102 does not detect a fiducial marker in one or more images and / or does not detect a fiducial marker associated with a detected object in one or more images, the user device 102 may attempt to detect a non-fiducial marker of known size and estimate the size of the detected object based on a first number of pixels associated with the object detected in the image, the known size of the detected non-fiducial marker, and a second number of pixels associated with the detected non-fiducial marker using object recognition techniques (e.g., by using a software package such as OpenCV, and / or ZXing, etc.). In some non-limiting embodiments or aspects, if a non-fiducial marker of known size is not detected in one or more images, the user device 102 may use object recognition techniques to detect other components (e.g., catheter components, etc.) whose sizes are known using object recognition techniques.

[0180] like Fig. 12BAs shown, at step 1214, process 1200 includes determining the distance of the object from the image acquisition device based on at least one feature associated with the object in the image space of the image, the size of the object, and one or more calibration parameters. For example, user device 102 may determine the distance of the object from the image acquisition device (e.g., the distance from the center of the object to the center of the image acquisition device, etc.) based on the at least one feature associated with the object in the image space of the image, the size of the object, and the one or more calibration parameters. As an example, user device 102 may determine the distance D' of the object from the image acquisition device as the product of the size of the object W' (e.g., width, etc.) and the effective focal length F of the image acquisition device divided by the pixel size P' (e.g., pixel width, etc.) of the object in the image, for example, D'=(W'×F) / P'. For example, when the pixel size P' of the object in the image is 35.15 pixels in width (for example, the left side L is 35 pixels and the right side R is 35.3 pixels, etc.), the size of the object is 1 cm in width, and the effective focal length F of the image acquisition device is 1770 pixels, the distance D' of the object from the image acquisition device can be calculated as (1×1770) / 35.15=50.35 cm.

[0181] In some non-limiting embodiments or aspects, the object may include a 3D object that includes a single marker (e.g., a fiducial marker, a non-fiducial marker, etc.) that is used to compare a main surface of the object including the marker with the remaining surface of the object that does not include the marker. For example, the remaining surface of the object that does not include the marker may include an unmarked area or a surface that may be a colored surface or a colorless surface, a textured surface or a flat surface, and / or another surface that is used to distinguish the remaining surface or other surfaces from the main surface in the 2D image. As an example, the user device 102 may determine one or more angles at which the image is captured based on the percentage of the remaining surface visible in the image including the object, and determine the distance of the object from the image acquisition device based on the one or more angles and one or more known dimensions of the main surface including the marker. In such an example, the user device 102 may estimate the distance of the object using a stereo camera. For example, as described in more detail herein, the user device 102 may use a stereo camera to capture images from different perspectives, and the user device 102 may process the images captured from the stereo camera at different perspectives to determine the angular deviation between the objects, and use the angular deviation between the objects and the number of pixels associated with the objects to determine the distance.

[0182] like Fig. 12B As shown, at step 1216, process 1200 includes providing a distance of the object from the image acquisition device. For example, user device 102 may provide a distance of the object from the image acquisition device.

[0183] In some non-limiting embodiments or aspects, the user device 102 can provide the distance of the object from the image acquisition device by displaying the distance of the object from the image acquisition device in real time on the display of the user device 102, which can guide the user holding the user device 102 including the image acquisition device to keep the user device 102 within a specified working distance threshold for acquiring high-quality images. In such an example, the user device 102 can display the real-time distance of the object from the image acquisition device simultaneously with the image acquired by the image acquisition device (e.g., simultaneously with an image for estimating the distance of the object from the image acquisition device, etc.). For example, also refer to Fig.14 , Fig.14 In some non-limiting embodiments or aspects of a display 1400 including distances of multiple medical devices from an image acquisition device, the user device 102 may provide a display 1400 including a real-time view acquired by the image acquisition device, the real-time view including multiple medical devices 108, the distances of the medical devices from the image acquisition device being associated with the multiple medical devices, and simultaneously displayed in the display 1400. In some non-limiting embodiments or aspects, the user device 102 may simultaneously provide the display 1400 with the medical devices 108 as described herein. Figure 4 The at least one IV line described in step 408 includes a plurality of pairs of medical devices determined to be connected to each other.

[0184] In some non-limiting embodiments or aspects, the user device 102 may provide a distance of an object from an image acquisition device for use herein with reference to Figure 4 The one or more other processes and / or calculations described in step 404 of the present invention may be used to determine position information associated with the three-dimensional (3D) position of the plurality of medical devices 108 relative to the image acquisition device, and / or may be used herein to refer to Figure 4 The user device 102 may determine multiple pairs of interconnected medical devices in the plurality of medical devices 108 through one or more of the processes and / or calculations described in step 406. For example, the user device 102 may use the distance of an object from the image acquisition device to determine whether the object is connected to another object in the image. As an example, the object may include a medical device in the plurality of medical devices 108 and / or a reference marker 110 associated with a medical device in the plurality of medical devices 108. In such an example, as described herein, the user device 102 may determine the distance of each of the plurality of medical devices 108 from the image acquisition device, and determine the depth difference between a pair of medical devices from the image acquisition device based on the distance of the pair of medical devices from the image acquisition device, thereby enabling extraction of 3D position information associated with the plurality of medical devices 108.

[0185] In some non-limiting embodiments or aspects, an image including an object is captured using an image capture device (e.g., as referred to herein). Figure 4 Step 402 of process 400 and / or Fig. 12A 1208 of process 1200 in step 1204) may include using the same image acquisition device (e.g., the same digital monocular camera, etc.) to acquire multiple images at different angles relative to one or more objects in the image. For example, the multiple images may be Figure 4 12 and / or step 1214 of process 1200 in FIG. 12 to calculate object distance and / or 3D position information of one or more objects, for example by using a multi-view depth estimation process that utilizes triangulation to calculate 3D position information of one or more objects.

[0186] Reference now Fig.15 , Fig.15 15 is a flow chart of a non-limiting embodiment or aspect of a process 1500 for estimating a distance of an object from an image acquisition device. In some non-limiting embodiments or aspects, one or more of the multiple steps of process 1500 may be performed by a user device 102 (e.g., one or more devices of a system of user device 102, etc.) (e.g., completely, partially, etc.). In some non-limiting embodiments or aspects, one or more of the multiple steps of process 1500 may be performed by another device or a group of devices (e.g., management system 104 (e.g., one or more devices of management system 104, etc.)) that is separate from or includes user device 102 (e.g., completely, partially, etc.).

[0187] like Fig.15 As shown, at step 1502, process 1500 includes using an image acquisition device to acquire a first image including the object at a first angle relative to the object. For example, user device 102 (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.) can acquire a first image including the object at a first angle relative to the object.

[0188] like Fig.15 As shown, at step 1504, process 1500 includes using an image acquisition device to acquire a second image including the object at a second angle relative to the object, the second angle being different from the first angle. For example, user device 102 (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.) can acquire a second image including the object at a second angle relative to the object, the second angle being different from the first angle. As an example, also refer to Fig.16, relative to an object that may be located at a point p in a 3D space, the image acquisition device may translate from a position C1 where a first image is acquired to a position C2 where a second image is acquired.

[0189] like Fig.15 As shown, at step 1506, process 1500 includes detecting a plurality of first interest points associated with the object in the first image and a plurality of second interest points associated with the object in the second image. For example, user device 102 may use an image feature detector algorithm to detect a plurality of first interest points associated with the object in the first image and a plurality of second interest points associated with the object in the second image. As an example, user device 102 may use one or more image feature detector algorithms as described in a 2016 paper titled “Image Features Detection, Description and Matching” by Hassaballah, M., A AAbdelmgeid, and Hammam A.Alshazly (hereinafter referred to as Hassaballah et al.), the entire contents of which are incorporated herein by reference.

[0190] like Fig.15 As shown, at step 1508, process 1500 includes generating a plurality of first descriptor vectors associated with the plurality of first interest points and a plurality of second descriptor vectors associated with the plurality of second interest points. For example, user device 102 may use an image feature descriptor algorithm, a plurality of first descriptor vectors associated with the plurality of first interest points, and a plurality of second descriptor vectors associated with the plurality of second interest points. As an example, user device 102 may use one or more image feature descriptor algorithms as described by Hassaballah et al.

[0191] like Fig.15 As shown, at step 1510, process 1500 includes matching at least one first point of interest in the plurality of first points of interest with at least one second point of interest in the plurality of second points of interest. For example, user device 102 may use a feature matching algorithm to match at least one first point of interest in the plurality of first points of interest with at least one second point of interest in the plurality of second points of interest based on the plurality of first descriptor vectors associated with the plurality of first points of interest and the plurality of second descriptor vectors associated with the plurality of second points of interest. As an example, user device 102 may use one or more feature matching algorithms as described in Hassaballah et al.

[0192] like Fig.15As shown, at step 1512, process 1500 includes determining position information associated with a three-dimensional (3D) position of the object relative to the image acquisition device. For example, user device 102 may determine position information associated with a three-dimensional (3D) position of the object relative to the image acquisition device based on at least one first interest point in the plurality of first interest points that matches at least one second interest point in the plurality of second interest points. As an example, user device 102 may use the referenced Figure 4 12 and / or step 1214 of process 1200 in FIG. 12 , to calculate object distance and / or 3D position information of one or more objects, for example by using a multi-view depth estimation process that uses triangulation of one or more first interest points matched with one or more second interest points to calculate 3D position information of one or more objects.

[0193] Therefore, non-limiting embodiments or aspects of the present disclosure may use predefined or known detected marker features and geometric and / or triangulation properties to identify ROIs and estimate the distance between an object (e.g., a fiducial marker, etc.) and an image acquisition device, and / or apply image processing and mathematical logic to estimate the distance, which provides real-time calculation, estimation, and output of distance at a lower computational cost than existing convolutional neural network (CNN)-based methods for better usability, and uses the same single camera setting to simulate a stereo setting to capture multi-view images, thereby achieving triangulation through Detect-Describe-Match pipelines, which can reduce sensor load and avoid expensive stereo camera calibration calculations.

[0194] Reference now Fig.17 , Fig.17 17 is a flow chart of a non-limiting embodiment or aspect of a process 1700 for estimating a yaw angle of an object from an image acquisition device. In some non-limiting embodiments or aspects, one or more of the multiple steps of process 1700 may be performed by a user device 102 (e.g., one or more devices of a system of user device 102, etc.) (e.g., completely, partially, etc.). In some non-limiting embodiments or aspects, one or more of the multiple steps of process 1700 may be performed by another device or a group of devices (e.g., management system 104 (e.g., one or more devices of management system 104, etc.)) that is separate from or includes user device 102 (e.g., completely, partially, etc.).

[0195] like Fig.17As shown, at step 1702, process 1700 includes acquiring an image including a fiducial marker. For example, user device 102 may acquire an image including a fiducial marker 110 (e.g., a fiducial marker 110 attached to medical device 108, etc.) from an image acquisition device (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.). As an example, user device 102 (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.) may acquire an image including a fiducial marker 110 (e.g., a fiducial marker 110 attached to medical device 108, etc.). For example, user device 102 (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.) may acquire an image including a fiducial marker 110 at step 402 of process 400 and / or as part of another process separate from or including process 400 (e.g., an independent implementation of process 1700 for estimating a yaw angle of an object from an image acquisition device). In such an example, the fiducial marker may have a known width and a known height. For example, the fiducial marker may include an AprilTag.

[0196] like Fig.17 As shown, process 1700 includes extracting the pixel width and pixel height of the fiducial marker from the image at step 1704. For example, user device 102 may extract the pixel width and pixel height of fiducial marker 110 in image space of the image.

[0197] User device 102 may detect a region of interest (ROI) including fiducial marker 110 in the image by using object recognition techniques, and calculate the number of pixels in the image space of the image associated with the pixel width and pixel height of fiducial marker 110 to extract the pixel width and pixel height of the fiducial marker from the image. For example, user device 102 may determine the number of pixels between sides, between edges, and / or between corners of an object in the image space of the image as the pixel size P (e.g., pixel width, pixel height, etc.) of the object. As an example, referring again to Fig.13 , Fig.131 is an annotated image of an example AprilTag, and the user device 102 may use object recognition technology (e.g., OpenCV, etc.) to detect the AprilTag as an object in the ROI in the image, the object including the upper right pixel C1 of the AprilTag, the upper left pixel C2 of the AprilTag, the lower left pixel C3 of the AprilTag, and / or the lower right pixel C4 of the AprilTag. The user device 102 may determine the right side R of the reference marker 110 as C1 minus C4 (e.g., R=C1-C4, etc.), and / or determine the left side L of the reference marker 110 as C2 minus C3 (e.g., L=C2-C3, etc.). The user device 102 may determine the pixel width P of the reference marker 110 as W Determined as the sum of the right side R of the calibration mark and the left side L of the calibration mark divided by 2 (eg, P W =(L+R) / 2, etc.). User device 102 may determine the top side T of fiducial marker 110 as C2 minus C1 (e.g., T=C2-C1, etc.), and / or determine the bottom side B of fiducial marker 110 as C3 minus C4 (e.g., B=C3-C4, etc.). User device 102 may determine the pixel height P of fiducial marker 110 as h Determined as the sum of the top side of the fiducial marker 110 and the bottom side of the fiducial marker 110 divided by 2 (eg, P h =(T+B) / 2, etc.).

[0198] like Fig.17 As shown, at step 1706, process 1700 includes determining a yaw angle of the fiducial marker relative to the image acquisition device. For example, user device 102 may estimate the yaw angle of the fiducial marker relative to the image acquisition device based on the pixel width, pixel height, and the known width and known height. As an example, user device 102 may estimate the yaw angle of the fiducial marker relative to the image acquisition device according to the following equation (1):

[0199] Yaw angle = α*β (1)

[0200] Where β is a multiplication factor and α is determined according to the following equation (2):

[0201] α=((w k / h k )–(w p / h p ))*90 (2)

[0202] where w k is the known width of the fiducial mark, h k is the known height of the fiducial marker, w p is the pixel width of the fiducial marker in the image space of the image, and h pis the pixel height of the fiducial marker in the image space of the image. In some non-limiting embodiments or aspects, the known width w of the fiducial marker is k The known height h from the reference mark k For example, the known width w of the fiducial mark is k The known height h of the AprilTag used as a reference marker k The ratio is one to one.

[0203] According to the pixel width w p With pixel height h p The multiplication factor β (eg, a weight factor, etc.) is determined from a plurality of pre-stored multiplication factors. For example, the user device 102 may retrieve the value of the multiplication factor β from a lookup table, which includes a weight factor according to the pixel width w. p With pixel height h p As an example, Table 1 below includes the ratio of the width and height of the label for the multiplication factor β calculated for labels at different angles and distances. GT in Table 1 refers to the ground truth.

[0204]

[0205]

[0206] Table 1

[0207] like Fig.17 As shown, at step 1708, process 1700 includes providing a yaw angle of the fiducial marker relative to the image acquisition device. For example, user device 102 may provide a yaw angle of fiducial marker 110 relative to the image acquisition device.

[0208] In some non-limiting embodiments or aspects, the user device 102 may provide the yaw angle of the fiducial marker 110 relative to the image acquisition device by displaying the yaw angle of the fiducial marker 110 relative to the image acquisition device in real time on the display of the user device 102, which may guide the user holding the user device 102 including the image acquisition device to keep the user device 102 within a specified working angle threshold for acquiring high-quality images. In such an example, the user device 102 may display the real-time yaw angle of the fiducial marker 110 deviating from the image acquisition device simultaneously with the image acquired by the image acquisition device (e.g., simultaneously with an image used to estimate the distance of the fiducial marker 110 from the image acquisition device, etc.). For example, the user device 102 may provide a display including a real-time view acquired by the image acquisition device, the real-time view including a plurality of medical devices 108 having corresponding fiducial markers 110, the distances of these fiducial markers from the image acquisition device and / or the yaw angles of these fiducial markers from the image acquisition device being displayed simultaneously in the display in association with the plurality of medical devices 108. In some non-limiting embodiments or aspects, the user device 102 may simultaneously provide the display 1400 with the information provided herein as referenced in the accompanying drawings. Figure 4 The at least one IV line described in step 408 includes a plurality of pairs of medical devices determined to be connected to each other.

[0209] In some non-limiting embodiments or aspects, the user device 102 may provide a yaw angle of the fiducial marker 110 from the image acquisition device for reference herein. Figure 4 One or more other processes and / or calculations described in step 404 of the present invention determine position information associated with the three-dimensional (3D) position of the plurality of medical devices 108 and / or their corresponding fiducial markers 110 relative to the image acquisition device, and / or for use herein with reference to Figure 4 The one or more processes and / or calculations described in step 406 of the embodiment of the present invention can be used to determine multiple pairs of medical devices in the plurality of medical devices 108 that are connected to each other. For example, the user device 102 can use the yaw angle of the fiducial marker 110 away from the image acquisition device and / or the yaw angle relative to another fiducial marker 110 to determine whether the medical device 108 including the fiducial marker 110 is connected to another medical device 108 in the image that includes another fiducial marker 110. As an example, the fiducial marker 110 can be associated with a medical device in the plurality of medical devices 108, and the user device 102 can determine the yaw angle of each medical device in the plurality of medical devices 108 away from the image acquisition device as described herein, thereby enabling the extraction of 3D position information associated with the plurality of medical devices 108.

[0210] Reference now Fig.18 , Fig.1818 is a flowchart of a non-limiting embodiment or aspect of a process 1800 for estimating a roll angle of an object from an image acquisition device. In some non-limiting embodiments or aspects, one or more of the multiple steps of process 1800 may be performed by a user device 102 (e.g., one or more devices of a system of user device 102, etc.) (e.g., completely, partially, etc.). In some non-limiting embodiments or aspects, one or more of the multiple steps of process 1800 may be performed by another device or a group of devices (e.g., management system 104 (e.g., one or more devices of management system 104, etc.)) that is separate from or includes user device 102 (e.g., completely, partially, etc.).

[0211] like Fig.18 As shown, at step 1802, process 1800 includes acquiring an image including a fiducial marker. For example, user device 102 may acquire an image including a fiducial marker 110 (e.g., a fiducial marker 110 attached to medical device 108, etc.) from an image acquisition device (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.). As an example, user device 102 (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.) may acquire an image including a fiducial marker 110 (e.g., a fiducial marker 110 attached to medical device 108, etc.). For example, user device 102 (e.g., an image acquisition device of user device 102, a monocular digital camera of user device 102, etc.) may acquire an image including a fiducial marker 110 at step 402 of process 400 and / or as part of another process separate from or including process 400 (e.g., an independent implementation of process 1700 for estimating a yaw angle of an object from an image acquisition device). In such an example, the fiducial marker may have a known width and a known height. For example, the fiducial marker may include an AprilTag.

[0212] like Fig.18 As shown, at step 1804, process 1800 includes extracting a centroid associated with the fiducial marker in image space of the image. For example, user device 102 may extract a centroid associated with fiducial marker 110 in image space of the image.

[0213] User device 102 may detect a region of interest (ROI) in the image that includes the calibration marker by using object recognition techniques and counting the number of pixels in the image space of the calibration image that are associated with the pixel size P of fiducial marker 110 to extract one or more features associated with fiducial marker 110 in the image space of the image. For example, user device 102 may determine the number of pixels between sides, between edges, and / or between corners of fiducial marker 110 in the image space of the image. As an example, also refer to Fig.19A , Fig.19A An annotated image of a square or rectangle representing the area between the corners of the AprilTag in a clockwise order, the user device 102 can use object recognition technology (e.g., OpenCV, etc.) to detect the AprilTag as a fiducial marker 110 in the ROI in the image, the fiducial marker including the upper left pixel C1 of the AprilTag, the upper right pixel C2 of the AprilTag, the lower right pixel C3 of the AprilTag, and / or the lower left pixel C4 of the AprilTag. For example, the marker detection from the AprilTag library can provide the vertices and tag IDs of the detected marker as C1 to C4, and these corners C1 to C4 can be used to calculate features, such as a bounding box, pixel distances of the sides of the marker, and / or the centroid of the marker, etc. The user device 102 can determine the centroid of the AprilTag as being at a distance of half of its height and half of its bottom or width. For example, user device 102 may determine the y centroid coordinate of AprilTag as C1 minus c4 divided by 2 (e.g., Y_bar=H / 2=(C1–C4) / 2, etc.), and / or determine the x centroid coordinate of AprilTag as C3 minus C4 divided by 2 (e.g., X_bar=B / 2=(C3–C4) / 2, etc.).

[0214] like Fig.18 As shown, at step 1806, process 1800 includes determining a position vector of the fiducial marker based on the centroid of the fiducial marker in the image space of the image. For example, user device 102 may determine the position vector of fiducial marker 110 based on the centroid of fiducial marker 110 in the image space of the image, a known width of the fiducial marker, and / or a known height of the fiducial marker. As an example, user device 102 may use the centroid to calculate the position vector x, y, and / or z of fiducial marker 110 by calculating the Euclidean distance between the centroid of fiducial marker 110 in the image space of the image and the midpoint of one side (e.g., X_bar or W / 2, Y_bar or H / 2, etc.), which may provide pixel lengths or distances and / or directions of the x, y, and z position vectors.

[0215] Reference again Fig.19A, the user device 102 can calculate the orientation axis and center for each of the angles C1 to C4. Fig.19B , Fig.19B is formed Fig.19A A coordinate diagram of the points of a square or rectangle representing the area between the corners of the AprilTag, for corners C1 to C4 ordered in a clockwise direction, in order to calculate the x position vector and the y position vector to identify the direction of the AprilTag, the points forming the square or rectangle on the x-axis and the y-axis can be used. As an example, the center or centroid calculation may include the average of the individual points (C1, C2, C3, C4) corresponding to each corner of the AprilTag, in Fig.19B In the example, it can be [(0,1)+(1,1)+(1,0)+(0,0)] / 4=(0.5,0.5). Different points (e.g., C1–C4 and C2–C3) can point to the projection on the y axis. For example, the y position vector or y vector (yVec) can include the sum of the differences of the points, Fig.19B In the example, it can be [(0,1)-(0,0)]+[(1,1)-(1,0)]=(0,2). Similarly, different points (such as C2-C1 and C3-C4) can point to the projection on the x-axis. For example, the x position vector or x vector (xVec) can include the sum of the differences of the points, Fig.19B In the example, it can be [(1,1)-(0,1)]+[(1,0)-(0,0)]=(2,0).

[0216] Also refer to Fig.19C , Fig.19C yes Fig.19A and Fig.19B To plot the coordinates of the orientation axis of the AprilTag in the image, the user device 102 may stack the center calculation value, the y position vector or vVec, and the x position vector or xVec in an N×2×3 array (e.g., in a tagCore array data structure to return a combined array index by index (where [:,:,0] = center, [:,:,1] = yVec, and [:,:,2] = xVec), and plot the corresponding yVec and xVec by adding a center point to the image (e.g., cv2.line(im,center+yVec.astype('int'),center,(255,0,0),int(2*im.shape[0] / 1000)), cv2.line(im,center+xVec.astype('int'),center,(0,0,255),int(2*im.shape[0] / 1000)), etc.). For example, Fig.19CAs shown, user device 102 may be between (center+yVec) and the center of the y orientation (e.g., between Fig.19C In the example of , a line is drawn between (0.5,2.5) and (0.5,0.5)); and / or a line is drawn between (center+xVec) and the center of the x-position (e.g., Fig.19C In the example, it is the line between (2.5,0.5) and (0.5,0.5).

[0217] like Fig.18 As shown, at step 1808, process 1800 includes determining a slope of a position vector of a fiducial marker relative to a coordinate base axis of an image including the fiducial marker. For example, user device 102 may determine a slope of a position vector of fiducial marker 110 relative to a coordinate base axis of an image including fiducial marker 110. As an example, also refer to Fig. 20A , Fig. 20A The coordinate base axes of the image including the AprilTag are shown, and the x position vector or xVec can be compared with the x axis of the image including the fiducial marker 110. In such an example, the slope of the position vector of the fiducial marker 110 relative to the coordinate base axes of the image including the fiducial marker 110 can be determined according to the following equation (3):

[0218] Slope = (y2 –y1) / (x2 – x1) (3)

[0219] Where x1, y1 are the coordinates of a first point at the first end of a line or position vector, and x2, y2 are the coordinates of a second point at the second end of a line or position vector (e.g., between the center of mass and the y position vector point, between the center of mass and the x position vector point, etc.).

[0220] For example, also refer to Fig. 20B , for an example scene of a tilted AprilTag detected using an x ​​position vector of (3.0,2.6) and a y position vector of (2.6,2.0) and the centroid at (2.5,2.5), Fig. 20B An example x and y position vector of the AprilTag relative to the coordinate base axis of the image is shown. To calculate the slope, the centroid and y position vector can be considered such that (2.5, 2.5) = (x2, y2) and (2.6, 2.0) = (x1, y1), resulting in a slope = -5.

[0221] like Fig.18As shown, at step 1810, process 1800 includes determining a roll angle of the fiducial marker relative to the image acquisition device based on a slope of a position vector of the fiducial marker. For example, user device 102 may determine a roll angle of fiducial marker 110 relative to the image acquisition device based on a slope of a position vector of the fiducial marker relative to a coordinate base axis of the image. As an example, the roll angle of fiducial marker 110 relative to the image acquisition device may be determined according to the following equation (4):

[0222] Roll angle = tan-1(slope) (4)

[0223] For example, refer again to Fig. 20B For the example scenario of a tilted AprilTag detected using an x ​​position vector of (3.0, 2.6), a y position vector of (2.6, 2.0), a center of mass at (2.5, 2.5), and a calculated slope = -5, the roll angle can be calculated as roll angle = 78.7 degrees.

[0224] like Fig.18 As shown, at step 1812, process 1800 includes providing a roll angle of the fiducial marker relative to the image acquisition device. For example, user device 102 may provide a roll angle of fiducial marker 110 relative to the image acquisition device.

[0225] In some non-limiting embodiments or aspects, the user device 102 may provide the roll angle of the fiducial marker 110 relative to the image acquisition device by displaying the roll angle of the fiducial marker 110 deviating from the image acquisition device in real time on the display of the user device 102, which may guide the user holding the user device 102 including the image acquisition device to keep the user device 102 within a specified working angle threshold for acquiring high-quality images. In such an example, the user device 102 may display the real-time roll angle of the fiducial marker 110 deviating from the image acquisition device simultaneously with the image acquired by the image acquisition device (e.g., simultaneously with an image used to estimate the distance of the fiducial marker 110 from the image acquisition device, etc.). For example, the user device 102 may provide a display including a real-time view acquired by the image acquisition device, the real-time view including a plurality of medical devices 108 having corresponding fiducial markers 110, the distances of the fiducial markers from the image acquisition device, the yaw angles from the image acquisition device, and / or the roll angles from the image acquisition device being associated with the plurality of medical devices 108 being displayed simultaneously in the display. In some non-limiting embodiments or aspects, the user device 102 may simultaneously provide to the display 1400 the display 1400 as referred to herein. Figure 4 The at least one IV line described in step 408 includes a plurality of pairs of medical devices determined to be connected to each other.

[0226] In some non-limiting embodiments or aspects, the user device 102 may provide a roll angle of the fiducial marker 110 relative to the image acquisition device for use herein with reference to Figure 4 One or more other processes and / or calculations described in step 404 of the present invention determine position information associated with the three-dimensional (3D) position of the plurality of medical devices 108 and / or their corresponding fiducial markers 110 relative to the image acquisition device, and / or for use herein with reference to Figure 4 The one or more processes and / or calculations described in step 406 of the embodiment of the present invention can be used to determine multiple pairs of medical devices in the plurality of medical devices 108 that are connected to each other. For example, the user device 102 can use the roll angle of the fiducial marker 110 relative to the image acquisition device and / or the roll angle relative to another fiducial marker 110 to determine whether the medical device 108 including the fiducial marker 110 is connected to another medical device 108 including another fiducial marker 110 in the image. As an example, the fiducial marker 110 can be associated with a medical device in the plurality of medical devices 108, and the user device 102 can determine the roll angle of each medical device in the plurality of medical devices 108 from the image acquisition device as described herein, thereby enabling the extraction of 3D position information associated with the plurality of medical devices 108.

[0227] In some non-limiting embodiments or aspects, the user device 102 may determine and / or provide a distance from the fiducial marker 110 to the image acquisition device, a yaw angle of the fiducial marker 110 relative to the image acquisition device, and / or a roll angle of the fiducial marker 110 relative to the image acquisition device. For example, the user device 102 may provide the distance from the fiducial marker 110 to the image acquisition device, the yaw angle of the fiducial marker 110 relative to the image acquisition device, and / or the roll angle of the fiducial marker 110 relative to the image acquisition device by displaying the distance from the fiducial marker 110 to the image acquisition device, the yaw angle of the fiducial marker 110 relative to the image acquisition device, and / or the roll angle of the fiducial marker 110 relative to the image acquisition device simultaneously with an image including the fiducial marker 110 on a display. As an example, the user device 102 may provide the distance of the fiducial marker 110 from the image acquisition device, the yaw angle of the fiducial marker 110 relative to the image acquisition device, and the roll angle of the fiducial marker 110 relative to the image acquisition device in the following manner: Use the distance of the fiducial marker 110 from the image acquisition device, the yaw angle of the fiducial marker 110 relative to the image acquisition device, and the roll angle of the fiducial marker 110 relative to the image acquisition device to determine whether the medical device associated with the fiducial marker 110 is connected to another medical device in the image (the other medical device may be associated with another fiducial marker).

[0228] Although various embodiments or aspects have been described in detail for purposes of illustration and description, it will be understood that such details are for purposes of illustration and description only, and that the embodiments or aspects are not limited to the disclosed embodiments or aspects, but rather, the embodiments or aspects are intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it will be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment or aspect. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed may be directly dependent on only one claim, the disclosure of possible implementations includes the combination of each dependent claim with each other claim in a group of claims.

Claims

1. A system comprising: at least one processor coupled to the memory and programmed and / or configured to: Acquire an image including a fiducial marker, the image being acquired by an image acquisition device, wherein the fiducial marker has a known width and a known height; extracting a pixel width and a pixel height of the fiducial marker in image space of the image; estimating a yaw angle of the fiducial marker relative to the image acquisition device based on the pixel width, the pixel height, and the known width and the known height; and The yaw angle of the fiducial marker relative to the image acquisition device is provided.

2. The system according to claim 1, wherein: The at least one processor is programmed and / or configured to estimate the yaw angle of the fiducial marker relative to the image acquisition device according to the following equation: Yaw angle = α*β Where β is the multiplication factor and α is determined according to the following equation: α=((w k / h k )–(w p / h p ))*90 where w k is the known width of the fiducial mark, h k is the known height of the fiducial mark, w p is the pixel width of the fiducial marker in the image space of the image, and h p is the pixel height of the fiducial marker in the image space of the image.

3. The system according to claim 2, wherein: The multiplication factor β is based on the pixel width w p With the pixel height h p The ratio is determined from multiple pre-stored multiplication factors.

4. The system according to claim 2, wherein: The known width w of the fiducial mark k The known height h of the reference mark k The ratio is one to one.

5. The system according to claim 1, wherein: The reference marker includes an AprilTag.

6. A method comprising: acquiring, using at least one processor, an image including a fiducial marker, the image acquired by an image acquisition device, wherein the fiducial marker has a known width and a known height; extracting, using at least one processor, a pixel width and a pixel height of the fiducial marker in image space of the image; estimating, using at least one processor, a yaw angle of the fiducial marker relative to the image acquisition device based on the pixel width, the pixel height, and the known width and the known height; and The at least one processor is used to provide the yaw angle of the fiducial marker relative to the image acquisition device.

7. The method according to claim 6, wherein: The at least one processor estimates the yaw angle of the fiducial marker relative to the image acquisition device according to the following equation: Yaw angle = α*β Where β is the multiplication factor and α is determined according to the following equation: α=((w k / h k )–(w p / h p ))*90 where w k is the known width of the fiducial mark, h k is the known height of the fiducial mark, w p is the pixel width of the fiducial marker in the image space of the image, and h p is the pixel height of the fiducial marker in the image space of the image.

8. The method according to claim 7, wherein: The multiplication factor β is based on the pixel width w p With the pixel height h p The ratio is determined from multiple pre-stored multiplication factors.

9. The method according to claim 7, wherein: The known width w of the fiducial mark k The known height h of the reference mark k The ratio is one to one.

10. The method according to claim 6, wherein: The reference marker includes an AprilTag.

11. A system comprising: at least one processor coupled to the memory and programmed and / or configured to: Acquire an image including a fiducial marker, the image being acquired by an image acquisition device, wherein the fiducial marker has a known width and a known height; extracting a centroid of the fiducial marker in image space of the image; determining a position vector of the fiducial marker based on the centroid, the known width, and the known height; determining a slope of the position vector of the fiducial marker relative to a coordinate base axis of the image; determining a roll angle of the fiducial marker relative to the image acquisition device based on the slope of the position vector of the fiducial marker relative to a coordinate base axis of the image; and The roll angle of the fiducial marker relative to the image capture device is provided.

12. The system according to claim 11, wherein: The at least one processor is programmed and / or configured to estimate the slope of the position vector of the fiducial marker relative to a coordinate base axis of the image according to the following equation: Slope = (y2–y1) / (x2–x1) Where x1, y1 are the coordinates of a first point at a first end of the position vector, and x2, y2 are the coordinates of a second point at a second end of the position vector.

13. The system according to claim 12, wherein: The at least one processor is programmed and / or configured to determine the roll angle of the fiducial marker relative to the image acquisition device based on the slope of the position vector of the fiducial marker relative to the coordinate base axis of the image according to the following equation: Roll angle = tan-1 (slope).

14. The system according to claim 11, wherein: The reference marker includes an AprilTag.

15. The system of claim 14, wherein: The centroid of the fiducial marker in the image space of the image comprises an average of points at the corners of the AprilTag, wherein the position vector comprises the sum of differences of multiple points on an axis of the image.

16. A method comprising: acquiring, using at least one processor, an image including a fiducial marker, the image acquired by an image acquisition device, wherein the fiducial marker has a known width and a known height; extracting, using the at least one processor, a centroid of the fiducial marker in image space of the image; determining, using the at least one processor, a position vector of the fiducial marker based on the centroid, the known width, and the known height; determining, using the at least one processor, a slope of the position vector of the fiducial marker relative to a coordinate base axis of the image; determining, using the at least one processor, a roll angle of the fiducial marker relative to the image acquisition device based on the slope of the position vector of the fiducial marker relative to a coordinate base axis of the image; and Using the at least one processor, providing the roll angle of the fiducial marker relative to the image acquisition device.

17. The method according to claim 16, wherein: The at least one processor estimates the slope of the position vector of the fiducial marker relative to a coordinate base axis of the image according to the following equation: Slope = (y2–y1) / (x2–x1) Where x1, y1 are the coordinates of a first point at a first end of the position vector, and x2, y2 are the coordinates of a second point at a second end of the position vector.

18. The method according to claim 17, wherein: The at least one processor determines the roll angle of the fiducial marker relative to the image acquisition device based on the slope of the position vector of the fiducial marker relative to the coordinate base axis of the image according to the following equation: Roll angle = tan-1 (slope).

19. The method according to claim 16, wherein: The reference marker includes an AprilTag.

20. The method according to claim 16, wherein: The centroid of the fiducial marker in the image space of the image comprises an average of points at the corners of the AprilTag, wherein the position vector comprises the sum of differences of multiple points on an axis of the image.

Images