Improved drug preparation method and system using imaging technology

Abstract

A system for determining the presence of fluid or air in a transparent tube in a drug preparation device is provided. [Solution] The system includes a processing circuit (PC) configured to receive a camera image of the transparent tube, where the transparent tube is positioned in front of a patterned background from the camera's perspective, whereby one or more regions of the transparent tube are depicted in the received image, each depicted region being associated with a respective refractive index of the patterned background; identify one or more of the regions of the transparent tube in the received image using image processing techniques; and determine characteristics of the contents of the transparent tube from at least one or more of the identified regions.

Description

[Technical Field]

[0001] The subject matter of this disclosure relates to the use of imaging in automation, and in particular to the implementation of a system for validation of pharmaceutical compounding devices (PCDs). [Background technology]

[0002] The problem of packaging in PCDs has been recognized in the prior art, and various techniques have been developed to provide a solution to it.

[0003] (General Description) 1. A system for determining a volume of fluid drawn into a syringe, comprising: a first processing circuit operably connectable to a camera configured to photograph the syringe from a first distance; and a second processing circuit including a processor and a memory; The first processing circuit a) receiving a digital image of the syringe from the camera; b) in the resulting digital image using image processing, a closed syringe hard stop; configured to determine data indicative of a pixel distance between a top of the syringe piston inserted into the syringe; The second processing circuit a) receiving a value indicative of an amount of fluid aspirated into the syringe from a fluid aspirating subsystem configured to retract a plunger arm inserted into the syringe; b) in response to the calculated fluid volume differing from the fluid volume received from the fluid aspiration subsystem by an amount that meets an alert threshold; Issue a warning, Thereby, a system is provided that is configured to non-gravimetrically verify the amount of fluid drawn into the syringe.

[0004] According to one aspect of the presently disclosed subject matter, there is provided a computer system for determining the presence of fluid or air in a transparent tube in a drug preparation device, the computer system comprising: a) receiving a camera image of the transparent tube, the transparent tube being positioned in front of a patterned background from a camera perspective; receiving, whereby the received image depicts one or more regions of the transparent tube, each depicted region being associated with a respective refractive index of the patterned background; b) using image processing techniques to identify one or more of said regions of said transparent tube in said received image; and c) determining a characteristic of the contents of said transparent tube from one or more of said identified regions.

[0005] In addition to the features described above, systems according to this aspect of the presently disclosed subject matter may include one or more of the following listed features (i) through (vi), in any desired combination or permutation that is technically possible: (i) The PC is further configured to determine a characteristic of the contents based on the refractivity associated with the one or more of the identified regions, thereby providing an indication of the type of contents present in the transparent tube. (ii) The PC determines whether the associated refractivity of one of the identified regions matches the refractivity characteristics of the air within the transparent tube. (iii) The PC determines whether the associated refractive index of one of the identified regions matches the refractive index characteristics of the liquid within the transparent tube. (iv) The PC is further configured to determine a volume of the region of the transparent tube based on a pixel dimension of one of the identified regions of the image. (v) the PC is further configured to determine a volume of a gap between regions of the transparent tube according to a pixel distance between a first identified region of the image and a second identified region of the image. (vi) the PC is further configured to issue an alert in response to a refractive index associated with one of the identified regions not matching an expected refractive index of the fluid.

[0006] According to another aspect of the presently disclosed subject matter, there is provided a computer-implemented method for determining the presence of liquid or air in a transparent tube in a drug preparation device, the method comprising: a) receiving a camera image of the transparent tube, the transparent tube being positioned in front of a patterned background from a camera perspective; receiving, whereby the received image depicts one or more regions of the transparent tube, each depicted region being associated with a respective refractive index of the patterned background; b) using image processing techniques to identify one or more of said regions of said transparent tube in said received image; and c) determining a characteristic of the contents of said transparent tube from at least one or more of said identified regions.

[0007] This aspect of the presently disclosed subject matter may optionally further include one or more of features (i)-(vi) listed above with respect to the system, in any desired combination or permutation technically possible, mutatis mutandis.

[0008] According to another aspect of the presently disclosed subject matter, there is provided a computer program product including a computer-readable non-transitory storage medium containing program instructions that, when read by a processor, cause the processing circuitry to perform a method for determining the presence of fluid or air in a transparent tube in a pharmaceutical preparation device, the method comprising: a) receiving a camera image of the transparent tube, the transparent tube being positioned in front of a patterned background from a camera perspective; receiving, whereby the received image depicts one or more regions of the transparent tube, each depicted region being associated with a respective refractive index of the patterned background; b) using image processing techniques to identify one or more of said regions of said transparent tube in said received image; and c) determining a characteristic of the contents of said transparent tube from at least one or more of said identified regions.

[0009] This aspect of the presently disclosed subject matter may optionally further include one or more of features (i)-(vi) listed above with respect to the system, in any desired combination or permutation technically possible, mutatis mutandis.

[0010] According to another aspect of the presently disclosed subject matter, there is provided a robotic system for aspirating fluid from a fluid container into a syringe, the system comprising: a) controlling the fluid aspiration subsystem to withdraw a plunger arm of a syringe operably connected to a first fluid container by a withdrawal distance; b) receiving a digital image of the syringe; c) in response to detecting bubbles or voids in the captured digital image using image processing techniques; The plunger arm is A robotic system is provided that includes a processing circuit (PC) configured to control the fluid aspiration subsystem to push a pushing distance according to the size of the bubble or void determined based on the captured image.

[0011] In addition to the features described above, systems according to this aspect of the presently disclosed subject matter may include one or more of the following listed features (i) through (xii), in any desired combination or permutation technically possible. (i) The PC further: d) controlling the fluid suction subsystem to depress the plunger arm; configured to repeat a) through c) for one or more additional iterations; in each iteration, said control of said fluid aspiration subsystem withdraws said plunger arm a respective incremental withdrawal distance until said plunger arm is withdrawn a full withdrawal distance from said syringe; The full withdrawal distance is determined based on the syringe fill volume required. (ii) The PC further: d) in response to the fluid container being empty, controlling the container exchange subsystem to exchange the first fluid container for a second fluid container; e) repeating a) through c) for one or more additional iterations; in each iteration, said control of said fluid aspiration subsystem withdraws said plunger arm a respective incremental withdrawal distance until said plunger arm is withdrawn a full withdrawal distance from said syringe; The full withdrawal distance is determined based on the syringe fill volume required. (iii) The PC further: receiving a digital image of the syringe after aspiration; using image processing techniques to estimate the volume of fluid from the digital image after the aspiration; in response to the estimated fluid volume differing from the required syringe fill volume by an amount that exceeds a fluid volume tolerance; It is configured to issue a warning. (iv) The withdrawal distance is the full withdrawal distance determined based on the syringe fill volume required. (v) the PC further controls the actuator to move the plunger arm the pull-out distance; The fluid aspiration subsystem is configured to control the fluid aspiration subsystem to retract the plunger arm. (vi) The PC further: The device is configured to calculate the pushing distance based on at least a pixel diameter of the bubble or void in the captured digital image. (vii) The PC is further configured to reduce a speed of an actuator moving the plunger arm in response to the number of consecutively detected bubbles or voids satisfying a threshold number of consecutively detected bubbles. (viii) The PC is further configured to issue an alert that the fluid container is misconfigured in response to the number of consecutively detected bubbles or voids satisfying a threshold number of consecutively detected bubbles. (ix) the digital image is taken with the syringe positioned in front of a patterned background from a camera's perspective; The received image thereby depicts one or more regions of the syringe, each depicted region being associated with a respective refractive index of the patterned background. The above PC further: a. using image processing techniques to identify one or more of said regions of said syringe in said captured image; b. detecting air bubbles or voids in the captured digital image by determining the presence of air bubbles or voids in the syringe based at least on one or more of the identified regions. (x) the PC is further configured to determine the presence of the bubble or void based on the refractivity associated with the one or more of the identified regions. (xi) The empty fluid container state is based on the PC performing image processing on an image of the fluid container. (xii) The fluid container empty condition is based on the PC maintaining the fluid container fill level.

[0012] According to another aspect of the presently disclosed subject matter, there is provided a computer-implemented method for aspirating fluid from a fluid container into a syringe, the method comprising: a) controlling a fluid aspiration subsystem to withdraw a plunger arm of a syringe operably connected to a first fluid container by a withdrawal distance; b) receiving a digital image of the syringe; and c) in response to detecting bubbles or voids in the captured digital image using image processing techniques; The plunger arm is pushed in a distance corresponding to the size of the bubble or void determined based on the captured image. and controlling the fluid aspiration subsystem.

[0013] This aspect of the presently disclosed subject matter may optionally further include one or more of features (i)-(xii) listed above with respect to the system, in any desired combination or permutation technically possible, mutatis mutandis.

[0014] According to another aspect of the presently disclosed subject matter, there is provided a computer program product including a computer-readable non-transitory storage medium containing program instructions that, when read by a processor, cause the processing circuitry to perform a method of aspirating fluid from a fluid container into a syringe, the method comprising: a) controlling a fluid aspiration subsystem to withdraw a plunger arm of a syringe operably connected to a first fluid container by a withdrawal distance; b) receiving a digital image of the syringe; and c) in response to detecting bubbles or voids in the captured digital image using image processing techniques; The plunger arm is pushed in a distance corresponding to the size of the bubble or void determined based on the captured image. and controlling the fluid aspiration subsystem.

[0015] This aspect of the presently disclosed subject matter may optionally further include one or more of features (i)-(xii) listed above with respect to the system, in any desired combination or permutation technically possible, mutatis mutandis.

[0016] According to one aspect of the presently disclosed subject matter, there is provided a computer system for verifying the aspiration of a required amount of fluid into a syringe, the computer system comprising: a processing circuit (PC) operably connectable to a plunger holder configured to withdraw a plunger inserted into the syringe, the syringe being operably connected to a fluid container, the PC comprising: a) controlling the plunger holder to withdraw the plunger a first plunger withdrawal distance based on the required amount, thereby moving the plunger holder from a first plunger holder position to a second plunger holder position; b) a computer system configured to receive a digital image of the syringe taken after said withdrawal, and to use image processing techniques on said digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid.

[0017] In addition to the features described above, systems according to this aspect of the presently disclosed subject matter may include one or more of the following listed features (i): (i) The PC further states that, before a), configured to receive a first digital image of the plunger holder at the first plunger holder position; The PC furthermore, after b), in response to the current amount of fluid in the syringe not matching the required amount of fluid; a. receiving a second digital image of the plunger holder at the second plunger holder position; b. using image processing techniques on the first digital image of the plunger holder and the second digital image of the plunger holder to determine whether the second position of the plunger holder corresponds to the first position of the plunger holder and the first plunger withdrawal distance; c. in response to the second position of the plunger holder being consistent with the first position of the plunger holder and the first plunger travel distance; The syringe is configured to issue a warning that the syringe has been misconfigured.

[0018] According to another aspect of the presently disclosed subject matter, there is provided a computer-implemented method for verifying aspiration of a required amount of fluid into a syringe, the method comprising: a) controlling the plunger holder to withdraw the plunger a first plunger withdrawal distance based on a required amount, thereby moving the plunger holder from a first plunger holder position to a second plunger holder position; b) receiving a digital image of the syringe taken after said withdrawal; and using image processing techniques on said digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid.

[0019] This aspect of the presently disclosed subject matter may optionally further include features (i) listed above with respect to the system, mutatis mutandis, in any desired combination or permutation technically possible.

[0020] According to another aspect of the presently disclosed subject matter, there is provided a computer program product including a computer-readable non-transitory storage medium containing program instructions that, when read by a processor, cause the processing circuitry to perform a method for verifying aspiration of a required amount of fluid into a syringe, the method comprising: a) controlling the plunger holder to withdraw the plunger a first plunger withdrawal distance based on a required amount, thereby moving the plunger holder from a first plunger holder position to a second plunger holder position; b) receiving a digital image of the syringe taken after said withdrawal, and using image processing techniques on said digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid.

[0021] This aspect of the presently disclosed subject matter may optionally further include features (i) listed above with respect to the system, mutatis mutandis, in any desired combination or permutation technically possible.

[0022] According to one aspect of the presently disclosed subject matter, there is provided a computer system for verifying the aspiration of a required amount of fluid into a syringe, the computer system comprising: a processing circuit (PC) operably connectable to a plunger holder configured to withdraw a plunger inserted into the syringe, the syringe being operably connected to a fluid container, the PC comprising: a) controlling the plunger holder to depress the plunger a first plunger depression distance based on the required amount, thereby moving the plunger holder from the first plunger holder position to a second plunger holder position; b) a computer system configured to receive a digital image of the syringe taken after the depression of the plunger, and to use image processing techniques on the digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid remaining.

[0023] In addition to the features described above, systems according to this aspect of the presently disclosed subject matter may include one or more of the following listed features (i): i) The PC further states that, before a), configured to receive a first digital image of the plunger holder at the first plunger holder position; The PC furthermore, after b), in response to the current amount of fluid in the syringe not matching the required remaining amount of fluid; a. receiving a second digital image of the plunger holder at the second plunger holder position; b. applying image processing techniques to the first digital image of the plunger holder and the second digital image of the plunger holder to determine whether the second position of the plunger holder corresponds to the first position of the plunger holder and the first plunger depression distance; c. in response to the second position of the plunger holder being consistent with the first position of the plunger holder and the first plunger depression distance; The syringe is configured to issue a warning that the syringe has been misconfigured.

[0024] According to another aspect of the presently disclosed subject matter, there is provided a computer-implemented method for verifying aspiration of a required amount of fluid into a syringe, the method comprising: a) controlling the plunger holder to depress the plunger a first plunger insertion distance based on a required amount, thereby moving the plunger holder from a first plunger holder position to a second plunger holder position; b) receiving a digital image of the syringe taken after the depression of the plunger, and using image processing techniques on the digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required remaining amount of fluid.

[0025] This aspect of the presently disclosed subject matter may optionally further include feature (i) listed above with respect to the system.

[0026] According to another aspect of the presently disclosed subject matter, there is provided a computer program product including a computer-readable non-transitory storage medium containing program instructions that, when read by a processor, cause the processing circuitry to perform a method for verifying aspiration of a required amount of fluid into a syringe, the method comprising: a) controlling the plunger holder to depress the plunger a first plunger insertion distance based on a required amount, thereby moving the plunger holder from a first plunger holder position to a second plunger holder position; b) receiving a digital image of the syringe taken after the depression of the plunger, and using image processing techniques on the digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required remaining amount of fluid.

[0027] This aspect of the presently disclosed subject matter may optionally further include feature (i) listed above with respect to the system. [Brief explanation of the drawings]

[0028] In order to understand the invention and to see how it may be carried out in practice, embodiments will now be described, by way of non-limiting example, with reference to the accompanying drawings, in which:

[0029] [Figure 1A] FIG. 1 shows a block diagram of an exemplary prior art PCD using gravimetric verification of aspirated fluid volume. [Figure 1B] FIG. 1 shows a block diagram of an exemplary prior art PCD using gravimetric verification of aspirated fluid volume. [Figure 2] 1 shows a block diagram of an exemplary PCD with imaging-based confirmation of aspirated fluid volume, according to some embodiments of the presently disclosed subject matter. [Figure 3A] 1 illustrates an exemplary syringe, fluid container, and associated components, according to some embodiments of the presently disclosed subject matter. [Figure 3B] 1 illustrates an exemplary syringe manipulator, according to some embodiments of the presently disclosed subject matter. [Figure 4A] 1 shows a flowchart of an example of an imaging-based method for verifying the amount of fluid aspirated by a PCD, according to some embodiments of the presently disclosed subject matter. [Figure 4B]1 shows a flowchart of an exemplary method for imaging-based aspirated fluid volume confirmation and syringe misconfiguration status determination by PCD, according to some embodiments of the presently disclosed subject matter. [Figure 5A] 1 shows a flowchart of an example of a method for mitigating bubbles or voids that occur during fluid aspiration by a PCD based on imaging, according to some embodiments of the presently disclosed subject matter. [Figure 5B] 1 shows a flowchart of an example of a method for mitigating bubbles or voids that occur during fluid aspiration by a PCD based on imaging, according to some embodiments of the presently disclosed subject matter. [Figure 5C] 10 shows a flowchart of another example of a method for mitigating bubbles or voids that occur during fluid aspiration by a PCD based on imaging, according to some embodiments of the presently disclosed subject matter. [Figure 6A] 1 illustrates an exemplary pharmaceutical preparation device (PCD), according to some embodiments of the presently disclosed subject matter. [Figure 6B] 1 illustrates an exemplary syringe assembly, according to some embodiments of the presently disclosed subject matter. [Figure 6C] 6 shows an example of a syringe manipulator comprising an engagement arm 655, a gripping arm 665, and a plunger holder 675 configured to engage with a plunger arm 666 of a syringe, according to some embodiments of the presently disclosed subject matter. [Figure 6D] 6 shows an example of a transparent (non-opaque) tube 650 according to some embodiments of the presently disclosed subject matter. [Figure 7] 1 illustrates exemplary subsystems of a pharmaceutical compounding device (PCD), according to some embodiments of the presently disclosed subject matter. [Figure 8A] 1 illustrates an exemplary transparent tube according to some embodiments of the presently disclosed subject matter. [Figure 8B] 1 illustrates an exemplary transparent tube in front of a refractive pattern background, according to some embodiments of the presently disclosed subject matter. [Figure 9] 1 shows a flowchart of an exemplary method for identifying fluid or air within a region of a transparent tube from a captured image. [Figure 10A] 1 shows a flowchart of an exemplary method of drug preparation in a PCD, according to some embodiments of the presently disclosed subject matter, where refractive index-based fluid discrimination is used in the process. [Figure 10B] 10 shows a flowchart of an additional exemplary method of drug preparation in a PCD, in which refractive index-based fluid discrimination is used in the process, according to some embodiments of the presently disclosed subject matter. [Figure 10C] 10 shows a flowchart of an additional exemplary method of drug preparation in a PCD, in which refractive index-based fluid discrimination is used in the process, according to some embodiments of the presently disclosed subject matter. [Figure 11A] 1 shows a flowchart of an exemplary method of drug preparation harmonization process in a PCD resulting in a primed syringe, according to some embodiments of the presently disclosed subject matter. [Figure 11B] 1 shows a flowchart of an exemplary method of drug preparation reconciliation process in a PCD resulting in a prepared infusion bag, according to some embodiments of the presently disclosed subject matter. DETAILED DESCRIPTION OF THE INVENTION

[0030] In the following embodiments of the present disclosure, numerous specific details are described to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the subject matter of the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the subject matter of the present disclosure.

[0031] Unless otherwise indicated, as will be apparent from the discussion that follows, discussions throughout the specification utilizing terms such as "processing," "operating," "comparing," "determining," "calculating," "receiving," "providing," "obtaining," "detecting," and the like refer to computer actions and / or processes that manipulate and / or transform data into other data, said data being expressed as physical quantities, e.g., electronic quantities, and / or representing objects. The term "computer" should be interpreted broadly to encompass any type of hardware-based electronic device having data processing capabilities, including, for example, as non-limiting examples, the processors, mitigation units, and inspection units disclosed in the present application.

[0032] As used herein, "non-transitory storage" and "non-transitory storage medium" should be interpreted broadly to encompass volatile or non-volatile computer memory suitable for the subject matter of this disclosure. Operations according to the teachings herein may be performed by a computer specially constructed for the desired purpose, or by a general-purpose computer specially constructed for the desired purpose by a computer program stored on a non-transitory computer-readable storage medium.

[0033] Embodiments of the presently disclosed subject matter are not described with reference to any particular programming language. A variety of programming languages ​​may be used to implement the teachings of the presently disclosed subject matter as described herein.

[0034] Referring to FIG. 1A, FIG. 1 shows an example of a prior art robotic injection preparation system for the filling stage.

[0035] Fluid aspiration subsystem 110 may be a subsystem that aspirates fluid from fluid container 145 into syringe barrel 125 .

[0036] The fluid aspiration subsystem 110 can include a processor-based controller (not shown) that can control the mechanical plunger arm controller 160 to aspirate fluid from the fluid container 145 into the syringe barrel 125 by withdrawing the plunger arm 115 from the syringe barrel 125.

[0037] The syringe tip 165 of the syringe 125 can be inserted into the fluid container 145, for example, by a needle or other type of connector that penetrates the fluid container. The syringe piston 105 can be inserted into the syringe 125. When the components of the syringe piston 105 and plunger arm 115 are fully inserted, the syringe piston 105 can contact the closed syringe hard stop 155. The plunger arm 115 of the syringe piston 105 can be mechanically withdrawn from within the syringe 125.

[0038] System controller 170 may be a separate controller including processor 130A and memory 140A. System controller 170 may be operatively connected (e.g., via a bus or network) to fluid aspiration subsystem 110. System controller 170 may instruct fluid aspiration subsystem 110 to aspirate a particular amount (e.g., volume) of fluid (such as a medication to be injected into a patient), for example, into syringe 125.

[0039] The weighing system 180 may be a separate station within the PCD.

[0040] FIG. 1B illustrates an exemplary prior art robotic injection preparation system during a fill accuracy verification step that may occur after the fill step.

[0041] A mechanism such as a robotic arm (not shown) can move the syringe 125 from the fluid aspiration subsystem 110 to the weighing system 180. The weighing system 180 may include a digital scale or other suitable mechanism for measuring the mass of the filled syringe. The system controller 170 can be operatively connected to the weighing system 180 and can receive the measured mass value of the syringe 125. The system controller 170 can then issue an alert in situations where the measured mass of the filled syringe does not match an expected value.

[0042] Referring to FIG. 2, FIG. 2 illustrates an exemplary robotic drug preparation system with optical process monitoring, according to some embodiments of the presently disclosed subject matter.

[0043] Fluid aspiration subsystem 210 may be a subsystem that aspirates fluid from fluid container 245 into syringe 225. Fluid aspiration subsystem 210 may also inject fluid from syringe 225 into container 245.

[0044] Fluid aspiration subsystem 210 can include a processor-based controller (not shown) that can control a mechanical plunger arm controller 260 (also referred to herein as a plunger arm holder or plunger holder) to aspirate fluid from fluid container 245 into the body of syringe 225 by withdrawing plunger arm 215 (also referred to herein as a plunger) from syringe 225. In some embodiments, plunger arm controller 260 can form part of a syringe manipulator that is configured to move the syringe (e.g., as a whole) and / or move particular components of the syringe, such as the plunger arm, e.g., as further described with reference to FIG. 3B below.

[0045] The tip 265 of the syringe 225 can be inserted into the fluid container 245 by a needle or other connector that pierces or otherwise fluidly connects the container. The syringe piston 205 can be inserted into the syringe barrel 225. When the components of the syringe piston 205 and plunger arm 215 are fully inserted, the syringe piston 205 can contact the closed syringe hard stop 255. The plunger arm 215 of the syringe piston 205 can be mechanically withdrawn from within the syringe barrel 225.

[0046] System controller 270 may be an independent controller including processor 230B and memory 240B. System controller 270 may be operatively connected (e.g., via a bus or network) to fluid aspiration subsystem 210. System controller 270 may instruct fluid aspiration subsystem 210 to aspirate a particular amount (e.g., volume) of fluid (such as a medication to be injected into a patient), for example, into syringe 225.

[0047] Camera 290 may be a digital camera configured to optically monitor and control fluid aspiration by fluid aspiration system 210. Camera 290 may be positioned to take a digital image of syringe 225 from a particular distance, referred to herein as camera distance 235. Additionally or alternatively, camera 290 may be positioned to take a digital image of a syringe manipulator (or parts thereof, such as a plunger holder), regardless of the presence of a syringe. In other words, camera 290 may be configured to take images of replaceable (e.g., disposable) components, such as syringes, and / or permanent components of a machine, such as a syringe manipulator.

[0048] In some embodiments, camera 290 is located at a fixed position relative to syringe 225. In some embodiments, camera 290 can be moved manually or automatically to different positions or distances relative to syringe 225. In some other embodiments, the camera position can be fixed and the syringe can be moved manually or automatically (e.g., by a syringe manipulator) into the field of view of camera 290.

[0049] The camera controller 220 may be operatively connected to the camera 290 and may include a processor 230A and a memory 240A. The camera controller may apply camera control methods and provide digital images to the system controller 270.

[0050] System controller 270 may be operatively connected to camera 290 and may include processor 230B and memory 240B. The system controller may apply a system control method such as that described below with reference to Figures 4-5. System controller 270 may include processing circuitry (not shown), which may include processor 230B and memory 240B.

[0051] Processor 230B may be any suitable hardware-based electronic device having data processing capabilities, such as, for example, a general-purpose processor, a digital signal processor (DSP), a specialized application-specific integrated circuit (ASIC), one or more cores in a multi-core processor, etc. Processor 230B may also be comprised of, for example, multiple processors, multiple ASICs, virtual processors, combinations thereof, etc.

[0052] Memory 240B may be, for example, any suitable type of volatile and / or non-volatile storage device and may include, for example, a single physical memory component or multiple physical memory components. Memory 240B may also include virtual memory. Memory 240B may be configured to store various data used, for example, in computations.

[0053] The processing circuitry may be configured to execute several functional modules in accordance with computer-readable instructions embodied in a non-transitory computer-readable storage medium. Such functional modules are hereinafter referred to as included in the processing circuitry. These modules may include, for example, an image processing unit 275.

[0054] Referring to FIG. 3A, FIG. 3A shows an exemplary image of a syringe configured to aspirate fluid (eg, via a fluid aspirating subsystem) according to some embodiments of the presently disclosed subject matter.

[0055] 3A shows the fluid reservoir 345 (245) and the syringe components: plunger arm 315 (215), syringe piston 305 (205), syringe barrel 325 (255), and closed syringe hard stop 355 (255). The syringe tip is inserted into a connector that receives fluid from the fluid reservoir 345 (245). Therefore, the syringe tip is not visible in this image.

[0056] In FIG. 3A, the syringe is not drawing fluid, so the syringe piston 305 is in contact with the closed syringe hard stop 355.

[0057] In some embodiments of the presently disclosed subject matter, system controller 270 processes an image including a fluid container and / or syringe, such as the image shown in FIG. 3A, to determine data indicative of the amount of fluid in the syringe.

[0058] FIG. 3B illustrates an exemplary syringe manipulator according to some embodiments of the presently disclosed subject matter.

[0059] Gripper 365 may be a component of a syringe manipulator that grips, for example, a syringe barrel, a syringe hub, or a syringe adapter (connector).

[0060] The plunger holder 375 may be a component of the syringe manipulator that grips the syringe plunger arm 315. The plunger holder 375 may be shaped with a recess that receives the flange of the plunger arm.

[0061] The teachings of the presently disclosed subject matter are not limited to the robotic drug preparation system with optical process monitoring and related components described with reference to Figures 2, 3A, and 3B. Equivalent and / or modified functionality may be integrated or divided in other ways and may be implemented in any suitable combination of software and firmware and / or software and hardware and executed on any suitable device.

[0062] Reference is now made to FIG. 4A, which illustrates an exemplary imaging-based method for verifying accuracy of robotic aspiration of fluid into a syringe, according to some embodiments of the presently disclosed subject matter.

[0063] The system controller 270 (e.g., image processing unit 275) may receive (step 410A) a digital image of the syringe 225 (e.g., when partially or completely filled with fluid aspirated by the fluid aspiration subsystem 210). The digital image may be taken by the camera 290 and may be taken at a camera distance 235 away from the syringe 225. In some examples, the syringe tip 265 is not visible in the received digital image.

[0064] The system controller 270 (eg, image processing unit 275) can process the received digital image (step 430A) to determine the amount of fluid in the syringe.

[0065] In some embodiments, the system controller 270 (e.g., the image processing unit 275) compares an image of the syringe with the piston (or plunger arm) in an initial position with an image of the syringe with the piston (or plunger arm) in a subsequent position. The system controller 270 can then determine the distance in pixels from the original piston (or plunger arm) position to the subsequent piston (or plunger arm) position.

[0066] System controller 270 can then estimate the volume by utilizing the determined pixel distance in combination with a pixel-to-volume value associated with the syringe type (e.g., syringe bore diameter), image resolution, and camera distance 235. In some embodiments, system controller 270 can perform perspective correction before determining the pixel distance. In some embodiments, system controller 270 can perform correction for nonlinear distortions due to lenses before determining the pixel distance.

[0067] In some embodiments, the system controller 270 (eg, image processing unit 275) identifies the position of the piston in the image and determines the distance (in pixels) from the top of the cylinder to the fixed point.

[0068] In some embodiments, the system controller 270 (eg, image processing unit 275) identifies graduations on the syringe to directly determine the volume.

[0069] In some embodiments, the system controller 270 (eg, image processing unit 275) identifies the position of the syringe handle to determine its withdrawal range.

[0070] In some embodiments, the system controller 270 (eg, the image processing unit 275) determines the fluid volume using another suitable method.

[0071] In some embodiments, the system controller 270 determines the pixel-to-volume values ​​during a calibration procedure in which, for example, the system controller 270 receives one or more images of a syringe along with associated volume values ​​determined, for example, by human input.

[0072] Using system controller 270 and camera controller 220, the PCD system can perform safety checks on fluid aspiration by fluid aspiration subsystem 210. In this manner, in some embodiments, the PCD system can avoid the inclusion of a weighing system and its associated costs. Furthermore, by using image-based injection confirmation, the need to move and weigh the syringe is eliminated, reducing injection preparation time and increasing throughput of the PCD system.

[0073] More specifically, the system controller 270 may obtain data (e.g., from a user interface or a remote control system) indicating the amount of fluid required for injection. In some embodiments, it is contemplated that the system controller 270 controls the fluid aspiration subsystem 210 to retract the plunger arm 215 and thereby aspiration the fluid.

[0074] If the estimated fluid volume (i.e., the fluid volume obtained from the image) differs from the required volume (step 440A), and if the difference meets a volume error threshold, the system controller 270 may (optionally) generate an alert (e.g., a sound, a message on a control console, stopping further processing, etc.) (step 450A).

[0075] In some embodiments, instead of or in addition to issuing an alert, the system controller 270 can, for example, control the fluid aspiration subsystem 210 to correct the volume and then repeat the imaging-based confirmation process (step 460A).

[0076] Reference is now made to FIG. 4B, which illustrates another exemplary method for verifying the accuracy of robotic syringe manipulation based on imaging, according to some embodiments of the presently disclosed subject matter.

[0077] The method described in FIG. 4B is applicable to both drawing fluid into a syringe from an operably connected fluid container (such as a vial or drip bag) and injecting fluid from a syringe into an operably connected fluid container.

[0078] The system controller 270 (e.g., image processing unit 275) may capture and receive (step 405B) a digital image of the plunger holder 375 and / or the plunger arm held by the holder in an initial position (e.g., before the start of a fluid injection or fluid withdrawal operation). The digital image may be captured by the camera 290 and may be taken at a camera distance 235 away from the syringe 225. In some examples, the syringe tip 265 is not visible in the received digital image.

[0079] System controller 270 (e.g., image processing unit 275) may then send a signal to fluid aspiration system 260 (e.g., an actuator / linear driver (e.g., servo motor) controlling the movement of the syringe manipulator), causing it to move a specific distance (thereby moving plunger holder 375 and plunger arm 315) (step 410B). For example, system controller 270 (e.g., image processing unit 275) may move plunger holder 375 to move the plunger arm a specific plunger withdrawal distance relative to syringe barrel 325 (e.g., to draw a required amount of fluid into the syringe), or a specific injection distance into syringe barrel 325 (e.g., to inject the entire contents of the syringe into, for example, a vial or an IV bag).

[0080] In response to an indication from fluid aspiration system 260 (e.g., a driver) that the requested syringe manipulator movement is complete, system controller 270 (e.g., image processing unit 275) may then capture and receive digital images of syringe 225 and / or syringe manipulator or portions thereof (step 415B). Note that other operations may occur between the initiation and completion of plunger arm movement (e.g., air bubble detection and mitigation).

[0081] The system controller 270 (e.g., image processing unit 275) can then use image processing to verify that the amount of fluid in the syringe is correct (e.g., using the methods described above with reference to FIG. 4A) (step 420B).

[0082] It should be noted that when the entire contents of the syringe have been injected, the syringe can be expected to be empty, e.g., the estimated amount of fluid in the syringe is expected to be zero, whereby the syringe piston 105 contacts or is close to the closed syringe hard stop 155.

[0083] If the volume indicated by the image processing is accurate, eg, matches (within a threshold) the expected volume associated with the plunger travel, then syringe operation is confirmed (step 425B).

[0084] If the volume indicated by the image processing is not accurate, e.g., differs from the expected volume associated with the amount of plunger movement (by more than a threshold), system controller 270 (e.g., image processing unit 275) may then receive a digital image of the plunger holder 375 and / or plunger arm at their current positions (step 430B). In some examples, this digital image may be the same as the digital image showing syringe 225 after the syringe operation described above is completed (i.e., an image showing both syringe 225 and plunger holder 375 taken after syringe aspirate / syringe injection).

[0085] The system controller 270 (e.g., image processing unit 275) can then, for example, evaluate the position of the plunger holder and / or plunger arm in the first and second images and, through image processing, determine that the plunger holder and / or plunger arm has moved the plunger withdrawal distance or plunger injection distance (as appropriate) (step 435B).

[0086] If the plunger holder and / or plunger arm move the required amount, it can be assumed that the aspiration or injection failure is due to a defective or misconfigured syringe. Thus, the system controller 270 (e.g., image processing unit 275) can issue a "syringe misconfigured" warning (step 440B).

[0087] As used herein, a "misconfigured syringe" may refer to a syringe that is not properly aligned with the fluid container (e.g., vial, IV bag) from which fluid is to be transferred, a syringe that is not properly gripped by the syringe manipulator, a syringe that is not properly connected to a syringe adapter, a defective or deformed syringe, or other condition that may prevent fluid transfer to or from the syringe. When a "misconfigured syringe alert" occurs, the alert may be resolved automatically by the system (e.g., in the case of a misaligned syringe, the system may attempt to reconnect the syringe to the fluid container) or may be resolved by an operator of the system (e.g., by the operator replacing a defective syringe).

[0088] 5A, which illustrates an imaging-based method for removing air bubbles (including bubbles caused by end-of-life conditions of the container) in robotic injection preparation, according to some embodiments of the presently disclosed subject matter. In some embodiments, the method allows for full utilization of the fluid container during syringe filling, and also allows for bubble-free completion of filling from a newly filled fluid container.

[0089] In some embodiments, the system controller 270 maintains data indicating the current amount of fluid held in the current fluid container 245 (e.g., vial, IV bag, etc.) being aspirated by the fluid aspiration subsystem. The system controller 270 may do this, for example, by receiving data regarding the amount of fluid initially contained in the fluid container 245 at the start of aspiration of fluid from the fluid container 245. The system controller 270 may then update the data indicating the current amount of fluid, for example, each time it controls the filling of a syringe.

[0090] The system controller 270 may begin this process by determining the plunger arm extension distance for syringe filling (step 510A).

[0091] In some examples, the system controller 270 determines this distance depending on the syringe fill volume required, and therefore the fluid aspiration system 210 will withdraw the plunger arm 215 a certain distance (e.g., in millimeters) from the syringe barrel 225, thereby causing the syringe barrel 225 to be filled with the full volume (e.g., in milliliters) of fluid required as part of the injection preparation.

[0092] However, if the current amount of fluid in fluid container 245 is less than the required syringe fill amount (a situation referred to herein as a "container underfill condition"), system controller 270 can utilize a plunger arm withdrawal distance that is dependent on (e.g., consistent with) the amount of fluid remaining in the fluid container. In this case, system controller 270 can then control fluid aspiration subsystem 210 to withdraw any remaining fluid from fluid container 245, remove any resulting air bubbles / voids, and then control fluid aspiration subsystem 210 to replace the vial and complete the syringe fill, as described in the following steps. Thus, in this example, the air bubbles / voids are used as an indicator to detect when the remaining amount of fluid in the fluid container has been withdrawn.

[0093] The system controller 270 may then control the fluid aspiration subsystem 210 to withdraw the plunger arm 215 to the determined plunger arm withdrawal distance (step 520A).

[0094] Next, system controller 270 may receive a digital image of syringe 225 (step 530A), which may be, for example, a photograph taken by camera 290 at camera distance 235.

[0095] System controller 270 may then use image processing methods on the received digital image to determine whether any air bubbles or voids are present in the syringe (step 540A).

[0096] In some embodiments, if the system controller 270 does not detect an air bubble or void when it extends the plunger arm a plunger arm extension distance corresponding to the amount of fluid remaining in the fluid container 245, the system controller 270 can extend the plunger arm a next extension distance and attempt to detect the air bubble or void again.

[0097] In some embodiments (e.g., when the syringe 225 is oriented with the syringe tip 265 pointing upward), an air bubble is typically a quasi-spherical region of air within the fluid, while a void is typically a fluid-free region adjacent the closed syringe hard stop 255. Additionally, the system controller 270 can use appropriate image processing techniques to identify and analyze the air bubbles and / or voids.

[0098] If air bubbles or voids are indeed present, the system controller 270 can estimate the volume of the air bubbles or voids in the syringe. If no air bubbles or voids are present, the syringe filling operation is complete.

[0099] The presence of air bubbles or voids can also make it difficult to obtain the desired volume of fluid, as the air bubbles themselves occupy a certain volume in the syringe.

[0100] The system controller 270 may then calculate a reinsertion distance of the plunger arm according to the estimated volume of the air bubble or void (step 550A). As used herein, "reinsertion" and "reinsertion distance" may refer to pushing (advancing) the plunger arm toward the closed syringe hard stop. Optionally, reinsertion occurs after the plunger arm is withdrawn (pulled away) from the closed syringe hard stop.

[0101] In some embodiments, the system controller 270 determines the reinsertion distance of the plunger arm from the pixel height of the void and a given pixel-to-distance value (i.e., the physical distance (e.g., in millimeters) of a single pixel at the current camera resolution and camera distance 235). In some embodiments, the system controller 270 uses a reinsertion distance of the plunger arm when a void is detected that is equal to the product of the pixel height and the given pixel-to-distance value.

[0102] In some other embodiments, the system controller 270 determines the reinsertion distance of the plunger arm by first calculating the volume of air in the void from the pixel height and a given pixel-to-volume value (i.e., the volume (e.g., in milliliters) in the syringe associated with a single pixel at the current camera resolution and camera distance 235). In some embodiments, the system controller 270 uses a volume of air equal to the product of the pixel height, the inner diameter of the syringe 225, and the given pixel-to-distance value. The system controller 270 can then determine the reinsertion distance from the estimated volume, for example, using the inner diameter of the syringe 225.

[0103] Exemplary methods for receiving and deriving pixel-to-distance and pixel-to-volume values ​​have been described above.

[0104] The system controller 270 may then reinsert the plunger arm 215 a determined reinsertion distance sufficient to remove the air bubble (step 560A).

[0105] In some instances, removing the air bubble in other ways, such as by fully reinserting the plunger arm 215 into the syringe 225, may damage the syringe.

[0106] For example, in response to early detection of a container underfill condition, the system controller 270 may control the container exchange subsystem 210 to replace the (now empty) fluid container 245 with a new, filled fluid container 245 (step 570A). In some embodiments, the container exchange subsystem may include a robotic arm or other suitable manipulator configured to engage an empty fluid container, move the empty container to a designated location (e.g., a waste tray), and engage a new container (e.g., from a container tray, container conveyor, etc.) and place it in fluid communication with the syringe.

[0107] System controller 270 can now return to step 510A to complete syringe filling and perform another bubble / void check.

[0108] In some embodiments, the system controller 270 records the number of consecutive air bubble detections when filling the syringe 225. If this number meets a given threshold of consecutive detected air bubbles, the system controller 270 can issue a warning (e.g., on a console, using a particular sound, etc.) that the fluid container is misconfigured, indicating that a misconfiguration may have occurred that allowed air to enter the syringe.

[0109] By aspirating the original fluid container 245 last and aspirating fluid from the new fluid container 245 first as described above (i.e., aspirating air and removing it before replacing the fluid container), waste is prevented and the possibility of spills is reduced.

[0110] Now, referring to FIG. 5B, FIG. 5B shows an imaging-based method for removing air bubbles (including air bubbles generated by the end-of-life condition of the container) during robotic injection preparation, according to some embodiments of the presently disclosed subject matter.

[0111] In some embodiments, the method allows for full utilization of the fluid container during syringe filling, thereby increasing efficiency while reducing waste, maintenance, and potential operator exposure to hazardous materials. In some embodiments, the method further allows for bubble-free completion of the fill from a newly filled fluid container.

[0112] In some embodiments, the system controller 270 continuously maintains data indicating the current volume of fluid held in the current fluid container 245 (e.g., vial, intravenous (IV) bag, etc.) being aspirated by the fluid aspiration subsystem. The system controller 270 can do this, for example, by receiving data regarding the volume of fluid initially contained in the fluid container 245 at the start of aspiration of fluid from the fluid container 245. The system controller 270 can then update the data indicating the current volume of fluid, for example, each time it controls the filling of a syringe.

[0113] In some embodiments, camera 290 (or another camera not shown) can capture images depicting fluid container 245. In some such embodiments, system controller 270 can perform image processing on these captured images to determine whether fluid container 245 is empty or the amount of fluid remaining in fluid container 245.

[0114] The system controller 270 can begin the process by controlling the fluid aspiration subsystem 210 (step 510B) to operably connect a syringe (e.g., syringe tip 265) to the fluid container 245 (e.g., via a vial adapter).

[0115] Next, system controller 270 can control fluid aspiration subsystem 210 to retract plunger arm 215 inserted into syringe barrel 225 (step 520B).

[0116] In some examples, the system controller 270 determines the withdrawal distance of the plunger arm 215 based on the required syringe fill volume, such that the fluid aspiration system 210 withdraws the plunger arm 215 a certain distance (e.g., in millimeters) from the closed syringe hard stop 255, such that upon completion of withdrawal, the syringe barrel 225 is filled with the full volume (e.g., in milliliters) of fluid required as part of the injection preparation.

[0117] To improve system efficiency, for example, system controller 270 can use a plunger arm withdrawal distance that is less than the distance required to withdraw the required syringe fill or the distance required to empty fluid container 245. Specifically, system controller 270 can sequentially withdraw less fluid than required or available, and then capture an image to evaluate whether an air bubble or void condition has occurred. In this manner, air bubbles and voids can be detected and mitigated earlier, improving the speed of the system.

[0118] In some such instances, image capture and bubble assessment may occur while fluid aspiration continues, and in such cases, the bubble mitigation procedure, as described below, takes into account the movement of the plunger arm 215 that occurs after image capture.

[0119] Next, the system controller 270 can receive a digital image of the syringe 225 (step 530B), which may be, for example, a photograph taken by the camera 290 at the camera distance 235 after the fluid aspiration subsystem 210 has extended the plunger arm 215 to the extension distance.

[0120] System controller 270 may then use image processing methods on the received digital image to determine whether any air bubbles or voids are present in the syringe (step 540B).

[0121] In some embodiments (e.g., when the syringe 225 is oriented with the syringe tip 265 pointing upward), an air bubble is typically a quasi-spherical region of air within a fluid, while a void is typically a fluid-free region adjacent the closed syringe hard stop 255. Additionally, the system controller 270 can use appropriate image processing techniques to identify and evaluate air bubbles and / or voids.

[0122] In some embodiments, the image is captured in front of a patterned background, resulting in different regions of the syringe image exhibiting the patterned background having different refractive indices. In such embodiments, image processing methods based on identifying refractive regions can be employed, as described below.

[0123] If bubbles or voids are indeed present, system controller 270 can estimate the volume of the bubbles or voids in the syringe (e.g., using image processing methods based on identifying refractive regions, as described below, machine learning classification, or other image processing methods). If no bubbles or voids are present, the syringe filling operation is complete, and system controller 270 can perform an image-based confirmation of the amount of fluid in the syringe, as described in step 580B below.

[0124] The system controller 270 can then calculate the re-insertion distance of the plunger arm depending on the estimated volume of the air bubble or void (step 550B).

[0125] In some embodiments, the system controller 270 determines the reinsertion distance of the plunger arm from the pixel height of the void and a given pixel-to-distance value (i.e., the physical distance (e.g., in millimeters) of a single pixel at the current camera resolution and camera distance 235). In some embodiments, the system controller 270 uses a reinsertion distance of the plunger arm when a void is detected that is equal to the product of the pixel height and the given pixel-to-distance value.

[0126] In some other embodiments, the system controller 270 determines the reinsertion distance of the plunger arm by first calculating the volume of air in the void from the pixel height and a given pixel-to-volume value (i.e., the volume (e.g., in milliliters) in the syringe associated with a single pixel at the current camera resolution and camera distance 235). In some embodiments, the system controller 270 uses a volume of air equal to the product of the pixel height, the inner diameter of the syringe 225, and the given pixel-to-distance value. The system controller 270 can then determine the reinsertion distance from the estimated volume, for example, using the inner diameter of the syringe 225.

[0127] An exemplary method for receiving and deriving pixel-to-distance and pixel-to-volume values ​​has been described above with reference to FIG. 4A.

[0128] In some other embodiments, the system controller 270 determines the reinsertion distance of the plunger arm using machine learning classification of the image of the syringe. In some such embodiments, the system controller 270 determines the reinsertion distance of the plunger arm without first determining the volume of the air bubble or void.

[0129] In some other embodiments, the system controller 270 determines the reinsertion distance of the plunger arm by another suitable method.

[0130] The system controller 270 may then remove any air bubbles or voids by reinserting the plunger arm 215 the determined reinsertion distance (step 560B).

[0131] In some instances, removing the air bubble in other ways, such as by fully reinserting the plunger arm 215 into the syringe 225, may damage the syringe.

[0132] In some embodiments, in response to detecting an empty container, the system controller 270 can control a container exchange subsystem (not shown) to replace the (now empty) fluid container 245 with a new, filled fluid container 245. By drawing out and alleviating air bubbles / voids, all of the fluid in the vial is utilized, thereby improving efficiency and reducing maintenance and operator exposure to hazardous chemicals.

[0133] In some embodiments, the system controller 270 records the number of consecutive times that an air bubble is detected when filling the syringe 225 .

[0134] In some embodiments, if this number meets a given threshold of consecutive detected air bubbles, the system controller 270 may issue a warning (e.g., on the console, using a particular sound, etc.) that the fluid container is misconfigured, indicating that there may be a misconfiguration that is causing air to enter the syringe.

[0135] In some embodiments, if this count meets a given consecutively detected bubble threshold, system controller 270 can modify parameters of the fluid extraction performed by the fluid extraction subsystem. For example, in response to detecting an air bubble, system controller 270 can reduce the fluid aspiration rate and / or motor power.

[0136] Next, system controller 270 may evaluate whether the fluid aspiration completion criteria (step 570B) are met. If the fluid aspiration completion criteria are not met, system controller 270 may return to step 520B to complete the syringe fill and perform another bubble / void check. In some embodiments, the fluid aspiration completion criteria indicates whether plunger arm 215 has been withdrawn a full withdrawal distance, i.e., the distance (e.g., in millimeters) from closed syringe hard stop 255, which, when completed, will fill syringe barrel 225 with the full volume (e.g., in milliliters) of fluid required as part of the injection preparation.

[0137] If the fluid aspiration completion criteria are met, the system controller 270 may receive a digital image of the syringe and verify the amount of fluid in the syringe (step 580B), for example, as described above with reference to Figures 4A and 4B.

[0138] FIG. 5C illustrates a variation of a method for removing air bubbles (including bubbles generated by end-of-life conditions of the container) during robotic injection preparation based on imaging according to some embodiments of the presently disclosed subject matter.

[0139] The method of FIG. 5C may be suitable for instances where it is known in advance that the contents of the vial equal the required syringe fill volume.

[0140] Similarly, the method of FIG. 5C may be suitable for instances where it is known in advance that the contents of the vial are substantially equal (ie, equal within a tolerance) to the required syringe fill volume.

[0141] In the method of FIG. 5C, the system controller 270 stops aspirating fluid (step 570C) in response to the fluid container being empty (e.g., as detected from an image of the fluid container 245 or from continuous maintenance of the current volume of the fluid container 245).

[0142] In this manner, the method ensures that the contents of the fluid container 245 are fully utilized, improves efficiency while facilitating maintenance, and avoids operator exposure to hazardous materials.

[0143] The teachings of the presently disclosed subject matter are not constrained by the flowcharts shown in Figures 4A, 4B, 5A, 5B, and 5C. The illustrated operations may be performed in an order other than that shown. For example, operations 520B and 530B shown in succession may be performed substantially simultaneously or in the reverse order. Also, although the flowcharts have been described with reference to elements of the system of Figure 2, this is in no way constraining, and the operations may be performed by elements other than those described herein.

[0144] Reference is now made to FIG. 6A, which illustrates an exemplary pharmaceutical preparation device (PCD) according to some embodiments of the presently disclosed subject matter.

[0145] The PCD may include a vial holder 640, which holds a fluid vial 645 that holds the fluid to be drawn into the syringe.

[0146] The PCD may include a syringe queue 600 containing syringes that can be used with the PCD.

[0147] The PCD may include a syringe transport 620 (also called a pump) that transports the syringe.

[0148] FIG. 6B shows an exemplary syringe assembly including a connector, a septum, and a plunger flange.

[0149] FIG. 6C shows an exemplary engagement arm, an exemplary gripper arm, and an exemplary plunger arm of a PCD.

[0150] FIG. 6D shows an example of a transparent tube 650 that can be used with a PCD.

[0151] Referring now to FIG. 7, FIG. 7 illustrates an exemplary subsystem of a drug compounding device (PCD) configured to perform refraction-based detection of fluid and / or air in a transparent tube in accordance with some embodiments of the presently disclosed subject matter.

[0152] Transparent tubing 730 may be any type of transparent tubing or fluid transport or storage medium, such as the type of tubing that may be used to prepare pharmaceutical injectables. In some embodiments, transparent tubing 730 may be flexible tubing, as shown in FIG. 6D. In some embodiments, the tubing may be rigid. In some embodiments, transparent tubing 730 may be a fluid storage and / or transfer container, such as a syringe, vial, IV bag, or the like.

[0153] As used herein, "non-opaque" (FIG. 6D) may include transparent, translucent, or other materials that allow at least some light to pass through.

[0154] Camera 710 may be any type of suitable image capturing device and may include a lens 720 that can be directed in a particular lens direction.

[0155] The refractive pattern background 740 may be a flat or non-flat surface (e.g., cardboard, metal, or any other suitable material) having or exhibiting a pattern, e.g., a contrasting pattern such as contrasting black and white parallel lines. In some embodiments, the lines are diagonal. More commonly, the lines are angled relative to the major axis of the tube, e.g., the longitudinal axis of the tube, to form an asymmetric pattern relative to the major axis of the transparent tube.

[0156] In some embodiments, the transparent tube 730 is positioned to be interposed between the lens 720 and the refractive pattern background 740. As a result, in some instances, images captured by the camera 710 will show the transparent tube 730 in front of the refractive pattern background 770 (see, e.g., FIG. 8B below).

[0157] System controller 750 may be operatively connected to camera 710 and may receive images captured by camera 710. System controller 750 may include processing circuitry 760, which may include processor 230B and memory 240B.

[0158] Processor 770 may be any suitable hardware-based electronic device having data processing capabilities, such as, for example, a general-purpose processor, a digital signal processor (DSP), a specialized application-specific integrated circuit (ASIC), a graphical processing unit (GPU), one or more cores in a multi-core processor, etc. Processor 770 may also be comprised of, for example, multiple processors, multiple ASICs, virtual processors, combinations thereof, etc.

[0159] Memory 780 may be, for example, any suitable type of volatile and / or non-volatile storage device and may include, for example, a single physical memory component or multiple physical memory components. Memory 780 may also include virtual memory. Memory 780 may be configured to store various data used, for example, in operations.

[0160] The processing circuitry 760 may be configured to execute several functional modules in accordance with computer-readable instructions embodied in a non-transitory computer-readable storage medium. Such functional modules are hereinafter referred to as included in the processing circuitry. These modules may include, for example, a fluid identifier 790.

[0161] Fluid identification unit 790 can receive images captured by camera 710, for example, an image of transparent tube 730 in front of patterned background 740 at a particular stage in the process of drug preparation. Fluid identification unit 790 can then identify the contents of multiple portions of transparent tube 730 depending on how much the patterned background(s) detected on transparent tube 730 are refracted, as described later herein.

[0162] The teachings of the presently disclosed subject matter are not limited to the reflectance-based fluid detection subsystem and associated components described with reference to Figure 7. Equivalent and / or modified functionality may be integrated or divided in other ways and may be implemented in any suitable combination of software and firmware and / or software and hardware and executed on any suitable device.

[0163] FIG. 8A shows an exemplary transparent tube according to some embodiments of the presently disclosed subject matter.

[0164] In Figure 8A, a transparent tube 810A is partially filled with fluid and contains voids. Specifically, there is a fluid region 830A, an air region 820A, and a void 840A. While these regions are distinguishable, the contents of the tube are not necessarily apparent. This is because, for example, depending on the type of tube, both the fluid and the air appear differently. Therefore, detecting the type of fluid using computer vision can be difficult or prone to error.

[0165] FIG. 8B shows an example image of a transparent tube in front of a refractive pattern background, according to some embodiments of the presently disclosed subject matter.

[0166] Fluids are known to refract light to varying degrees depending on their viscosity, and the degree of refraction by a particular substance is sometimes known as its refractive index (see https: / / en.wikipedia.org / wiki / Refractive_index). Non-transparent tube 810A can also result in a certain level of refraction.

[0167] 8B, four regions of the transparent tube 810B are clearly distinguishable, and distinct variations in the refractive index of the refractive pattern background 810B are clearly visible: two regions 840B show patterns refracted by the tube 810B alone, and two regions 850B show patterns refracted by both the tube 810B and the fluid within the tube 810B.

[0168] Thus, system controller 750 can identify the contents of the region of tube 810B and take appropriate action, such as continuing or stopping the aspiration or injection of fluid, or mitigating the presence of air bubbles or voids.

[0169] Additionally, the refractive index of the region within tube 810B indicates the type of fluid within the tube, and system controller 750 can issue a warning if the fluid within tube 810B exhibits a refractive index different from the expected refractive index of the fluid, as this may indicate an error in the drug preparation.

[0170] Reference is now made to FIG. 9, which illustrates a flowchart of an exemplary method for identifying fluid or air within a region of a transparent tube from a received image, according to some embodiments of the disclosed subject matter.

[0171] The processing circuitry 760 (e.g., fluid identification unit 790) may segment the image into multiple image regions (step 910). For example, the processing circuitry 760 (e.g., fluid identification unit 790) may use an image processing-oriented machine learning (ML) model, such as Yolo5, to segment and define specific regions of the image. In processing the example image of FIG. 8B, such a model may identify regions 810B, 840B, and 850B by identifying differences in the patterns resulting from differential diffraction.

[0172] The processing circuitry 760 (e.g., fluid identification unit 790) can use various methods to determine a characteristic of the contents of the transparent tube from at least one of the image regions, such as identifying the type of fluid, such as identifying the presence of a gas (e.g., air) or a liquid, identifying the viscosity of the fluid, identifying the turbidity of the fluid, or any other characteristic associated with the contents of the tube.

[0173] As a non-limiting example, the processing circuitry 760 (e.g., fluid identification unit 790) can infer from the presence of two distinct image regions (as identified by Yolo5) that a transparent tube is partially filled. Similarly, the processing circuitry 760 (e.g., fluid identification unit 790) can infer from the presence of a first image region characterized by a first diffraction pattern, a second image region characterized by a second diffraction pattern, and a third image region characterized by the first diffraction pattern that the material located within the tube is not continuous, e.g., the presence of voids causes the fluid to be non-continuous. The remaining steps of FIG. 9 describe a method for identifying the presence and size (if any) of air and / or fluid regions within a transparent tube.

[0174] Optionally, the processing circuit 760 (eg, fluid identification unit 790 ) can then identify image regions 920 that include the refraction pattern background 740 .

[0175] In some embodiments, the processing circuitry 760 (e.g., fluid identification unit 790) can then use the pattern of the refractible pattern background 740 to match image regions that may be refracted by the refractible pattern background 740. In some other embodiments, the pattern of the refractible pattern background 740 is preset or has otherwise been previously made available to the processing circuitry 760 (e.g., fluid identification unit 790). Thus, in such embodiments, it is not necessary to identify image regions that include the refractible pattern background 740.

[0176] As a non-limiting example, in the image of FIG. 8B, processing circuitry 760 (eg, fluid identification unit 790) may identify refractive pattern background 810B in the image.

[0177] The processing circuitry 760 (eg, fluid identification unit 790) may then identify one or more regions 930 of the captured image that include refraction of the refractible pattern background 740.

[0178] In some embodiments, the processing circuit 760 (e.g., fluid identification unit 790) analyzes each segmented region of the image to find a refraction pattern. In some other embodiments, the processing circuit 760 (e.g., fluid identification unit 790) stops analyzing after detecting a single region containing a refraction pattern. In some other embodiments, the processing circuit 760 (e.g., fluid identification unit 790) stops according to some other criteria.

[0179] In some embodiments, processing circuit 760 (e.g., fluid identification unit 790) analyzes a region only after determining (using a suitable method) that the region is located within transparent tube 810B. In some other embodiments, processing circuit 760 (e.g., fluid identification unit 790) analyzes all regions. In some other embodiments, processing circuit 760 (e.g., fluid identification unit 790) selects regions for analysis according to different criteria.

[0180] In some embodiments, the processing circuitry 760 (e.g., fluid identification unit 790) evaluates (using appropriate image processing methods) whether the image region matches an expected (e.g., preset depending on the expected fluid type) refractivity of the refractible pattern background 740. In some other embodiments, the processing circuitry 760 (e.g., fluid identification unit 790) evaluates (using appropriate image processing methods) whether the image region matches any refractivity of the refractible pattern background 740. In some such embodiments, the processing circuitry 760 (e.g., fluid identification unit 790) can determine the refractivity (e.g., refractive index) of the fluid or air (possibly in combination with the refractive index of the transparent tube 810B).

[0181] Next, the processing circuit 760 (e.g., fluid identification unit 790) may determine, for one or more of the identified image regions found to contain refraction of the refractible pattern background 740, whether the refraction of the region satisfies the respective refraction criteria for identifying an image region having particular content (step 940).

[0182] In some embodiments, the refraction criterion may be a match to a particular refraction (e.g., refractive index value). For example, the refraction criterion may be whether the refraction of the imaged region matches the refraction expected for a particular fluid (e.g., saline) or gas (e.g., air) present in a particular type of transparent tubing 730. "Match" in this context means match within a variation interval so that normal variations in the fluid, tubing, or imaging do not affect the correct identification of the fluid or air.

[0183] In some embodiments, the refraction criterion may be meeting a refraction threshold (e.g., a refractive index value). For example, the refraction criterion may be whether the refraction of the image region meets a minimum refraction resulting from the presence of any type of fluid in all known types of transparent tubes 730. In such an example, if the determined refraction meets the refraction criterion, the refraction may indicate the presence of fluid in the transparent tube 230, and if the determined refraction does not meet the refraction criterion, the refraction may indicate the absence of fluid in the transparent tube 730.

[0184] The teachings of the presently disclosed subject matter are not bound by the flowchart shown in Figure 9, and the operations shown may be performed in an order other than that shown. For example, steps 930 and 940 of operations shown in succession may be performed substantially simultaneously or in the reverse order. Also, while this flowchart has been described with reference to elements of the system of Figure 7, it should be noted that this is in no way binding, and these operations may be performed by elements other than those described herein.

[0185] 10A-10C, which show a flowchart of an exemplary method for drug preparation in a PCD, according to some embodiments of the presently disclosed subject matter, where refraction-based fluid discrimination is used in the process.

[0186] In FIG. 10A, processing circuitry 760 (eg, fluid identification unit 790) can receive a camera image of non-opaque fluid 730 (step 1010A).

[0187] Processing circuitry 760 (e.g., fluid identification unit 790) may then identify a first region of the tube in the image that includes background refracted by fluid or air (possibly in combination with refraction by transparent tube 730) (step 1020A). Identifying image regions by fluid or air may be done, for example, by methods such as those described above with reference to FIG.

[0188] The processing circuit 760 (e.g., fluid identification unit 790) can then use the pixel size of the first region in combination with an appropriate method to determine the amount of fluid (or air) in the transparent tube 730 (step 1030A).

[0189] In FIG. 10B, processing circuitry 760 (eg, fluid identification unit 790) can receive a camera image of non-opaque fluid 730 (step 1010B).

[0190] The processing circuitry 760 (e.g., fluid identification unit 790) can then identify a first region of the tube in the image that includes a background refracted by fluid or air (possibly in combination with refraction by the transparent tube 730) (step 1020B) and can determine the refraction (e.g., refractive index) of this image region.

[0191] The processing circuit 760 (e.g., fluid identification unit 790) can hold data indicative of, for example, an expected refractivity. For example, the processing circuit 760 (e.g., fluid identification unit 790) can recognize the type of drug preparation fluid that should be present in the transparent tube 730. The processing circuit 760 (e.g., fluid identification unit 790) can then detect whether the refractivity of the first region matches the expected refractivity.

[0192] If the refractivity of the first region does not match the expected refractivity, this may indicate a defect in the drug preparation process.

[0193] Thus, in response to the refractivity of the first region not matching the expected refractivity, the processing circuit 760 (e.g., fluid identification unit 790) may issue a warning (step 1030B).

[0194] In FIG. 10C, processing circuitry 760 (eg, fluid identification unit 790) can receive a camera image of transparent tube 730 (step 1010C).

[0195] The processing circuit 760 (e.g., fluid identification unit 790) can then identify a first region of the tube in the image that includes background refracted by fluid or air (possibly in combination with refraction by the transparent tube 730) (step 1020C).

[0196] The processing circuitry 760 (e.g., fluid identification unit 790) can then identify a second region of the tube in the image that includes background refracted by fluid or air (possibly in combination with refraction by the transparent tube 730) (step 1030C).

[0197] The processing circuit 760 (e.g., fluid identification unit 790) can then use the pixel distance between the first image region and the second image region in combination with an appropriate pixel-to-volume calculation method to determine the volume of the void between the two fluid regions within the transparent tube 730 (step 1040C).

[0198] The teachings of the presently disclosed subject matter are not bound by the flowcharts shown in Figures 10A-10C, and the illustrated operations may be performed in an order other than that shown. Also, although the flowcharts have been described with reference to elements of the system of Figure 7, this is in no way binding, and the operations may be performed by elements other than those described herein.

[0199] FIG. 11A shows an exemplary flow of a robotic compounding process that results in a syringe prepared for injection into a patient.

[0200] The processing circuitry can position the syringe manipulator at a device position configured for operation between the syringe and the fluid container (e.g., a vial) by operably connecting the syringe to the fluid container (e.g., via a vial adapter and / or syringe connector, or directly) (step 1110A).

[0201] The processing circuitry can then control the syringe manipulator, particularly the plunger holder, to perform a "flush" operation on the syringe (step 1120A), i.e., expel any air present, if the syringe has not yet been used. More specifically, the processing circuitry can control the syringe manipulator to withdraw the plunger from the syringe barrel, e.g., by 2 millimeters, and then reinsert the plunger.

[0202] The processing circuitry may aspirate the required amount of fluid into the syringe and verify the aspirated amount (step 1130A). These processes are described in detail above with reference to Figures 5A-5C and 4A-4B, respectively. During these processes, images may be taken and processed for one or more of the following purposes: verifying the amount of fluid aspirated, identifying and / or mitigating air bubbles or voids, determining when the vial is empty, and determining whether the syringe is misaligned.

[0203] At this stage, the operator may prepare labels (stickers) (eg, in English and / or containing bar codes) for the fluids being prepared.

[0204] The processing circuitry can wait for the application of the sticker to the syringe and, in response to detecting the sticker based on imaging, perform an image-based confirmation as to whether the sticker is accurate (step 1140A). If the sticker is accurate, i.e., consistent with preparation by the robotic compounding process, the processing circuitry can control the syringe manipulator to disconnect the syringe from the fluid container (step 1150A). On the other hand, if the sticker is not accurate, the processing circuitry can notify the operator that a new sticker will be needed.

[0205] The processing circuitry can then verify that the operator has removed the syringe, i.e., the syringe manipulator is empty (step 1160A). Finally, the processing circuitry can move the syringe manipulator to the loading zone (i.e., the location where a new syringe will be initially removed) (step 1170A).

[0206] FIG. 11B shows an exemplary flow of a robotic compounding process that results in an IV bag for patient administration or storage.

[0207] The initial steps of this method are the same as those described with respect to Figure 11A above.

[0208] After injecting the fluid into the syringe, the processing circuitry can move the syringe manipulator to an IV fill position (step 1140B). The processing circuitry can then connect the syringe to an IV bag (step 1150B).

[0209] The processing circuitry can control the syringe manipulator to transfer fluid from the syringe into the IV bag (step 1160B) (eg, as described above in FIGS. 5A-5C).

[0210] The processing circuitry can receive an image of the syringe and, using image processing techniques, can confirm that the contents of the syringe have been completely transferred into the IV bag (e.g., as described above in Figures 4A-5B) (step 1170B).

[0211] The processing circuitry can then disconnect the syringe from the IV bag (step 1180B).

[0212] Once the IV bag is ready (step 1190B), the process can end. Similar to the process of Figure 11A, the processing circuitry can maintain connection with the IV bag until a label or sticker (e.g., in English or a bar code describing the contents of the IV bag) is applied by the operator. The processing circuitry can then verify the label or sticker and, if successful, release the IV bag.

[0213] If the IV bag is not ready (step 1190B), the process may return to step 1110B.

[0214] The teachings of the presently disclosed subject matter are not constrained by the flowcharts shown in Figures 11A-10B, and the illustrated operations may be performed in an order other than that shown. Also, although the flowcharts have been described with reference to elements of a system such as that shown in Figure 2, this is in no way constraining, and the operations may be performed by elements other than those described herein. Summary of the Invention

[0215] Example 1 1. A system for determining the presence of fluid or air in a transparent tube in a drug preparation device, comprising: a) receiving a camera image of the transparent tube, the transparent tube being positioned in front of a patterned background from a camera perspective; whereby the received image depicts one or more regions of the transparent tube, each depicted region being associated with a respective refractive index of the patterned background; and b) using image processing techniques to identify one or more of the regions of the transparent tube in the received image; and and c) determining a characteristic of the contents of said transparent tube from one or more of said identified regions.

[0216] Example 2 The system of Example 1, wherein the PC is further configured to determine a characteristic of the contents based on the refractive index associated with the one or more of the identified regions, thereby providing an indication of the type of contents present in the transparent tube.

[0217] Example 3 3. The system of claim 2, wherein the PC determines whether the associated refractive index of one of the identified regions matches the refractive index characteristics of air within the transparent tube.

[0218] Example 4 3. The system of claim 2, wherein the PC determines whether the associated refractive index of one of the identified regions matches the refractive index characteristics of the liquid in the transparent tube.

[0219] Example 5 The PC further comprises: 5. The system of any one of Examples 1 to 4, configured to determine a volume of the region of the transparent tube based on pixel dimensions of one of the identified regions of the image.

[0220] Example 6 The PC further comprises: 6. The system of any one of Examples 1 to 5, configured to determine a volume of a gap between regions of the transparent tube according to a pixel distance between a first identified region of the image and a second identified region of the image.

[0221] Example 7 The PC further comprises: In response to a refractive index associated with one of the identified regions not matching an expected fluid refractive index, 7. The system of any one of Examples 1 to 6, configured to issue an alert.

[0222] Example 8 1. A method for determining the presence of liquid or air in a transparent tube in a drug preparation device based on a processing circuit, comprising: a) receiving a camera image of the transparent tube, the transparent tube being positioned in front of a patterned background from a camera perspective; receiving, whereby the received image depicts one or more regions of the transparent tube, each depicted region being associated with a respective refractive index of the patterned background; b) using image processing techniques to identify one or more of the regions of the transparent tube in the received image; and and c) determining a characteristic of the contents of said transparent tube from at least one or more of said identified regions.

[0223] Example 9 A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the steps recited in Example 8.

[0224] Example 10 the camera positionable to capture an image of the transparent tube; and the patterned background positionable to appear behind the transparent tube in an image captured by the camera.

[0225] Example 11 11. The system of example 10, wherein the patterned background comprises regions of parallel lines.

[0226] Example 12 12. The system of claim 11, wherein the parallel lines are oriented obliquely relative to the major axis of the tube.

[0227] Example 13 The system of Example 11, wherein each of the parallel lines is at an angle of 1 to 179 degrees relative to the major axis of the tube.

[0228] Example 14 1. A robotic system for aspirating fluid from a fluid container into a syringe, comprising: a) controlling the fluid aspiration subsystem to withdraw a plunger arm of a syringe operably connected to a first fluid container by a withdrawal distance; b) receiving a digital image of the syringe; c) in response to detecting bubbles or voids in the captured digital image using image processing techniques; A robotic system comprising a processing circuit (PC) configured to control the fluid suction subsystem to push the plunger arm a pushing distance corresponding to the size of the bubble or void determined based on the captured image.

[0229] Example 15 The PC further comprises: d) controlling the fluid suction subsystem to depress the plunger arm; configured to repeat a) through c) for one or more additional iterations; in each iteration, the control of the fluid aspiration subsystem withdraws the plunger arm a respective incremental withdrawal distance until the plunger arm is withdrawn a full withdrawal distance from the syringe; 15. The system of example 14, wherein the full withdrawal distance is determined based on a required syringe fill volume.

[0230] Example 16 The PC further comprises: d) in response to the fluid container being empty, controlling a container exchange subsystem to exchange the first fluid container for a second fluid container; e) repeating a) through c) for one or more additional iterations; in each iteration, the control of the fluid aspiration subsystem withdraws the plunger arm a respective incremental withdrawal distance until the plunger arm is withdrawn a full withdrawal distance from the syringe; 15. The system of example 14, wherein the full withdrawal distance is determined based on a required syringe fill volume.

[0231] Example 17 The PC further comprises: receiving a digital image of the syringe after aspiration; using image processing techniques to estimate the volume of fluid from the post-aspiration digital image; in response to the estimated fluid volume differing from the required syringe fill volume by an amount that exceeds a fluid volume tolerance; 17. The system of any one of Examples 14 to 16, configured to issue an alert.

[0232] Example 18 18. The system of any one of Examples 14 to 17, wherein the withdrawal distance is a full withdrawal distance determined based on a required syringe fill volume.

[0233] Example 19 The PC further controls the actuator to move the plunger arm the pull-out distance, 19. The system of any one of Examples 14 to 18, configured to control the fluid suction subsystem to retract the plunger arm.

[0234] Example 20 The PC further comprises: 20. The system of any one of Examples 14 to 19, configured to calculate the pushing distance based at least on the pixel diameter of the bubble or void in the captured digital image.

[0235] Example 21 The PC further receives, in response to the number of consecutively detected bubbles or voids satisfying a consecutively detected bubble threshold, 21. The system of any one of Examples 14 to 20, configured to reduce the speed of the actuator that moves the plunger arm.

[0236] Example 22 The PC further receives, in response to the number of consecutively detected bubbles or voids satisfying a consecutively detected bubble threshold, 22. The system of any one of Examples 14 to 21, configured to issue a warning that the fluid container is misconfigured.

[0237] Example 23 the digital image is captured with the syringe positioned in front of a patterned background from a camera's perspective, whereby one or more regions of the syringe are depicted in the received image, each depicted region being associated with a respective refractive index of the patterned background; The PC further comprises: a) using image processing techniques to identify one or more of said regions of said syringe in said captured image; b) detecting air bubbles or voids in the captured digital image by determining the presence of air bubbles or voids in the syringe based on at least one or more of the identified regions.

[0238] Example 24 The system of Example 23, wherein the PC is further configured to determine the presence of bubbles or voids based on the refractive index associated with the one or more of the identified regions.

[0239] Example 25 17. The system of embodiment 16, wherein the empty fluid container state is based on the PC performing image processing on an image of the fluid container.

[0240] Example 26 Example 17. The system of Example 16, wherein the fluid container empty state is based on the PC maintaining a fluid container fill level.

[0241] Example 27 1. A method of aspirating fluid from a fluid container into a syringe based on a processing circuit, comprising: a) controlling a fluid aspiration subsystem to withdraw a plunger arm of a syringe operably connected to a first fluid container by a withdrawal distance; b) receiving a digital image of the syringe; and c) in response to detecting bubbles or voids in the captured digital image using image processing techniques; and controlling the fluid aspiration subsystem to depress the plunger arm a depression distance corresponding to a size of the bubble or void determined based on the captured image.

[0242] Example 28 When executed by a computer, A computer-readable storage medium containing instructions for causing the computer to perform the steps recited in Example 27.

[0243] Example 29 1. A system for aspirating fluid from a fluid container into a syringe, comprising: a) controlling a fluid aspiration subsystem to withdraw a plunger arm of a syringe operably connected to a first fluid container containing a first fluid quantity by a withdrawal distance; b) receiving a digital image of the syringe; c) in response to detecting bubbles or voids in the captured digital image using image processing techniques; reinserting the plunger arm a reinsertion distance according to the size of the bubble or void determined based on the captured image; controlling the fluid aspiration subsystem; d) repeating steps a) to c) until the fluid container is empty; The system includes a processing circuit (PC) configured to repeat the process of controlling the fluid aspiration subsystem to extend the plunger arm a respective incremental extension distance in each iteration.

[0244] Example 30 28. The system of claim 27, wherein the empty fluid container state is based on the PC performing image processing on an image of the fluid container.

[0245] Example 31 28. The system of example 27, wherein the fluid container empty state is based on the PC maintaining a fluid container fill level.

[0246] Example 32 1. A method of aspirating fluid from a fluid container into a syringe based on a processing circuit, comprising: a) controlling a fluid aspiration subsystem to withdraw a plunger arm of a syringe operably connected to a first fluid container containing a first fluid quantity by a withdrawal distance; b) receiving a digital image of the syringe; and c) in response to detecting bubbles or voids in the captured digital image using image processing techniques; controlling the fluid aspiration subsystem to reinsert the plunger arm a reinsertion distance dependent on the size of the bubble or void determined based on the captured image; d) repeating steps a) to c) until the fluid container is empty; The method includes a processing circuit (PC) configured to repeat the process of controlling the fluid aspiration subsystem to withdraw the plunger arm a respective incremental withdrawal distance in each iteration.

[0247] Example 33 33. The method of example 32, wherein the empty fluid container state is based on the PC performing image processing on an image of the fluid container.

[0248] Example 34 33. The method of example 32, wherein the fluid reservoir empty state is based on the PC maintaining a fluid reservoir fill level.

[0249] Example 35 A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the steps recited in any one of Examples 32 to 34.

[0250] Example 36 1. A system for verifying the aspiration of a required amount of fluid into a syringe, the system comprising: a processing circuit (PC) operably connectable to a plunger holder configured to withdraw a plunger inserted into the syringe, the syringe being operably connected to a fluid container; The PC is a) controlling the plunger holder to withdraw the plunger a first plunger withdrawal distance based on the required amount, thereby moving the plunger holder from a first plunger holder position to a second plunger holder position; b) receiving a digital image of the syringe taken after said withdrawal, and using image processing techniques on said digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid.

[0251] Example 37 The PC further comprises, before a), configured to receive a first digital image of the plunger holder at the first plunger holder position; The PC further comprises, after b), in response to the current amount of fluid in the syringe not matching the required amount of fluid; a. receiving a second digital image of the plunger holder at the second plunger holder position; b. using image processing techniques on the first digital image of the plunger holder and the second digital image of the plunger holder to determine whether the second position of the plunger holder corresponds to the first position of the plunger holder and the first plunger withdrawal distance; c. in response to the second position of the plunger holder being consistent with the first position of the plunger holder and the first plunger travel distance; The system of Example 36, configured to issue a warning that the syringe is misconfigured.

[0252] Example 38 1. A method for verifying the aspiration of a required amount of fluid into a syringe based on a processing circuit, comprising: a) controlling the plunger holder to withdraw the plunger a first plunger withdrawal distance based on a required amount, thereby moving the plunger holder from a first plunger holder position to a second plunger holder position; b) receiving a digital image of the syringe taken after said withdrawal; and using image processing techniques on said digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid.

[0253] Example 39 Before a), receiving a first digital image of the plunger holder at the first plunger holder position; After b), in response to the current amount of fluid in the syringe not matching the required amount of fluid; a. receiving a second digital image of the plunger holder at the second plunger holder position; b. using image processing techniques on the first digital image of the plunger holder and the second digital image of the plunger holder to determine whether the second position of the plunger holder corresponds to the first position of the plunger holder and the first plunger withdrawal distance; c. in response to the second position of the plunger holder being consistent with the first position of the plunger holder and the first plunger travel distance; 39. The method of example 38, further comprising issuing a warning that the syringe is misconfigured.

[0254] Example 40 A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the steps recited in any one of Examples 38 to 39.

[0255] Example 41 1. A system for verifying transfer of a required amount of fluid from a syringe into a fluid container, the system comprising: a processing circuit (PC) operably connected to a plunger holder for operating a plunger inserted into the syringe, the syringe being operably connected to the fluid container; The PC is a) controlling the plunger holder to depress the plunger a first plunger depression distance based on the required amount, thereby moving the plunger holder from the first plunger holder position to a second plunger holder position; b) receiving a digital image of the syringe taken after the depression of the plunger, and using image processing techniques on the digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid remaining.

[0256] Example 42 The PC further comprises, before a), configured to receive a first digital image of the plunger holder at the first plunger holder position; The PC further comprises, after b), in response to the current amount of fluid in the syringe not matching the required remaining amount of fluid; a. receiving a second digital image of the plunger holder at the second plunger holder position; b. using image processing techniques on the first digital image of the plunger holder and the second digital image of the plunger holder to determine whether the second position of the plunger holder corresponds to the first position of the plunger holder and the first plunger depression distance; c. in response to the second position of the plunger holder being consistent with the first position of the plunger holder and the first plunger depression distance; The system of Example 41, configured to issue a warning that the syringe is misconfigured.

[0257] Example 43 1. A method for verifying the aspiration of a required amount of fluid into a syringe based on a processing circuit, comprising: a) controlling the plunger holder to depress the plunger a first plunger insertion distance based on a required amount, thereby moving the plunger holder from a first plunger holder position to a second plunger holder position; b) receiving a digital image of the syringe taken after the depression of the plunger, and using image processing techniques on the digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid.

[0258] Example 44 The method further comprises, before a), receiving a first digital image of the plunger holder at the first plunger holder position; The method further comprises, after b), in response to the current amount of fluid in the syringe not matching the required remaining amount of fluid; a. receiving a second digital image of the plunger holder at the second plunger holder position; b. using image processing techniques on the first digital image of the plunger holder and the second digital image of the plunger holder to determine whether the second position of the plunger holder matches the first position of the plunger holder and the first plunger insertion distance; c. in response to the second position of the plunger holder being consistent with the first position of the plunger holder and the first plunger insertion distance; and issuing a warning that the syringe is misconfigured.

[0259] Example 45 A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the steps recited in any one of Examples 43 to 44.

[0260] The present invention is not limited to the application examples described in the description herein or illustrated in the drawings. The present invention is capable of other embodiments and of being practiced and carried out in various ways. Accordingly, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. Thus, those skilled in the art will appreciate that the conception underlying the present disclosure may readily be utilized as a basis for the designing of other structures, methods, and systems for carrying out some of the purposes of the subject matter of the present disclosure.

[0261] The system according to the invention is implemented, at least in part, on a suitably programmed computer. Likewise, the invention contemplates a computer program readable by a computer for carrying out the method of the invention. The invention further contemplates a non-transitory computer-readable memory tangibly embodying a program of instructions executable by a computer to carry out the method of the invention.

[0262] It will be readily apparent to those skilled in the art that various modifications and changes may be applied to the embodiments of the present invention as described above without departing from the scope thereof as defined in and by the appended claims.

Claims

1. 1. A system for determining the presence of fluid or air in a transparent tube in a drug preparation device, comprising: d) receiving a camera image of the transparent tube, the transparent tube being positioned in front of a patterned background from a camera perspective; receiving, whereby the received image depicts one or more regions of the transparent tube, each depicted region being associated with a respective refractive index of the patterned background; e) using image processing techniques to identify one or more of the regions of the transparent tube in the received image; f) determining a characteristic of the contents of said transparent tube from one or more of said identified regions.

2. 10. The system of claim 1, wherein the PC is further configured to determine a characteristic of the contents based on the refractivity associated with the one or more of the identified regions, thereby providing an indication of the type of contents present in the transparent tube.

3. The system of claim 2 , wherein the PC determines whether the associated refractivity of one of the identified regions matches the refractivity characteristics of air within the transparent tube.

4. The system of claim 2 , wherein the PC determines whether the associated refractive index of one of the identified regions matches a refractive index characteristic of a liquid within the transparent tube.

5. The PC further comprises: The system of claim 1 , configured to determine a volume of the region of the transparent tube based on pixel dimensions of one of the identified regions of the image.

6. The PC further comprises:

6. The system of claim 1, configured to determine a volume of a gap between regions of the transparent tube depending on a pixel distance between a first identified region of the image and a second identified region of the image.

7. The PC further comprises: In response to a refractive index associated with one of the identified regions not matching an expected fluid refractive index, 7. The system of claim 1, configured to issue a warning.

8. 1. A method for determining the presence of liquid or air in a transparent tube in a drug preparation device based on a processing circuit, comprising: d) receiving a camera image of the transparent tube, the transparent tube being positioned in front of a patterned background from a camera perspective; receiving, whereby the received image depicts one or more regions of the transparent tube, each depicted region being associated with a respective refractive index of the patterned background; e) using image processing techniques to identify one or more of the regions of the transparent tube in the received image; f) determining a characteristic of the contents of said transparent tube from at least one or more of said identified regions.

9. A computer-readable storage medium containing instructions that, when executed by a computer, cause the computer to perform the steps recited in claim 8.

10. the camera positionable to capture an image of the transparent tube; The system of claim 1 , further comprising: the patterned background positionable to appear behind the transparent tube in an image captured by the camera.

11. The system of claim 10 , wherein the patterned background comprises regions of parallel lines.

12. The system of claim 11 , wherein the parallel lines are oriented obliquely relative to a major axis of the tube.

13. The system of claim 11, wherein each of the parallel lines forms an angle of between 1 and 179 degrees with respect to the major axis of the tube.

14. 1. A robotic system for aspirating fluid from a fluid container into a syringe, comprising: f) controlling the fluid aspiration subsystem to withdraw a plunger arm of a syringe operatively connected to the first fluid container a withdrawal distance; g) receiving a digital image of the syringe; h) in response to detecting an air bubble or void in the captured digital image using image processing techniques; A robotic system comprising a processing circuit (PC) configured to control the fluid suction subsystem to push the plunger arm a pushing distance corresponding to the size of the bubble or void determined based on the captured image.

15. The PC further comprises: e) controlling the fluid aspiration subsystem to depress the plunger arm; configured to repeat a) through c) for one or more additional iterations; in each iteration, the control of the fluid aspiration subsystem withdraws the plunger arm a respective incremental withdrawal distance until the plunger arm is withdrawn a full withdrawal distance from the syringe; The system of claim 14, wherein the full withdrawal distance is determined based on a required syringe fill volume.

16. The PC further comprises: i) in response to a fluid container being empty, controlling a container exchange subsystem to exchange the first fluid container for a second fluid container; j) repeating a) through c) for one or more additional iterations; in each iteration, the control of the fluid aspiration subsystem withdraws the plunger arm a respective incremental withdrawal distance until the plunger arm is withdrawn a full withdrawal distance from the syringe; The system of claim 14, wherein the full withdrawal distance is determined based on a required syringe fill volume.

17. The PC further comprises: receiving a digital image of the syringe after aspiration; using image processing techniques to estimate the volume of fluid from the post-aspiration digital image; in response to the estimated fluid volume differing from the required syringe fill volume by an amount that exceeds a fluid volume tolerance; 17. A system according to any one of claims 14 to 16, configured to issue a warning.

18. 18. The system of any one of claims 14 to 17, wherein the withdrawal distance is a full withdrawal distance determined based on a required syringe fill volume.

19. The PC further controls the actuator to move the plunger arm the extension distance, 19. The system of claim 14, configured to control the fluid aspiration subsystem to retract the plunger arm.

20. The PC further comprises:

20. The system of any one of claims 14 to 19, configured to calculate the plunge distance based on at least a pixel diameter of the bubble or void in the captured digital image.

21. The PC further receives, in response to the number of consecutively detected bubbles or voids satisfying a consecutively detected bubble threshold, 21. The system of any one of claims 14 to 20, configured to reduce the speed of the actuator that moves the plunger arm.

22. The PC further receives, in response to the number of consecutively detected bubbles or voids satisfying a consecutively detected bubble threshold, 22. The system of any one of claims 14 to 21, configured to issue a warning that the fluid container has been misconfigured.

23. the digital image is captured with the syringe positioned in front of a patterned background from a camera's perspective, whereby one or more regions of the syringe are depicted in the received image, each depicted region being associated with a respective refractive index of the patterned background; The PC further comprises: c) using image processing techniques to identify one or more of said regions of said syringe in said captured image; d) detecting air bubbles or voids in the captured digital image by determining the presence of air bubbles or voids in the syringe based on at least one or more of the identified regions.

24. 24. The system of claim 23, wherein the PC is further configured to determine the presence of the bubbles or voids based on the refractivity associated with the one or more of the identified regions.

25. 17. The system of claim 16, wherein the empty fluid container status is based on the PC performing image processing on an image of the fluid container.

26. 17. The system of claim 16, wherein the fluid container empty condition is based on the PC maintaining a fluid container fill level.

27. 1. A method of aspirating fluid from a fluid container into a syringe based on a processing circuit, comprising: d) controlling the fluid aspiration subsystem to withdraw a plunger arm of a syringe operatively connected to the first fluid container a withdrawal distance; e) receiving a digital image of the syringe; f) in response to detecting an air bubble or void in the captured digital image using image processing techniques; and controlling the fluid aspiration subsystem to depress the plunger arm a depression distance dependent on the size of the bubble or void determined based on the captured image.

28. When executed by a computer, 30. A computer-readable storage medium containing instructions that cause the computer to perform the steps recited in claim 27.

29. 1. A system for aspirating fluid from a fluid container into a syringe, comprising: d) controlling the fluid aspiration subsystem to withdraw a plunger arm of a syringe operably connected to a first fluid container containing a first fluid quantity a withdrawal distance; e) receiving a digital image of the syringe; f) in response to detecting an air bubble or void in the captured digital image using image processing techniques; reinserting the plunger arm a reinsertion distance according to the size of the bubble or void determined based on the captured image; controlling the fluid aspiration subsystem; e) repeating a) through c) until the fluid container is empty; A system comprising a processing circuit (PC) configured to repeat the process of controlling the fluid aspiration subsystem to extend the plunger arm a respective incremental extension distance in each iteration.

30. 28. The system of claim 27, wherein the empty fluid container status is based on the PC performing image processing on an image of the fluid container.

31. 28. The system of claim 27, wherein the fluid container empty state is based on the PC maintaining a fluid container fill level.

32. 1. A method of aspirating fluid from a fluid container into a syringe based on a processing circuit, comprising: e) controlling the fluid aspiration subsystem to withdraw a plunger arm of a syringe operably connected to a first fluid container containing a first fluid quantity a withdrawal distance; f) receiving a digital image of the syringe; g) in response to detecting bubbles or voids in the captured digital image using image processing techniques; controlling the fluid aspiration subsystem to reinsert the plunger arm a reinsertion distance dependent on the size of the bubble or void determined based on the captured image; h) repeating a) through c) until the fluid container is empty; a processing circuit (PC) configured to repeat, in each iteration, the control of the fluid aspiration subsystem to withdraw the plunger arm a respective incremental withdrawal distance.

33. 33. The method of claim 32, wherein the fluid container empty status is based on the PC performing image processing on an image of the fluid container.

34. 33. The method of claim 32, wherein the fluid container empty condition is based on the PC maintaining a fluid container fill level.

35. 35. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the steps of any one of claims 32 to 34.

36. 1. A system for verifying the aspiration of a required amount of fluid into a syringe, the system comprising: a processing circuit (PC) operably connected to a plunger holder configured to withdraw a plunger inserted into the syringe, the syringe being operably connected to a fluid container; The PC is c) controlling the plunger holder to withdraw the plunger a first plunger withdrawal distance based on the required amount, thereby moving the plunger holder from a first plunger holder position to a second plunger holder position; d) receiving a digital image of the syringe taken after said withdrawal, and using image processing techniques on said digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid.

37. The PC further comprises, before a), configured to receive a first digital image of the plunger holder at the first plunger holder position; The PC further comprises, after b), in response to the current amount of fluid in the syringe not matching the required amount of fluid; receiving a second digital image of the plunger holder at the second plunger holder position; b. using image processing techniques on the first digital image of the plunger holder and the second digital image of the plunger holder to determine whether the second position of the plunger holder corresponds to the first position of the plunger holder and the first plunger pull-out distance; c) in response to the second position of the plunger holder being coincident with the first position of the plunger holder and the first plunger travel distance; 37. The system of claim 36, configured to issue a warning that the syringe is misconfigured.

38. 1. A method for verifying the aspiration of a required amount of fluid into a syringe based on a processing circuit, comprising: c) controlling the plunger holder to withdraw the plunger a first plunger withdrawal distance based on a required amount, thereby moving the plunger holder from a first plunger holder position to a second plunger holder position; d) receiving a digital image of the syringe taken after said withdrawal; and using image processing techniques on said digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid.

39. a) before receiving a first digital image of the plunger holder at the first plunger holder position; b) followed by in response to the current amount of fluid in the syringe not matching the required amount of fluid; d. receiving a second digital image of the plunger holder at the second plunger holder position; e. applying image processing techniques to the first digital image of the plunger holder and the second digital image of the plunger holder to determine whether the second position of the plunger holder corresponds to the first position of the plunger holder and the first plunger pull-out distance; f. in response to the second position of the plunger holder being coincident with the first position of the plunger holder and the first plunger travel distance; 39. The method of claim 38, further comprising issuing a warning that the syringe is misconfigured.

40. 40. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the steps recited in any one of claims 38 to 39.

41. 1. A system for verifying transfer of a required amount of fluid from a syringe into a fluid container, the system comprising: a processing circuit (PC) operably connected to a plunger holder for operating a plunger inserted into the syringe, the syringe being operably connected to the fluid container; The PC is c) controlling the plunger holder to depress the plunger a first plunger depression distance based on the required amount, thereby moving the plunger holder from the first plunger holder position to a second plunger holder position; d) receiving a digital image of the syringe taken after the depression of the plunger, and using image processing techniques on the digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid remaining.

42. The PC further comprises, before a), configured to receive a first digital image of the plunger holder at the first plunger holder position; The PC further comprises, after b), In response to the current amount of fluid in the syringe not matching the required remaining amount of fluid, receiving a second digital image of the plunger holder at the second plunger holder position; b) applying image processing techniques to the first digital image of the plunger holder and the second digital image of the plunger holder to determine whether the second position of the plunger holder corresponds to the first position of the plunger holder and the first plunger depression distance; c) in response to the second position of the plunger holder being consistent with the first position of the plunger holder and the first plunger depression distance; 42. The system of claim 41, configured to issue a warning that the syringe is misconfigured.

43. 1. A method for verifying the aspiration of a required amount of fluid into a syringe based on a processing circuit, comprising: c) controlling the plunger holder to depress the plunger a first plunger insertion distance based on a required amount, thereby moving the plunger holder from a first plunger holder position to a second plunger holder position; d) receiving a digital image of the syringe taken after the depression of the plunger, and using image processing techniques on the digital image of the syringe to determine whether the amount of fluid currently in the syringe matches the required amount of fluid.

44. The method further comprises, prior to a), receiving a first digital image of the plunger holder at the first plunger holder position; The method further comprises, after b), In response to the current amount of fluid in the syringe not matching the required remaining amount of fluid, d. receiving a second digital image of the plunger holder at the second plunger holder position; e. applying image processing techniques to the first digital image of the plunger holder and the second digital image of the plunger holder to determine whether the second position of the plunger holder corresponds to the first position of the plunger holder and the first plunger insertion distance; f. in response to the second position of the plunger holder being consistent with the first position of the plunger holder and the first plunger insertion distance; and issuing a warning that the syringe is misconfigured.

45. 45. A computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of any one of claims 43 to 44.

Images