Multi-camera imaging for intravenous preparation

Abstract

Systems and methods for preparing medications, such as for intravenous delivery. The device includes an infrared camera and an infrared light source, and can backlight and photograph items such as syringes under infrared light to improve clarity. A visible light camera, such as a color digital camera, can also be provided to record other parts of the preparation process. The device can have other sensors, such as a barcode scanner and a weight sensor, for collecting other data about the preparation process. In an example implementation, the device is a preparation assistance device that uses instructions displayed on a display to guide the user of the device to complete the preparation task. In other embodiments, the device can be a robot.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Non-Provisional Patent Application No. 15 / 865,038, filed on January 8, 2018, and Italian Patent Application no. 102017000134813, filed on November 23, 2017, the entire disclosures of which are incorporated herein by reference for all purposes. Background Art

[0003] Drug compounding is the preparation of a patient-specific drug by processing or combining components. Many drugs, especially those taken orally in pill form, are now prepared in a variety of forms and doses, so that pharmacies require little preparation, other than packing the appropriate number of pills in a bottle to match a doctor's prescription for a particular patient. However, for drugs delivered intravenously, compounding is customary, for example in a hospital pharmacy.

[0004] Typically, a doctor will prescribe a specific drug or combination of drugs for a specific patient to be delivered intravenously (IV). The pharmacy receives the prescription and prepares an IV solution with the appropriate amount of each prescribed drug. The prepared drugs are then sent to the hospital ward for administration to the patient.

[0005] It is critical that the correct drug product is prepared in the correct proportions without the introduction of contaminants. Detailed protocols can be developed for formulators to follow. The number of different protocols can be very large, as there may be a large number of different drug products available in a variety of packaging, prepared in a variety of ways, prepared in a variety of dosages and provided in a variety of different delivery vehicles.

[0006] Much of the work of compounding may be delegated to workers who are not registered pharmacists or to robotic machines. Therefore, meticulous records of the preparation of each drug may be kept so that the pharmacist can audit the preparation of each drug before it leaves the pharmacy. These records also allow the preparation of any particular drug to be audited at a later date if there are any questions of correctness. Summary of the invention

[0007] According to one aspect, the preparation auxiliary device includes a bracket for supporting articles. The material of the bracket is not opaque to infrared light. The preparation auxiliary device also includes an infrared digital camera, which is positioned to shoot at least a portion of the bracket from above. The preparation auxiliary device also includes a surface light source located below the bracket. The surface light source is configured to generate infrared light and guide the infrared light through the bracket and toward the digital camera. The preparation auxiliary device also includes a display and a controller. The controller is programmed to use one or more prompts displayed on the display to guide the user of the preparation auxiliary device throughout the drug preparation task. In some embodiments, the preparation auxiliary device further includes a weight sensor, on which the bracket is placed, and the weight sensor is configured to generate a signal indicating the weight of the bracket and any items on the bracket. In some embodiments, the infrared digital camera is a first digital camera, and the preparation auxiliary device further includes a second digital camera, which is positioned to shoot at least a portion of the bracket from above, and the second digital camera is a visible light camera. The second digital camera can be a color digital camera. In some embodiments, the preparation auxiliary device also includes a visible light source associated with the second digital camera, which can be controlled by a controller to emit light during the use of the first digital camera to take pictures. In some embodiments, the surface light source below the carriage includes a plurality of infrared light emitting diodes and a diffuser. In some embodiments, the surface light source below the carriage includes a plurality of infrared light emitting diodes, and the carriage is made of a diffuse material. In some embodiments, the preparation auxiliary device further includes a gantry across the carriage, and the infrared digital camera is mounted on the gantry. In some embodiments, the preparation auxiliary device further includes an adhesive label printer, and the controller is programmed to print an adhesive label once the drug preparation task is completed, and the adhesive label is affixed to a container containing a prepared drug during the preparation task. In some embodiments, the preparation auxiliary device further includes a barcode scanner, which is positioned to read a barcode from an item between the barcode scanner and the carriage. In some embodiments, the digital camera is a first digital camera, and the dispensing aid further includes a second digital camera positioned to photograph at least a portion of the carriage from above, the second digital camera being sensitive to visible light, and a gantry spanning the carriage, wherein the first digital camera and the second digital camera, and the barcode scanner are mounted on the gantry. In some embodiments, the controller is further programmed to analyze a digital photograph of the syringe taken by the digital camera under infrared light, and estimate the amount of liquid in the syringe based on the analysis of the digital photograph. In some embodiments, the carriage defines a recess having a certain shape and size for accommodating a syringe barrel flange of the syringe. In some embodiments, the controller is further programmed to analyze a digital photograph of the syringe taken by the digital camera under infrared light, identify the size of the syringe based on the analysis of the digital photograph, and annotate the digital photograph of the syringe with a volume indicator.

[0008] According to another aspect, a method for preparing a drug includes: receiving a syringe in an observation area between an infrared light source and an infrared digital camera, taking a digital photo of the syringe under infrared light using the infrared digital camera so that the syringe is illuminated from behind by the infrared light source, and storing the digital photo. In some embodiments, the method also includes automatically analyzing the digital photo to calculate the amount of liquid in the syringe. In some embodiments, the method also includes automatically analyzing the digital photo to determine the size and position of the syringe in the digital photo, and annotating the digital photo with a volume indicator. In some embodiments, the infrared light source and the infrared digital camera are included in a preparation auxiliary device, and the syringe is received between the infrared light source and the infrared digital camera, which includes manually receiving the syringe from a user of the preparation auxiliary device. In some embodiments, the preparation auxiliary device includes a display and a controller, and the method also includes using a series of prompts displayed on the display to guide the user of the preparation auxiliary device throughout the drug preparation task. In some embodiments, the infrared light source and the infrared digital camera are included in a preparation robot, and the syringe is received between the infrared light source and the infrared digital camera, which includes receiving the syringe from a robot mechanism. In some embodiments, the digital photograph is a first digital photograph, and the method further comprises taking a second digital photograph of the container involved in the compounding process using a visible light camera, and storing the second digital photograph in association with the first digital photograph. In some embodiments, taking the second digital photograph comprises taking a color photograph.

[0009] According to another aspect, the drug preparation device comprises an infrared light source, an infrared digital camera, an observation area between the infrared light source and the infrared digital camera, and a controller. The controller is programmed to control the infrared light source and the infrared digital camera to take photos of the object in the observation area so that the object is illuminated from behind by the infrared light source. In some embodiments, the drug preparation device is a preparation auxiliary device including a display, and a controller, which is further programmed to use the prompts displayed on the display to guide the user of the device throughout the drug preparation task. In some embodiments, the drug preparation device is a preparation robot. In some embodiments, the drug preparation device also includes a visible light camera under the control of the controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The patent or application file contains at least one drawing printed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0011] Figure 1 A compounding pharmacy according to an embodiment of the present invention is shown.

[0012] Figure 2A manual dispensing station according to an embodiment of the present invention is shown.

[0013] Figure 3 A dispensing aid according to an embodiment of the invention is shown.

[0014] Figure 4 The embodiment of the present invention is shown Figure 3 A downward oblique view of the dispensing aid in FIG.

[0015] Figure 5 The embodiment of the present invention is shown. Figure 3 Barcode scanning by the preparation auxiliary equipment.

[0016] Figure 6 A step in a preparation process according to an embodiment of the present invention is shown.

[0017] Figure 7 Another step of the formulation process of an embodiment of the present invention is shown.

[0018] Figure 8 Another step of the formulation process of an embodiment of the present invention is shown.

[0019] Fig. 9 Another step of the formulation process of an embodiment of the present invention is shown.

[0020] Fig.10 A picture that can be taken using a visible light camera according to an embodiment of the present invention is shown.

[0021] Fig.11 A photograph of a syringe according to an embodiment of the present invention that may be taken by an infrared camera is shown.

[0022] Fig.12 Another step in the formulation process according to an embodiment of the present invention is shown.

[0023] Fig.13 Another step of the formulation process according to an embodiment of the present invention is shown.

[0024] Fig.14 The embodiment of the present invention is shown Figure 3 The arrangement of the surface light source and infrared camera of the auxiliary equipment.

[0025] Fig.15 The infrared camera according to the embodiment of the present invention is used to take pictures. Fig.11 The photos are similar to the photos and show the image analysis.

[0026] Fig.16 The embodiment of the present invention is shown along Fig.15 Example of the brightness trace of the pixels of the column lines in the image.

[0027] Fig.17 A method for annotating an infrared image according to an embodiment of the present invention is shown.

[0028] Fig.18 The embodiment of the present invention is shown Figure 3 Arrangement of the visible light camera in the preparation auxiliary equipment.

[0029] Fig.19 Exemplary photographs of multiple syringes that may be taken by a visible light camera according to embodiments of the present invention are shown.

[0030] Fig. 20 The embodiment of the present invention is shown Figure 3 Simplified block diagram of the preparation auxiliary equipment. DETAILED DESCRIPTION

[0031] Figure 1 A compounding pharmacy 100 is shown according to an embodiment of the present invention. The operation of the pharmacy 100 is coordinated by a pharmacy server 101, which will be described in more detail below. The pharmacy server 101 receives orders for compounded medications, such as prescriptions from physicians. The pharmacy server 101 maintains a large number of records of the orders received, detailed protocols for compounding medications, records of medication preparations in response to orders, and other items. The pharmacy server 101 also assigns tasks to one or more compounding stations, which may include manual compounding stations, such as stations 102a and 102b, and one or more robotic compounders 103. The compounding stations may also report information to the pharmacy server 101, such as recording the compounding of each ordered medication.

[0032] Working materials are supplied to the compounding station from the material warehouse 104. The pharmacy server 101 can maintain the material inventory in the material warehouse 104 and can track the movement of drugs and materials within the pharmacy 100.

[0033] Finished products are reviewed by a pharmacist and delivered from the pharmacy 100 to their point of use, such as a patient's room where they are administered by a nurse. It will be appreciated that the above description is highly generalized and a working compounding pharmacy may have many other systems and facilities.

[0034] Figure 2 A manual dispensing station 102a is shown in accordance with an embodiment of the present invention. The dispensing station 102a includes a dispensing aid 201 on a surface 202. The dispensing aid 201 can be placed under a laminar flow hood 203 that flows filtered air through the dispensing aid 201 and the surface 202 to help avoid contamination of the material being processed and to protect the user of the dispensing station 102a.

[0035] In the example shown, compounding station 102a has received a supply for a simple compounding task. The drug supply in vial 204 is added to IV drip bag 205. A syringe 206 may be used to complete the transfer.

[0036] The compounding assistant 201 has several features and capabilities that will assist the compounder in correctly preparing the preparation in the IV drip bag 205 and fully documenting the process. The compounding assistant 201 has a network connection 207 to the pharmacy server 101, but the compounding assistant 201 can receive a protocol from the pharmacy server 101 that describes the steps required to perform the compounding task.

[0037] The dispensing aid 201 includes a display screen 208 on which instructions to the user may be presented or through which the user may input information. For example, the display screen 208 may be a touch screen display that is sensitive to touch and capable of distinguishing the location of the touch. The dispensing aid 201 also includes a tray 209 that provides a support for holding items while being weighed or photographed, as described in more detail below.

[0038] Figure 3 The dispensing aid 201 according to an embodiment of the present invention is shown with the tray 209 removed. Figure 3 A weight sensor 301, such as a load cell, for weighing the tray 209 and its contents is visible. A surface light source 302 is also visible. The surface light source 302 is a two-dimensionally extended or surface light source and emits light from many points or continuously over its entire surface. The surface light source 302 can be, for example, an infrared light panel that illuminates a portion of the tray 209 from below with infrared light.

[0039] Figure 4 A lower oblique view of a dispensing aid 201 according to an embodiment of the present invention is shown. A gantry 401 spans across a tray 209. A barcode scanner 402, a visible light camera 403, and an infrared camera 404 are located on the gantry 401. The visible light camera 403 may further include one or more light sources 405 for illuminating at least a portion of the tray 209 from above. The light source 405 may be, for example, one or more white light emitting diodes (LEDs) or another light source surrounding the visible light camera 403. For purposes of the present invention, light is "visible" if it includes wavelengths of light between approximately 400 and 700 nanometers. Light is "white" if it has sufficient wavelengths in the visible range to achieve reasonable complete color recognition.

[0040] The area above the tray 209 may be referred to as the viewing area of ​​the items to be photographed by the infrared camera 404 or the visible light camera 403, or scanned by the barcode scanner 402. In other embodiments, the items may not necessarily be illuminated from below and photographed from above. For example, in a dispensing robot, the robotic mechanism may hold the items to be photographed in the field of view of the camera in any orientation. For example, the items may be photographed from below or horizontally.

[0041] The barcode scanner 402 is positioned to read barcodes on items held in a viewing area between the tray 209 and the barcode scanner 402. The visible light camera 403 and the infrared camera 404 are positioned to take pictures of the items on the tray 209.

[0042] During the preparation of the medication, one or more of the weight sensor 301, barcode scanner 402, visible light camera 403, and infrared camera 404 may be used to provide documentation of how the medication was prepared and to avoid errors.

[0043] For example, to execute Figure 2 For the compounding task shown, the pharmacy server 101 sends detailed sequential instructions to the compounding auxiliary device 201, which then guides the user through the steps required to formulate a specific drug at a specific dosage for delivery in a specific delivery vehicle. In this example, the task may involve transferring 30,000 units of heparin (a common anticoagulant) from a vial containing 5,000 units / ml of heparin in solution to an IV drip bag. The volume of solution required for the transfer is 6 ml. The vial 204 and IV drip bag 205 have been provided to the compounding station 102a along with the syringe 206, which will be required to make the transfer.

[0044] First, the compounding aid 201 requires the user to present the vial 204 to the barcode scanner 402 so that the identification barcode on the vial 204 can be read and the system can verify whether the correct vial with the correct concentration has been provided. If not, an error message is then generated and the compounding task stops. The scanning process is Figure 5 , together with an exemplary prompt displayed on screen 208. The dispensing aid 201 can automatically recognize that a barcode has been detected and can move to the next step. Alternatively, confirmation from the user may be required in this step and other steps.

[0045] Figure 6The second step in the compounding process is shown, in which the initial weight of the vial 204 is collected. For this purpose, the vial 204 is placed on the tray 209. The tray 209 may include an icon 601 indicating where the vial 204 should be placed, and may also include mechanical features to help properly place the vial 204. For example, a gentle V-shaped groove may be formed in the tray 209. The compounding aid 201 can automatically identify the weight of the vial 204 on the tray 209, record the weight, and move to the next step of the subsequent compounding process.

[0046] In some embodiments, the vials 204 are photographed while on the tray 209 using a visible light camera 403, using ambient light, light from a light source 405, or a combination thereof.

[0047] Figure 7 A third step is shown in which an initial weight of IV bag 205 is collected. Thereafter, compounding aid 201 can prompt the user to draw the correct amount (6 ml) of solution from vial 204 into syringe 206.

[0048] Figure 8 The fourth step is shown, in which the Figure 6 The weight of the vial 204 after it has been drawn is obtained in a similar manner as the initial vial weight shown. The system can compare the two weights of the vial 204 to calculate the amount of solution drawn from the vial 204 for record keeping and verification that the correct amount of solution has been drawn.

[0049] Fig. 9 A fifth step is shown in which a full syringe is photographed. To this end, the tray 209 may include an icon 901 for placing the syringe 206 and may include mechanical features that help properly place and align the syringe 206 on the tray 209, such as a V-groove or groove 902 that is shaped and sized to receive the edge of the flange of the syringe 206 barrel. Other fiducial markings may also be present.

[0050] The syringe 206 may be photographed using a visible light camera 403 , but is preferably photographed using an infrared camera 404 . Fig.10 403. Fig.10 206 has been separated from the larger view for ease of illustration. Fig.10 The syringe 206 is easily visible in the photograph of , but the photograph has been affected by local glare 1001 and may have been affected by ambient light sources that are not under the control of the dispensing aid 201.

[0051] Fig.11 A photograph of syringe 206 is shown that can be taken using infrared camera 404. Tray 209 is not opaque to infrared radiation, so syringe 206 is backlit by infrared light source 302. For example, tray 209 can be substantially transparent to infrared radiation, or can be translucent. In some embodiments, tray 209 can be made of polycarbonate or another suitable polymer or blend of polymers. Infrared camera 404 can have a wavelength selective filter that passes infrared light to camera 404 but blocks the visible light spectrum. Thus, local glare from visible light is blocked from the syringe 206. Fig.11 206 is excluded from the photograph, resulting in clearer features of the syringe 206.

[0052] Regardless of the camera used, the dispensing aid 201 can automatically analyze the resulting photographs for any of a number of purposes. For example (see Fig.11 ), the position of the plunger 1101 of the syringe 206 can be automatically identified, and the amount of liquid 1102 to be drawn is calculated based on the known size of the syringe 206. In some embodiments, if bubbles such as bubbles 1103 are large enough to significantly affect the dose of the prepared drug, they may be detected and marked. In some schemes, the weight of the syringe 206 before and after drawing liquid from the vial 204 can be used to verify whether the correct amount of liquid is placed in the syringe 206. In this case, the preparation aid 201 can also photograph the syringe 206 at each weighing and analyze the photos to detect whether the syringe cap 1104 may have been mistakenly included in one weighing but not another. The reference marks 1105 on the tray 209 are placed in known positions and can also be detected in the photos and used to calibrate the distances in the photos.

[0053] Fig.12 A sixth step is shown in which IV bag 205 is re-weighed after solution is added from syringe 206. Compounding aid 201 can compare the before and after weights of bag 205 to verify that the correct amount of heparin solution was placed into bag 205.

[0054] Fig.13A seventh step is shown in which (assuming all checks in the system have verified that the compounding process was completed correctly), the compounding aid 201 prints a label 1301 using a label printer 1303 to be placed on the bag 205, and prompts the user to affix the label 1301 to the bag 205. The finished drug can then be delivered to its point of use, and any consumable items, such as syringes 206, disposed of. The user may be asked to confirm 1302 that the label 1301 has been affixed using the display 208. In some embodiments, a final photograph of the completed bag 205 may be taken for review by the pharmacist.

[0055] The compounding process described above is only one example, and many different compounding workflows may be implemented that have different steps, use different drug containers, collect different or additional information for process validation, or differ in other ways from the example shown.

[0056] Although the above example is shown in the context of compounding workstation 102a, a similar process may be followed to use a robotic compounder (e.g. Figure 1 The robotic compounder 103 shown in the figure is used for compounding. The robotic compounder is a generally closed machine that uses a robotic mechanism to handle medicine bottles, syringes, bags, etc. to prepare compounded medicines. The robotic compounder may include a scale, one or more cameras, a stirring device, a processing port, a material and material loading window, and a delivery window for delivering finished medicines. The robotic compounder is not subject to human error during the compounding process, but includes various weight and photo checks in its work to prevent improper loading of materials, mechanical failures, programming errors, etc.

[0057] Whether the dispensing is done manually or automatically, the data collected during the dispensing process is stored, for example, on the pharmacy server 101 and can be reviewed by the responsible pharmacist. For example, the pharmacist can verify that the correct type of medicine bottle containing the correct medicine has been identified through bar code scanning. The dosage can be verified by viewing a photo of the syringe, the front and back weight of the medicine bottle, the front and back weight of the bag, or any combination of these. Any digital photos taken during the dispensing process can be used for the pharmacist's inspection. For example, the pharmacist can look at the photo (such as Fig.11 ) to determine whether the liquid drawn into syringe 206 may contain excessive bubbles.

[0058] After completing the compounding task, the pharmacy server 101 may assign another compounding task to the compounding station 102 a and download another recipe to the compounding assistance device 201 according to the new task.

[0059] Fig.14The arrangement of the surface light source 302 and the infrared camera 404 of the preparation auxiliary device 201 according to the embodiment of the present invention is shown.

[0060] In this example, the surface light source 302 includes an array of infrared light emitting diodes (LEDs) 1401 mounted on a circuit board 1402. Light from the infrared LEDs 1401 passes through a diffuser 1403 and is scattered upward. Some of the light reaches the lens 1404 of the infrared camera 404, forming an image onto an electronic array light sensor 1405 that is correspondingly mounted on a printed circuit board 1406. The printed circuit board 1406 can be combined with a controller interface within the dispensing auxiliary device 201 to receive signals for controlling the electronic array light sensor 1405.

[0061] Infrared LED 1401 can emit light of near infrared wavelength, for example, between about 700 and 900 nanometers. In other embodiments, other wavelengths can be used. Diffuser 1403 provides substantially uniform backlight for items placed on surface light source 302 (e.g., syringe 206). Surface light source 302 can be controlled by an electronic controller in preparation auxiliary device 201. In some embodiments, tray 209 can be made of diffuse material and can be used as a supplement or replacement of diffuser 1403 to diffuse the light from surface light source 302.

[0062] In other embodiments, other kinds of light sources may be used, such as an edge-lit light guide with scattering features on one side. In this arrangement, the light source directs light into one or more edges of the light guide, and the light propagates by total internal reflection within the plate until it has one of the scattering features. Some of the scattered light is scattered out of the side of the plate opposite the scattering features. The scattering features are preferably distributed so that the intensity of the light leaving the plate is substantially uniform over the entire area of ​​the plate. A diffuser may also be used to further diffuse the light leaving the plate to achieve additional uniformity of brightness.

[0063] The electronic array light sensor 1405 can be, for example, a charge coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, or another suitable sensor. Typically, such sensors utilize the property of some semiconductor materials that, when the material is illuminated by light, free electrons are generated that are proportional to the light intensity. The sensor is divided into specific light-sensitive areas, called "pixels." To capture an image, the pixels are reset and then exposed for an exposure time. At the end of the exposure time, the amount of charge accumulated in each pixel is measured and converted to a numerical value. The array of these numerical values ​​can be referred to as a "digital image," where each value in the array represents the brightness of the light falling on the corresponding pixel.

[0064] In a CCD sensor, the accumulated charge is transferred away from the sensor to a charge amplifier whose output for each pixel is digitized. In a CMOS sensor, the accumulated charge can be read directly from each pixel without transfer.

[0065] The electronic array light sensor 1405 can have many pixels sufficient to resolve features of interest at the tray 209. In some embodiments, the electronic array light sensor 1405 can include an array of 2560×1920 pixels, or approximately five million pixels. Other array sizes can be used in other embodiments. The electronic array light sensor 1405 is sensitive to light of infrared wavelengths emitted by the surface light source 302. For example, the electronic array light sensor 1405 can be a silicon-based sensor that is sensitive to near-infrared light. The infrared camera 404 can include a filter (not shown) that excludes other wavelengths. The filter can be, for example, a dichroic filter that passes light of a wavelength of interest, but blocks light of other wavelengths, such as visible light.

[0066] As described above, infrared camera 404 can produce a picture of items on tray 209 that are relevant in some aspects to medication compounding that is clearer than a picture taken using visible light camera 403. For example, localized glare caused by ambient light in the room can be greatly eliminated. This clarity facilitates analysis of digital pictures taken using infrared camera 404 for measurements and annotations that may be helpful to a reviewing pharmacist.

[0067] Fig.15 Shown with Fig.11 , which is taken using an infrared camera such as infrared camera 404 and illustrates image analysis according to an embodiment of the present invention. Based on the known positions of the fiducial markers 1105 on the tray 209, the number of pixels in the electronic image sensor 1405, and the magnification of the optical system including the lens 1404, the controller within the dispensing aid 201 may "know" the relative positions of the fiducial markers 1105 (measured in image pixels). The controller can quickly locate the fiducial markers in the image by looking for a pattern of black dots near the expected positions of the fiducial markers in the image. The pixel positions of the fiducial markers in the image can be recorded for reference.

[0068] like Fig.15 As can be seen in FIG. 1 , the syringe flange 1501 of the syringe 206 has been placed in the groove 902 of the tray 209, and therefore the syringe flange 1501 is located relative to the reference mark 1105. Fig.15The controller can then query the brightness values ​​of pixels in the image near the fiducial marker to locate the edge of the syringe 206 in the "Y" direction. For example, the pixels along column lines 1502, 1503, and 1504 can be analyzed to find abrupt transitions from light to dark and from dark to light that indicate the presence of portions of the edge of the syringe 206.

[0069] Fig.16 An example of a brightness trajectory of pixels along column line 1502 is shown, moving in the +Y direction from lower left fiducial marker 1505. It can be assumed that the transition spanned by width W includes the needle of syringe 206. If no drop in brightness is detected at the expected location of the needle, the controller can assume that there is no needle attached to syringe 206. Assuming a drop is detected, the centerline of the area spanned by width W can be assumed to be the centerline of the needle. Width W can be compared to known dimensions of syringe components to determine whether a cap is present on the needle.

[0070] Reference again Fig.15 Similar trajectories may be performed along other lines to detect the presence and size of a Luer lock 1506, or the presence and size of a plunger (not labeled) of a syringe 206. The detected size may be compared to the stored sizes of a standard syringe to automatically determine the size of the syringe 206.

[0071] In other embodiments, other image processing techniques may be used to determine the position and size of the syringe from the digital image. For example, a correlation operation may be performed on a previously prepared syringe photograph. The previous photograph may be compared with the current photograph in multiple directions and positions to find the position most relevant to the syringe in the current photograph to determine the position of the syringe in the current photograph. The fiducial marker 1105 may also be found in this manner. In other embodiments, a synthetic syringe image may be used in the correlation operation. Many other techniques are also possible.

[0072] Once the size and position of the syringe 206 are known in pixel space, the controller can annotate the digital image of the syringe to assist the pharmacist in reviewing the compounding operation in the captured image.

[0073] In some embodiments, similar image processing techniques can be used to locate the plunger 1507 in the digital image. Given the location of the plunger 1507, the location and orientation of the syringe 206, and the dimensions of the syringe 206, an estimate of the volume of liquid in the syringe 206 can be calculated.

[0074] Fig.171 shows a way to annotate an infrared image according to an embodiment of the present invention. Once the size and position of the syringe 206 are determined, the size can be associated with a standard syringe having pre-recorded measurements, the standard recorded size including the position of the graduations 1701 on the syringe barrel indicating the volume of liquid in the syringe based on the position of the plunger. Fig.17 The graduation lines 1701 are highly visible in the image, but may not always be the case. Depending on the position of the syringe on the tray 209, the graduation lines 1701 may not be easily visible in any image. For example, the syringe 206 may have been placed on the tray 209 with the graduation lines 1701 facing downward, or the liquid in the syringe 206 may be opaque, hiding the graduation lines 1701.

[0075] Using the known size and position of the syringe 206, the dispensing aid 201 can annotate the image captured by either of its cameras to enhance readability of the plunger position. Fig.17 , the dispensing aid has changed some pixels of the image to display a line 1702 corresponding to the position of the calculated tick mark 1701, and has also added text 1703 showing the volume of liquid represented by the line 1702. In other embodiments, other kinds of annotations may be provided, such as using different colors.

[0076] Fig.18 The arrangement of a visible light camera 403 of a preparation aid 201 in an embodiment of the present invention is shown. The visible light camera 403 can be used to photograph the items on the tray 209 for verifying whether the correct components are used in the preparation task, for final verification that the resulting product looks as it should, or for other purposes. The visible light camera 403 can be a color camera and is particularly useful for recording the color of the preparation as an additional verification that the preparation is likely correct. (The exemplary infrared camera 404 described above cannot distinguish colors due to the narrow infrared wavelength band it records and the lack of any color filters on the pixels.)

[0077] Visible light camera 403 includes a lens 1801 that focuses light received within its field of view onto an electronic sensor array light sensor 1802, which in turn is mounted on a printed circuit board 1803. Visible light camera 403 may include a filter (not shown), such as a dichroic filter, that substantially blocks infrared wavelengths from reaching sensor 1802. Light source 405 may be used to supplement any ambient light that illuminates tray 209. For example, light source 405 may be a white LED for tray 209 and may be controlled by a controller within dispensing aid 201.

[0078] Electronic array light sensor 1802 may be a CCD sensor, a CMOS sensor, or another suitable sensor as described above, having sufficient pixels to resolve features of interest at tray 209. For example, sensor 1802 may include an array of 2560×1920 pixels, or approximately five million pixels. Other sensor sizes may be used. Sensor 1802 preferably includes color filters placed over each pixel so that visible light camera 403 can record images in color. For example, sensor 1802 may include red, green, and blue filters in the well-known Bayer pattern.

[0079] Visible light camera 403 and infrared camera 404 may be provided as preassembled camera modules that include a standard interface for control by dispensing aid 201. Suitable camera modules are available from Basler AG, Ahrensburg, Germany, and IDS Imaging Development Systems, Obersulm, Germany.

[0080] Fig.19 An exemplary photograph of multiple syringes 1901a-1901f is shown, which may be taken by visible light camera 403 using light source 405. Each syringe 1901a-1901f contains a different color of fluid. Fig.19 The fluid in the body is not necessarily medicine. Fig.19 It is shown that the visible light camera 403 can distinguish between multiple colors, and the photos taken with the visible light camera 403 can enable the pharmacist to verify that the compounded liquid has the expected color, thereby increasing confidence that the compounding is correct, or detect that the compounded liquid is not the expected color, indicating that the compounding may not have been completed correctly.

[0081] Fig. 20 A simplified block diagram of a preparation assistance device 201 according to an embodiment of the present invention is shown. The preparation assistance device 201 includes a controller 2001, which includes a processor 2002, a memory 2003, and a network interface 2004. The memory 2003 may include dynamic memory, non-volatile memory, mass storage, or other types of memory in any suitable combination. A portion of the memory 2003 stores instructions for the processor 2002, which when executed controls the operation of the preparation assistance device 201. Other types of information may also be stored in the memory 2003, such as digital images, temporary variables, and working copies of other types of information. The network interface 2004 allows the controller 2001 to communicate externally, for example, with a server such as the pharmacy server 101 described above.

[0082] The dispensing aid 201 further includes an infrared camera 404, a barcode scanner 402, and a visible light camera 403 as described above, all of which are in communication with and under the control of the controller 2001. The infrared light source 302 and the visible light source 405 are also under the control of the controller 2001, turning on and off at different times. In some embodiments, the intensity of light produced by one or both light sources can be adjusted under the control of the controller 2001. The touch screen display 208 can convey information to a user of the dispensing aid 201 and can receive instructions from the user.

[0083] The label printer 1303 receives commands and data for printing labels from the controller 2001. The weight sensor 301 provides a signal to the controller 2001 indicating the weight of the tray 209 and any items thereon.

[0084] Other architectures for configuring auxiliary device 201 may be used.

[0085] In the appended claims, the terms "a" or "an" are intended to mean "more than one". The terms "comprising" and variations thereof, such as "comprising" and "which comprise", when preceding the recitation of steps or elements, are intended to indicate that the addition of further steps or elements is optional and not excluded. It should be understood that any feasible combination of elements and features disclosed herein is also considered disclosed.

[0086] The present invention has now been described in some detail for purposes of clarity and understanding. However, one skilled in the art will appreciate that certain changes and modifications may be practiced within the scope of the appended claims.

Claims

1. Prepare auxiliary equipment, including: a bracket for supporting the article, wherein the material of the bracket is not opaque to infrared light; an infrared digital camera positioned to photograph at least a portion of the carriage from above; a surface light source located below the bracket, the surface light source being configured to generate infrared light and direct the infrared light through the bracket and toward the digital camera; monitor; as well as a controller programmed to guide a user of the dispensing aid through a medication dispensing task using one or more prompts displayed on the display; a visible light digital camera positioned to photograph at least a portion of the carriage from above; a visible light source positioned adjacent to the visible light digital camera and controllable by the controller to emit light during the taking of a picture using the visible light digital camera; A gantry spans across the bracket so that the gantry is fixed to the bracket at both ends of the gantry, wherein the infrared digital camera, the visible light digital camera, and the visible light source are mounted on the gantry.

2. The dispensing aid of claim 1, further comprising a weight sensor, the tray being placed on the weight sensor, the weight sensor being configured to generate a signal representative of the weight of the tray and any items on the tray.

3. The dispensing aid of claim 1, wherein the visible light digital camera is a color digital camera.

4. The preparation auxiliary device as claimed in claim 1, wherein the surface light source below the bracket comprises: a plurality of infrared light emitting diodes; and Diffuser.

5. The preparation aid of claim 1, wherein: The surface light source under the bracket includes a plurality of infrared light emitting diodes; and the bracket is made of a diffusion material.

6. The dispensing aid of claim 1, further comprising a gantry spanning across the bracket, wherein the infrared digital camera is mounted on the gantry.

7. The compounding assistance device of claim 1, further comprising an adhesive label printer, wherein the controller is programmed to print an adhesive label upon completion of a medication compounding task, the adhesive label to be affixed to a container containing medication compounded during the compounding task.

8. The dispensing aid of claim 1, further comprising a bar code scanner positioned to read a bar code from an item between the bar code scanner and the tray.

9. The preparation aid according to claim 8, The barcode scanner is installed on the door frame.

10. The dispensing aid of claim 1, wherein the controller is further programmed to: analyzing a digital photograph of the syringe taken by the digital camera under infrared light; and The amount of liquid in the syringe is estimated based on analysis of the digital photograph.

11. The dispensing aid of claim 1, wherein the bracket defines a recess having a certain shape and size for receiving a barrel flange of a syringe.

12. The dispensing aid of claim 1, wherein the controller is further programmed to: analyzing a digital photograph of the syringe taken by the digital camera under infrared light; identifying a size of the syringe based on analysis of the digital photograph; and The digital photograph of the syringe is annotated with volume indicators.

13. A method for preparing a medicine using the preparation auxiliary device according to any one of claims 1 to 12, the method comprising: receiving the syringe in a viewing area between an infrared light source and an infrared digital camera; taking a digital photograph of the syringe under infrared light using the infrared digital camera such that the syringe is backlit by the infrared light source; and The digital photo is stored.

14. The method of claim 13, further comprising automatically analyzing the digital photograph to calculate the amount of fluid in the syringe.

15. The method of claim 13, further comprising automatically analyzing the digital photograph to determine the size and location of the syringe in the digital photograph; and annotating the digital photograph with a volume indicator.

16. The method of claim 13, wherein: The infrared external light source and the infrared digital camera are included in a dispensing auxiliary device; and Receiving the syringe between the infrared external light source and the infrared digital camera includes manually receiving the syringe from a user of the dispensing aid.

17. The method of claim 16, wherein the compounding aid comprises a display and a controller, and the method further comprises guiding a user of the compounding aid through a medication compounding task using a series of prompts displayed on the display.

18. The method of claim 13, wherein: The infrared external light source and the infrared digital camera are included in a dispensing robot; And receiving the syringe between the infrared light source and the infrared digital camera includes receiving the syringe from a robotic mechanism.

19. The method of claim 13, wherein the digital photograph is a first digital photograph, the method further comprising: taking a second digital photograph of the container involved in the dispensing process using a visible light camera; and The second digital photograph is stored in association with the first digital photograph.

20. The method of claim 19, wherein taking the second digital photograph comprises taking a color photograph.

21. Drug preparation equipment, including: Infrared light source; Infrared digital camera; Visible light digital camera; a visible light source positioned adjacent to the visible light digital camera; An observation area, between the infrared surface light source and the infrared digital camera; as well as a controller programmed to control the infrared light source and the infrared digital camera to take a first photograph of an item in the viewing area such that the item is backlit by the infrared light source, and to take a second photograph of the viewing area using the visible light digital camera and to control the visible light source; The infrared digital camera, the visible light digital camera, and the visible light source are installed on a gantry.

22. The drug dispensing apparatus of claim 21, wherein: The drug dispensing device is a dispensing aid including a display; and The controller is further programmed to guide a user of the device through a dispensing task using prompts displayed on the display.

23. The drug compounding device of claim 21, wherein the drug compounding device is a compounding robot.

Images