Portable device for capturing images of medical events to reduce medical errors

Abstract

The present disclosure relates to a portable device for capturing images of medical events to reduce medical errors. Devices and methods are provided that have a camera and a medical event image capture application to capture images of medical condition management events involving products with markings, perform image processing and analysis of the captured images to discern product characteristics, artifacts, and / or related information from the markings and other image elements in the captured images, and perform human-machine interaction operations or issue an alert to the user about the selected information or request input or otherwise educate the user about aspects of medical condition management. The medical event image capture application provides any one or more of confirmation of correct or compatible products, dose confirmation, and detection of defective products or medications or misuse. The medical event image capture application automatically records medical event data into the patient's electronic record and assists in healthcare management.

Description

Technical Field

[0001] Exemplary embodiments generally relate to using a portable handheld device such as a smart phone to capture an image of a medical condition management event involving medical equipment, and a related smart phone application that processes the image and interacts with the user(s). Exemplary embodiments generally relate to a medical event image capture application that processes images of medical condition management events to reduce medical errors that may be caused by using incompatible injection devices and medications, drawing an incorrect amount of medication into a syringe or pen injector prior to delivery, using contaminated or incorrect medications, or damaged or incorrect injection supplies, etc. Background Art

[0002] Drug non - adherence is a globally important issue, especially in diabetes care. Fifty percent (50%) of all patients do not take their medications as prescribed. Non - adherence directly results in hundreds of thousands of deaths and avoidable medical and related costs in the billions of dollars.

[0003] Some smart phone applications use pictures of prescription labels to help patients re - order when their prescription drug supply runs low. However, these applications cannot directly identify the drug or dosage before the patient takes the medication and are not useful for syringes or pen injectors.

[0004] There are some smart phone applications that help users record medical events, such as injections. There are some smart injection devices that can help users automatically record the amount of drug dispensed and / or the amount of drug delivered.

[0005] Nonetheless, there remains a continuing need for methods and devices that help users (e.g., patients, their caregivers, their healthcare providers, and other medical condition management stakeholders such as payers / insurance companies, pharmacies, and medical product suppliers and distributors) obtain and use information related to medical condition management events to prevent medical errors (such as drug delivery errors), improve related processes (such as replenishing medical supplies), track adherence to medical condition management protocols or regimens, and share information among medical condition management stakeholders for purposes of optimal patient care treatment planning, billing, and insurance coverage. Summary of the Invention

[0006] Through exemplary embodiments, the above and other problems are overcome and additional advantages are achieved.

[0007] In accordance with aspects of the illustrative embodiments, a portable device for capturing images of medical events to reduce medical errors includes: an imaging device for imaging at least one medical product used during a medical event; a memory storing the images captured by the imaging device and program instructions for processing the captured images; a user interface configured to generate an output to a user; and a processor. The processor is adapted to execute the program instructions to: analyze the captured images associated with a medical event to detect characteristics of the medical product, the characteristics being selected from the group consisting of markings on the medical product and designated attributes of the medical product, analyze the detected characteristics to determine when a medical error has occurred. The medical error corresponds to a situation where the medical product is incompatible with the medical event, mishandled by the user, or malfunctioned. The processor is configured to generate an output to the user via the user interface that includes an alert related to the medical error.

[0008] One aspect of the exemplary embodiments provides a portable device, wherein at least one of the captured images in its memory corresponds to a medical event involving at least two medical products used together; and its processor is configured to: analyze at least one of the captured images to detect markings on each of the at least two medical products, analyze the markings on each of the at least two medical products using previously stored medical product data that is locally or remotely accessible by the processor, the previously stored medical product data including markings for corresponding products in a plurality of different medical products and, for each of the plurality of different medical products, corresponding markings for one or more other medical products indicated as being compatible with that medical product, and generate an output to the user when the processor determines, based on the previously stored medical product data, that the at least two medical products are incompatible.

[0009] One aspect of the exemplary embodiments provides a portable medical device, wherein the medical device is a drug delivery device having markings; and the processor is configured to: analyze a captured image of the drug delivery device and detect the markings, and analyze the captured image of the drug or other captured images of the drug of the drug delivery device and detect an indication of the amount of drug being delivered by the drug delivery device. Additionally, using previously stored medical product data that is locally or remotely accessible by the processor, the previously stored medical product data including a plurality of different drug delivery devices and their corresponding markings and, for each of the plurality of different drug delivery devices, specifications of the designated amount of drug that can be delivered via that drug delivery device, the processor determines the designated amount of drug corresponding to the drug delivery device associated with the markings detected from the captured image, and generates an alert via the user interface when it is determined that the detected indication of the amount of drug to be delivered is different from the designated amount.

[0010] For example, the detected indication of the amount of drug to be delivered corresponds to a mark associated with at least one of the level indicators adjacent to the dose input on the pen injector or the liquid level in the syringe barrel in the captured image. As another example, the processor uses an algorithm selected from two-dimensional image processing algorithms and three-dimensional image processing algorithms to analyze the captured image or other captured images and detect the amount of drug to be delivered by the drug delivery device.

[0011] One aspect of the exemplary embodiment provides a portable device, wherein the medical device is a drug delivery device having markings; and wherein the processor is configured to: analyze a captured image of the drug delivery device and detect the markings. Additionally, using previously stored medical product data that can be accessed by the processor locally and / or remotely, the previously stored medical product data including a plurality of different drug delivery devices and their corresponding markings and, for each of the plurality of different drug delivery devices, specifications of the designated amount of drug that can be delivered via the drug delivery device, the processor determines the designated amount of drug corresponding to the drug delivery device associated with the markings detected from the captured image, and generates an alert via a user interface when it is determined that the amount of drug prescribed to be delivered is different from the designated amount.

[0012] The processor is further configured to: for example, analyze the captured image of the drug in the drug delivery device or other captured images and detect the amount of drug to be delivered by the drug delivery device, the detected indication of the amount of drug to be delivered corresponding to a mark associated with at least one of the dose input on the pen injector and the level indicator adjacent to the liquid level in the syringe barrel in the captured image, and generate an alert via a user interface when it is determined that the amount of drug prescribed to be delivered is different from the detected indication of the amount of drug to be delivered.

[0013] One aspect of the exemplary embodiment provides a portable device, wherein the medical device is a drug delivery device; and the processor is configured to analyze a captured image of the drug delivery device, detect the amount of drug to be delivered by the drug delivery device, and store the detected indication of the amount of drug to be delivered in a memory device. The detected indication of the amount of drug to be delivered corresponds to, for example, a mark associated with at least one of the dose input on the pen injector and the level indicator adjacent to the liquid level in the syringe barrel in the captured image.

[0014] One aspect of the exemplary embodiment provides a portable device, wherein the program instructions include at least one of a two-dimensional image processing algorithm and a three-dimensional image processing algorithm for the processor to analyze the captured image.

[0015] One aspect of an exemplary embodiment provides a portable device, wherein the processor is configured to: analyze at least one captured image to detect characteristics of a medical product, the characteristics including at least one specified attribute of the medical product selected from the group consisting of: a selected color of the medical product, a selected size of the medical product, a selected shape factor of the medical product, the presence of a safety mechanism on the medical product, the absence of a safety mechanism on the medical product as compared to a stored image of the medical product having a safety mechanism, and analyze the detected characteristics using previously stored medical product data that is accessible to the processor via at least one of local and remote access to determine whether a medical error has occurred, the previously stored medical product data including specified specifications of image characteristics of the medical product corresponding to at least one specified attribute.

[0016] For example, the medical product is a liquid drug drawn into a syringe, and at least one specified attribute of the liquid drug is selected from the group consisting of: the opacity of the liquid drug, the presence of air bubbles in the liquid drug, the presence of particulates in the liquid drug. As another example, the previously stored medical product data includes specified specifications of image features for the at least one specified attribute of the liquid drug.

[0017] One aspect of an exemplary embodiment provides a portable device, the portable device being at least one of a mobile phone and a computing device having a wireless communication interface, and the memory being configured to store at least one integrated disease management (IDM) application, the IDM application including an IDM personal application operated by a user as a patient and / or an IDM professional application operated by a healthcare professional. The processor is also adapted to execute instructions according to the IDM application to operate the portable device in a cloud configuration with a remote IDM system, whereby the IDM application transmits data to and receives data from the IDM system during an application session.

[0018] One aspect of an exemplary embodiment provides a portable device, wherein the portable device operates according to the IDM personal application to transmit information from a captured image to the IDM system and store the information from the captured image at the IDM system, the information being selected from the group consisting of: a dose amount determined from at least one of the captured images, a medical event date and / or timestamp determined from at least one of the captured images, and / or a medical product identified from at least one of the captured images.

[0019] One aspect of an exemplary embodiment provides a portable device that operates according to an IDM professional application to determine patient information from information stored in an IDM system, the patient information including compliance data for a regimen of a prescription based on information related to a dose amount and a medical event date and / or time, medical product prescription update data based on information related to medical products identified from a captured image and a medical event date and / or time corresponding to the use of these medical products, and / or billing data corresponding to medical products identified from a captured image and at least one of a medical event date and / or time corresponding to the use of these medical products.

[0020] One aspect of an exemplary embodiment provides a portable device that is capable of wirelessly connecting to at least one other medical condition management device and obtaining medical event information therefrom. The processor is also adapted to execute instructions according to an IDM application to transmit the medical event information to an IDM system.

[0021] One aspect of an exemplary embodiment provides a portable device, wherein the cloud configuration includes a private cloud and a public cloud, and the portable device operates according to an IDM application to determine whether at least one of information related to a user and other data stored in a memory is proprietary data or non-proprietary data, and selectively transmits the proprietary data via the private cloud and the non-proprietary data via the public cloud.

[0022] One aspect of an exemplary embodiment provides a portable device for capturing an image of a medical event, including: an imaging device for imaging at least one medical product used during a medical event; a memory storing the image captured by the imaging device and program instructions for processing the captured image; a user interface configured to generate an output to a user; and a processor adapted to execute the program instructions to: analyze the captured image associated with the medical event to detect a characteristic of the medical product, the characteristic being selected from the group consisting of a mark on the medical product and a designated attribute of the medical product, store data related to the detected characteristic in the memory; and generate an output to the user via the user interface using the data related to the detected characteristic.

[0023] One aspect of an exemplary embodiment provides a portable device, wherein the portable device is a monitor for a selected medical condition, and the detected characteristic is a monitoring parameter detected by the monitor and indicated via a user interface associated with the monitor.

[0024] One aspect of an exemplary embodiment provides a portable device having a process that is also adapted to execute the program instructions to record a date and / or time associated with the detected characteristic.

[0025] In one aspect of the exemplary embodiments, the monitor is selected from the group consisting of a pulse oximeter, a thermometer, a blood pressure monitor, and a blood glucose monitor.

[0026] Additional and / or other aspects and advantages of the illustrative embodiments will be set forth in the following description, or will become apparent from the description, or may be learned by practice of the illustrative embodiments. The illustrative embodiments may include apparatuses and methods for operating these apparatuses having one or more than one aspect and / or one or more features and combinations thereof. The illustrative embodiments may include one or more features and / or combinations of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or other aspects and advantages of the illustrative embodiments will be more readily understood from the following detailed description in conjunction with the drawings, in which:

[0028] Figure 1 Devices having a medical event image capture application and different types of exemplary drug delivery products are depicted in accordance with the illustrative embodiments;

[0029] Figure 2 is a block diagram of a device having a medical event image capture application in accordance with the illustrative embodiments Figure 1 thereof;

[0030] Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 4A , Figure 4B and Figure 5 each depict a device having a medical event image capture application in accordance with the illustrative embodiments Figure 1 thereof that captures images of exemplary drug delivery products;

[0031] Figure 6 is a flowchart of exemplary operations performed by a device having a medical event image capture application in accordance with the illustrative embodiments including Figure 1 thereof;

[0032] Figure 7A , Figure 7B , Figure 7C , Figure 7D , Figure 7E , Figure 7F , Figure 7G and Figure 7H and Figure 8A , Figure 8B , Figure 8C , Figure 8D and Figure 8E and Figure 9A , Figure 9B andFigure 9C illustrates an example GUI screen generated by a device with a medical event image capture application according to an illustrative embodiment; and Figure 1 an example integrated disease management system of a device with a medical event image capture application using

[0033] Figure 10 is according to an illustrative embodiment. Figure 1 of a device with a medical event image capture application.

[0034] Throughout the drawings, like reference numerals will be understood to refer to like elements, features, and structures. Detailed Description

[0035] Reference will now be made in detail to the illustrative embodiments illustrated in the drawings. The embodiments described herein are illustrated by reference to the drawings and are not intended to limit the illustrative embodiments.

[0036] Referring Figure 1 and Figure 2 and according to an illustrative embodiment, a medical condition management event image capture application 40 is described herein, which can be a stand-alone application on a smart phone 20 or other portable device with a camera (e.g., iPad), or can be provided as an enhanced feature of a digital health (DH) application of a smart phone or other smart connected device. The medical event image capture application 40 uses an image of a marker 24 on a medical product 22 or an image of the medical product 22 (e.g., a camera image of the product 22 with the marker 24, as indicated at 25 in Figure 7B ) to automatically access and / or record additional information to assist in medical condition management. Different functions of the application 40 use additional information from the captured medical product image to reduce medical errors (e.g., the amount of mis-injected medication due to misuse or defect of the medical product).

[0037] Figure 1 shows an example smart phone 20 with a medical event image capture application 40 and a plurality of example medical products 22, such as but not limited to an injection pen needle assembly and an associated injection pen product, a syringe injection product, and a syringe safety injection product (e.g., BD AutoShield Duo TM pen needle and BD SafetyGlide TMA 6mm insulin syringe, which is designed to help prevent accidental needle stick injuries during injection, especially in a clinical setting), etc. Product 22 may be provided with different types of markings 24, which are printed or engraved directly on the product, or indirectly applied to product 22, for example, using a label including the markings. The markings 24 may be one or more of alphanumeric text, symbols, different colors, and optical recognition codes, such as barcodes, Quick Response (QR) codes, and Universal Product (UPC) codes, as well as other types of markings, the images of which can be captured via a camera and processed according to an aspect of the illustrative embodiment to decode them into relevant information regarding the item to which the markings are applied or relevant events related to the use of the item.

[0038] For example, the illustrative embodiments described herein refer to diabetes management and insulin injection. It should be understood that the markings, image capture, and application processing of the images according to the illustrative embodiments to obtain medical condition management event information and human-machine interaction based on that information can be used to reduce errors regarding the management or treatment of other medical conditions that require the use of various devices and medical condition management processes, such as surgical instruments, blood collection and delivery products, drugs other than long-acting and short-acting insulin (e.g., drugs related to hormone therapy, GLP-1, rheumatoid arthritis, or Crohn's disease treatment, and other drugs that require a dosing regimen and a certain level of control and monitoring), etc. For example, the illustrative embodiments can be used to reduce medical errors associated with self-injection of other types of drugs, correctly use surgical tools for selected medical procedures, correctly use equipment for IV infusion of liquid medicine to patients, etc. Additionally, the illustrative embodiments described herein are beneficial for a variety of different injection applications, such as veterinary treatment using an injection protocol, in addition to human patient injection events.

[0039] This document describes example embodiments regarding diabetes management and related injection products and events. However, it should be understood that these example embodiments can be implemented regarding other types of human and non-human animal medical conditions, medical events, and related condition management products. Additionally, the medical events need not be related to drug administration (e.g., instead, it can be used for surgical instrument preparation). Further, any drug administration application need not be limited to injections. For example, application 40 can be used for information capture and management of oral medication treatment regimens and / or topical treatment regimens. Regarding diabetes management, diabetes care companies produce a large number of insulin delivery or injection products that are essential for the treatment of diabetes in diabetic patients worldwide. These injection products are used by patients for self-injection and caregivers of diabetic patients, and can include, but are not limited to, injection pens, pen needle assemblies, syringes, needles of different sizes, different types of insulin in different form factors (e.g., vials, pre-filled syringes, cartridges for injection pens). For example, a patient's injection regimen may require a selected type of syringe, needle, and type of insulin, such that using the wrong type of insulin or syringe can affect the accuracy of the intended dose amount.

[0040] According to an illustrative embodiment, a device 20(1) with a medical event image capture application 40 provides image capture and image processing of (one or more) medical event devices to determine one or more of medical equipment correctness, delivery amount accuracy, and drug status; and (2) generates alerts and user guidance via a graphical user interface on the device 20 to reduce medical errors. Figure 2 is a block diagram depicting an example device 20. The device 20 is referred to as a smart phone, but it should be understood that the device 20 can be a dedicated medical management device or other portable handheld device (e.g., iPad) with a marked image capture or reader device 28 (such as a camera). The device 20 includes a processor 26 and a memory 36 that can store the medical event image capture application 40 according to the illustrative embodiment, as well as other device data, images, and applications. The device 20 can have one or more wireless communication interfaces 38, such as, for example, a wireless communication interface and a cellular communication interface enabled for. The device 20 can also have different user interfaces, such as a microphone 32, a touch screen 30, or one or more of other display devices that generate a graphical user interface (GUI) screen (such as the graphical user interface screen of the medical event image capture application 40), an optional keyboard, or other user input devices (not shown), and an audio signal output device (e.g., a speaker or buzzer) 34.

[0041] The medical event image capture application 40 is program code that provides image capture operations for labeled and / or injected products and image processing operations for the captured images. The image processing operations for the captured images can (1) decode or otherwise discern artifacts and associated information from the labels and other image elements in the captured images, and (2) perform human-machine interaction (HMI) operations or issue alerts to the user regarding the selected information and request input or otherwise educate the user regarding the relevant medical event. For example, the image capture operation captures an image from the device camera 28. The image processing operations for the captured images can implement two-dimensional (2D) image and / or three-dimensional (3D) image processing algorithms to detect the selected artifacts from the captured image(s). The image processing operations for the captured images can optionally include identification operations such as a QR code reader, barcode or UPC code reader, or optical character recognition (OCR) operation within the application 40. The HMI or other operations of the medical event image capture application 40 determine how the detected artifacts affect the medical condition management event and generate a GUI screen or other HMI output (e.g., an audible query or message output by the speaker 34 to the user) to educate the user or request user input.

[0042] According to an illustrative embodiment, with reference respectively to Figures 3A to 3E , Figure 4A and Figure 4B as well as Figure 5 at least three applications of the medical event image capture application 40 are described, namely, (1) confirming the correct injection device, (2) dose confirmation, and (3) detecting defective drugs, devices, or improper use. The application 40 can provide only one of these applications, a subset of any two of these applications, or all of these applications.

[0043] Regarding the first application of the medical event image capture application 40 (i.e., confirming the correct injection device 22), with reference to Figures 3A to 3E, a smartphone 20 with a medical event image capture application 40 is depicted, with one or more devices 22 having markings 24 (e.g., a vial of insulin and a syringe) within the image range 42 of the smartphone camera 28. The application 40 can identify the correct injection device by means of image recognition, QR codes or other machine-readable codes, the color or other distinguishing features of markings on the syringe or vial. Particularly in the case of syringes, there may be various needle sizes, barrel capacities, and scale markings specific to a certain type of drug. An example with an insulin syringe is shown below, but it represents most treatments using different types of drugs. Some injection products, such as those commercially available from Becton, Dickinson and Company or "BD", already have unique markings indicating that they should be used with certain types of drugs. These unique markings or tags (e.g., QR codes) can be made readable by the application 40 via the image processing algorithm of the application 40 to ensure that the patient or caregiver is using the correct type of syringe for the correct insulin, thereby reducing medication errors.

[0044] For example, the application 40 can be programmed to consult information stored locally or accessed remotely, including a table or other data storage structure, for these unique markings 24 associated with a particular injection product 22, thereby performing comparisons or other analyses to identify the item 22 in the captured image. Alternatively, these unique markings or tags 24 can be detected via the application 40 to automatically navigate the user to online educational materials (e.g., videos) regarding injection or other medical condition management skills. For example, if the tag 24 is a QR code, the application 40 can have a QR code scanner that converts the tag 24 into some useful form (such as the standard URL of a website). This form can be a symbol or character that classifies the device (e.g., belonging to a class of related products as described below), or the decoded QR code can direct the smartphone 20 to the URL of a web-based form via the smartphone's browser for a lookup operation related to the relevant product. Thus, the camera 28 in the smartphone 20 equipped with the application 40 having an integrated tag reader can scan the image of the QR code or other tag 24 on the item 22 to display text (e.g., contact information or instructions displayed via the GUI screen 30), or connect to a wireless network (e.g., connect to an HCP repository), or open a web page in the smartphone's browser. When the medical device or product 22 appears to be mismatched, the application 40 can also generate a GUI screen or other alert, as Figure 7F shown.

[0045] In Figure 3AIn the example depicted, the medical event image capture application 40 can be programmed to detect alphanumeric markings 24 (e.g., "U-500") in a captured image of an insulin vial and syringe held in front of the camera 28. The application 40 can be programmed to then consult locally stored or remotely accessed information that includes codes or markings corresponding to a respective medical product family (e.g., different sized syringes and corresponding types of insulin) that are compatible when used together for an accurate dose injection, such that the application 40 can confirm whether the user is using compatible devices from the same family of medical devices for an effective or accurate administration of medication.

[0046] Continuing to refer Figure 3A , manufacturers of diabetes care insulin and / or injection supplies typically provide markings (e.g., stock keeping unit (SKU) numbers or other product identification markings) on their respective products (e.g., syringes, insulin pens, needle assemblies, insulin vials and cartridges, injection safety products, etc.). According to one aspect of the illustrative embodiment, manufacturers and other injection supply companies can generate a table of compatible injection products in which the product codes of selected products are linked to a family of products that can be used to provide an injection, and the family of products is given a selected code (e.g., alphanumeric nomenclature or other machine-readable markings such as the QR code 24, as Figure 3B shown). When the user operates the smart phone 20 using the medical event image capture application 40 to capture an image of the item 22 that the user is employing for self-injection or injection into a patient, the item 22 within the viewing range 42 of the camera 28 of the smart phone is captured in the image, and the processor 26 processes the image according to the application 40.

[0047] The captured image can include image pixels representing Figure 3A the item 22 in Figure 3A (i.e., e.g., vial and syringe). It should be understood that

[0048] For example, referring Figure 3B, shows an example of a U40 syringe 22 with a QR code 24 that can be placed as part of the manufacturing process. The QR code reads: 0.3ml x 12.7mm U40. When the user scans the code before injection, the application 40 identifies the device 22 and confirms that the correct device 22 is being used, or warns the user if this is not the case. For example, using a U40 syringe with U100 or U500 insulin will result in an incorrect insulin dose being delivered.

[0049] The marker 24 can be a specific color on a label, an alphanumeric product name (e.g., U-500), a product code (e.g., Figure 3B the QR code 24 on the syringe 22 in ) or a product family (i.e., with or without a product code) identified by a combination of a QR code or other machine-readable code or marker. The product family code can be, for example, a single character or multiple characters, and these characters can be alphanumeric characters or symbols or other markers. In an example, a compatible injection product table can include product families “A, …, N” and can be stored locally or remotely with respect to the device 20. If all captured image items 22 have the same product family code “A”, then the processor 26 determines that the items 22 are compatible and optimized to give an accurate dose. On the other hand, if the processor 26 identifies two or more different product family codes in the (one or more) images of the captured items 22 (e.g., the QR code on a vial indicates product family “A” while the QR code on a syringe indicates product family “B”), then the processor 26 can be operated via the medical event image capture application 40 to generate an alert to the user. For example, the processor 26 can generate a GUI screen displayed on the touch screen 30 that advises the user of the detected incompatibility of the items 22 and optionally recommends a different size syringe from product family “A” to replace the image-captured syringe of family “B” for use with the detected vial of family “A”.

[0050] Reference Figure 3C , depicts an example of a BD AutoShield pen needle 22 with a QR code 24 that can be used to help drive better usage. In this case, the QR code 24 reads “Best injection practice” and triggers a feature within the medical event image capture application 40 to playback local content or take the user via a smartphone 20 browser to a website demonstrating good injection practice. Similar QR codes can also be used for patient education on other medical condition management topics.

[0051] Example product families in a local or remotely stored table or other data storage structure accessed via the application 40 can also be defined according to the healthcare environment, i.e., the clinical environment in which a healthcare provider (HCP) administers an injection to a patient, or the home healthcare environment in which the patient self-administers an injection or an injection is administered to the patient by a home healthcare caregiver or family member. Further reference is made to Figure 3C and the medical event image capture application 40 can optionally be configured to generate an alert to the HCP when a captured image of an injection product is missing a selected product family code 24 specified for an injection safety product. The alert can remind the HCP to use an injection safety product (e.g., BD AutoShield pen needle or BD SafetyGlide 6mm insulin syringe) designed to help prevent accidental needle stick injuries during an injection that may occur in the clinical environment.

[0052] According to one aspect of the illustrated embodiment, the item 22 captured in the image can be insulin pen injection supplies rather than syringe injection supplies, such as Figure 3D the pen needle 22 with the marking 24 as shown in Figure 3E or the package 22 of the pen needle of the selected size and the marking 24 on the package 22 as shown in

[0053] Regarding the medical event image capture application 40 and Figure 4A and Figure 4BSecond application (i.e., dose confirmation), according to an illustrative embodiment, the medical event image capture application 40 is configured to process the captured images to determine the amount of drug to be delivered. For a person administering an injection to themselves or to another person, dose measurement, dose confirmation, and tracking can be a challenge, which can be alleviated by the functionality of the medical event image capture application 40. For example, the application 40 can also be used to ensure that the patient is drawing the correct dose, particularly in the case of a syringe, but also in the case of an injection pen. In addition to dose confirmation, the combination of the functionality of the application 40 to determine the correct device and the correct dose is expected to drive improved compliance with treatment, reduce the likelihood of medication errors, and result in better outcomes for the patient.

[0054] Reference Figure 4A and Figure 4B , an item 22 for injection can be placed within the viewing range 42 of the camera 28. An image is captured that includes pixels representing injection level or delivery amount markings. The medical event image capture application 40 can be provided with 2D and / or 3D image processing algorithms that are configured to discern, for example, the syringe plunger position 44 ( Figure 4B ) corresponding to the amount drawn (i.e., the amount injected via the syringe 22) or the injection pen scale markings 44 ( Figure 4A ) corresponding to the amount of drug dialed to be delivered via the injection pen.

[0055] There are applications that allow a smartphone to communicate wirelessly with a wireless-enabled injection pen to wirelessly receive the dialed and / or delivered dose information. Thus, since these injection pens require a wireless communication interface to make them wireless-enabled devices, they introduce additional complexity and thus additional cost. In contrast, the medical event image capture application 40 with camera image processing according to an illustrative embodiment allows for automatic dose capture and confirmation of the correct dose without a wireless exchange between the device(s) 22 and the smartphone 20, and thus, does not add the complexity and cost of the injection device(s) 22 such as a pen.

[0056] According to a third application of the illustrative embodiment (i.e., determination of device or drug malfunction), the medical event image capture application 40 is configured to process the captured images to determine whether the device 22 or the drug is defective. For example, the application 40 can be used to ensure that the patient is drawing the correct dose by identifying the presence of air bubbles and informing the patient. Properly aspirating a dose of medication into the syringe 22 is a critical step that involves visually detecting any air bubbles in the syringe barrel and then removing the air bubbles from the syringe barrel. Typically, impaired vision of the user makes it difficult to detect air bubbles. Advantageously, the application 40 incorporates image recognition capabilities, where it can detect and quantify the volume of air bubbles in the syringe via processing of the captured images. For example, this processing can be accomplished using 3D image analysis, which is traditionally used to count features in various scientific fields, or typical 2D projected surface area image analysis. 2D image analysis algorithms can also be used to quantify the size of the air bubbles. As described below in conjunction with Figure 7C The application 40 can generate an alert to inform the user that air bubbles can be detected and need to be removed from the syringe barrel before delivery. The alert can be visual or audible, which is particularly useful for visually impaired users.

[0057] Figure 5 An example is shown in which a projected 2D image / photo of the syringe 22 with air bubbles 46 is obtained via the image capture operation of the application 40 and analyzed by application image processing and analysis software to identify the shape and size (and thus the volume) of the air bubbles 46 using the projected surface area in the 2D image. Such image processing can have multiple uses, such as monitoring the effectiveness of the patient's usage method, providing a better teaching or training tool, and potentially even tracking the accuracy of the aspirated dose. The medical event image capture application 40 can also employ captured image processing algorithms that detect other attributes of the item 22, such as determining whether there are particulates 48 in the drug or whether the drug is opaque and not clear enough (e.g., indicating that the drug has expired or is contaminated), or whether the medical device 22 is missing a safety cap or has a bent or broken needle, and other undesirable attributes. In any case, the application 40 can generate an alert to inform the user that an undesirable attribute of the device 22 or the drug has been detected, giving the user the opportunity to address the issue before delivering an incorrect dose.

[0058] The third application of the medical event image capture application 40 is particularly beneficial in a clinical environment. For example, a smart phone 20 with the application 40 can detect visible contamination (e.g., a pen with expired medication indicated as cloudy or having floating particles). The captured image processing algorithm of the application 40 can be configured to, for example, detect the opacity of the medication or whether the device is leaking. The captured image processing algorithm can also be configured to determine from the captured image of the injection article 22 whether it lacks safety features such as a sterilization cap or other needle - stick prevention devices and generate an alert to the HCP. Hospital safety procedures may require these safety features, and the application 40 can ensure compliance and assist with inventory management and restocking. For example, the application 40 can detect the devices and other supplies for which it captures and processes (one or more) images for an injection event. The processor 26 programmed via the application 40 can inform individuals and clinical environment personnel about supply levels based on the number of captured images of the supplies 22 used to assist with automatic re - ordering of supplies. In fact, many clinical environments provide iPhones or mobile devices to HCPs to alert and message when arriving at the patient's bedside. The application 40 can be provided to the HCP device 20 to enable them to capture images of medical condition management events and, by processing the image, collect information that helps automatically record injection data into the patient's electronic record, as well as assist with clinical environment management records regarding billing, inventory management and re - ordering, and care plan compliance and clinical effectiveness.

[0059] Example image processing algorithms for processing captured images according to illustrative embodiments include, but are not limited to, any of the following image processing and / or image analysis algorithms: image segmentation (e.g., for identifying the correct location of boundaries); image representation (e.g., for expressing the image in the form of a pixel map); detection and recognition (e.g., for identifying pre - quantified features); motion estimation (e.g., when using dynamic images); tracking (e.g., for tracking features that have been identified during the detection step(s) of the illustrative embodiment); surface and shape estimation (e.g., for bubble detection and volume quantification according to the illustrative embodiment); enhancement (e.g., for contrast stretching, noise filtering, histogram modification); restoration (e.g., for editing boundaries, aligning contrast, compensating for exposure); analysis (e.g., for identifying, classifying, and / or counting features); reconstruction; and data compression. Example platforms that support these example image processing and / or image analysis algorithms include, but are not limited to, commercially available platforms such as Matlab, Image J, Icy, ENVI, FIJI, ImageTool, ImagePro Plus, etc. and open - source platforms.

[0060] Figure 6is a flowchart of an example operation of a medical event image capture application 40 according to the illustrated embodiment. It will be understood that all three applications (i.e., (1) a correct or compatible device, (2) dose confirmation, and (3) medical device 22 and / or drug malfunction) or any two of these three applications, or only one of these applications, may be provided for application 40. The user uses the camera 28 function of the smart phone 20 to take an image of the device 22 for implementing a medical administration event (e.g., an injection) (block 50). The image processing operation of the captured image of application 40 determines the marker 24 or optionally determines other attributes of the (one or more) devices 22 (blocks 52 and 54). If it is determined from the captured image that there is a marker 24 or a specific attribute, then application 40 is configured to control the processor 26 to determine relevant device information (e.g., a table or other data structure stored in a local or remote computer memory device), such as whether the medical device 22 detected in the captured image belongs to the same product family or is otherwise compatible, or indicates the amount to be delivered, or indicates a malfunction (block 56). If the device 22 in the captured image is not compatible (e.g., part of a different product family as described above), then the processor 26 will generate an alert or a GUI screen to inform the user of an incorrect dose that may result from the incompatibility of the device 22 with the drug or other user errors or misuse of the medical device 22 when drawing the drug (block 60). As described above, the marker 24 detected by the image processing function of application 40 may also determine whether a QR code or other marker indicates that the user should receive a playback of educational information (blocks 62 and 64).

[0061] Continuing to refer Figure 6 , the processor 26 is controlled by the medical event image capture application 40 to determine whether the correct dose amount has been drawn, and if not, then generate an alert or a GUI screen for the user (blocks 66 and 68). If the processor 26 detects a device or drug malfunction, e.g., as described in connection with Figure 5 , then the processor 26 will generate an alert or a GUI screen to inform the user of the problem (blocks 70 and 72). Once it is confirmed via block 74 that the correct dose is to be delivered (e.g., via user input on a GUI screen generated by application 40 on the touch screen 30), the medical event data or information (e.g., one or more confirmed doses, detected malfunctions, product 22 codes, and other data obtained via the image processing operation of the captured image of application 40) may be stored locally or remotely for access and use by the patient and / or other medical condition management stakeholders (block 76). For example, the medical event data or information may be automatically uploaded to a repository to be included in the patient's electronic record for medical billing, for automatic replenishment of medical supplies 22 and / or care plan compliance tracking, and such as incorporated into the following in connection with Figure 10Other uses in the integrated disease management system described (box 80). As described in co-owned U.S. Patent No. 10,173,015, the application 40 can optionally be used with an injection site rotation algorithm (box 78). For example, the injection site rotation algorithm can recommend the next body site injection location, and the injection application 40 can capture an image of the injection event and confirm the delivery amount to record the injection. The injection site rotation algorithm can also record the body site injection.

[0062] Figure 7A , Figure 7B , Figure 7C , Figure 7D , Figure 7E , Figure 7F , Figure 7G and 7H is an example GUI screen generated by the medical event image capture application 40 on the device 20 for guiding the user to draw up the correct dosage amount (e.g., using a syringe) and to capture an image of the QR code or other marking 24 on the syringe 22, such as Figure 7A The captured image is shown in Figure 7B In screen 92. Figure 7C The user is alerted to the presence of detected bubbles in screen 94, which are determined on the captured image using an image processing algorithm. After eliminating the bubbles and drawing the correct amount of medication ( Figure 7D and Figure 7E ), the application 40 generates the screen 100 ( Figure 7F ), which requests confirmation that the correct syringe and type of insulin (e.g., detected in the vial using marking 24) are being used. Figure 7G and Figure 7H Is the confirmation of the delivery screen.

[0063] Figure 8A , Figure 8B , Figure 8C , Figure 8D and Figure 8E An example GUI screen generated by application 40 on device 20 for guiding a user to draw a dose (eg, using a syringe) is illustrated, wherein Figures 7B to 7D In addition to using application 40 to capture information from components such as injection devices or drug vials, application 40 is also used to record information from other medical devices such as monitors (e.g., blood glucose monitors (BGMs), pulse oximeters, thermometers, blood pressure monitors, and other devices that may not have wireless communication interfaces). Figure 9A , Figure 9B and Figure 9CIllustrated is an example GUI screen generated by application 40 on device 20 to capture information 24 from the display of BGM 22 using photo capture as described herein. Figure 9A Shown is a device (e.g., smart phone 20) that captures information 24 from the display of BGM 22 within the image range 42 of a smart phone camera 28. A camera image of information 24 (e.g., a glucose reading of 206 mg / dL at 10:43 am) is indicated at Figure 9B 25 in. Referring to Figure 9C , application 40 records the information in the camera image to the memory on smart phone 20 and displays the information to the user on smart phone screen 30. This passive information capture and recording using application 40 makes devices such as monitors without a wireless communication interface more versatile and cost-effective for users who cannot afford access to, for example, a continuous glucose monitoring system that automatically and wirelessly records glucose readings to another device. Application 40 provides an affordable solution for these users by providing an option to electronically record data on the monitor screens of these users, thereby reducing the likelihood of human error and addressing the known challenges of blood glucose tracking and transcription errors that they face.

[0064] The illustrative embodiments disclose various ways to better engage with the user and leverage the advantages of both the injection product 22 and the medical event image capture application 40 to enhance the user experience. Application 40 is used to identify specific features and activities, confirm that they are expected, provide confirmation to the user, and also enable the recording and tracking of information for later review. Overall, the integrated use of (one or more) devices 22 and application 40 is expected to drive better adherence and improved patient treatment outcomes. In addition, the combination of correct device 22 detection and the monitoring and recording of the delivered dose information is expected to provide more accurate data to enable better clinical decision-making and reduce the likelihood of medication errors. While application 40 (and the combined functionality of device 22) is primarily targeted at the self-injection patient population, it can equally be easily utilized in other settings (e.g., institutions and alternate sites) and also by caregivers (e.g., nurses, family members, etc.).

[0065] In addition to these various insulin delivery or injection products 22, diabetes care companies can also provide digital health (DH) applications, such as the BD Diabetes Care application (available from Becton Dickinson and Company), which enables patients to maintain improved control over their diabetes treatment regimens. For example, the BD Diabetes Care application helps patients and / or their caregivers record injections, record blood glucose values or glucose monitoring, record carbohydrate intake, and record exercise, all of which affect the patient's insulin injection needs.

[0066] A medical event image capture application 40 can also be integrated into a digital health application (e.g., the BD Diabetes Care application). For example, the medical event image capture application 40 and the information it generates can be automatically combined with other digital health application content (such as injection, exercise, carbohydrate intake, and blood glucose reading logs) to help patients and disease management stakeholders track a patient's compliance with a prescribed disease management regimen (e.g., the extent to which a patient maintains a target blood glucose level), reorder supplies (e.g., home healthcare supplies such as self-injection devices and medications, as well as pharmacy inventory), and automatically ship prescription drugs and medical supplies to the patient, or for commercial settings, inventory tracking, billing for medical events captured in a clinical environment, and the like. Accordingly, the embodiments shown herein provide convenience and other advantages for different categories of users (e.g., self-injections and caregiver-administered injections) in different categories of environments (e.g., home environments or other alternate sites such as nursing homes, long-term care facilities, and rehabilitation facilities, as well as clinical / hospital environments).

[0067] The medical event image capture application 40 can be a stand-alone application that communicates with a user (e.g., a patient) or other stakeholders in the user's medical condition management team, such as caregivers (e.g., parents, spouses, school nurses), healthcare providers, clinical environment administrators, pharmacies, payers (e.g., insurance companies), and medical device suppliers and distributors.

[0068] According to an illustrative embodiment, the medical event image capture application 40 can also be integrated into an Integrated Disease Management (IDM) system 150 as shown in FIG. 9. The IDM system 150 is understood to be useful for managing other types of diseases, including collecting, analyzing, and disseminating information to assist disease stakeholders (e.g., patients, caregivers, healthcare providers, disease management companies, pharmacies, product suppliers related to disease management, insurance companies, and other payers) in managing one or more diseases. The IDM system 150 can be used by many types of people, including but not limited to diabetic patients, non-diabetic individuals, caregivers, and healthcare professionals or healthcare entities such as disease management companies, pharmacies, product suppliers related to disease management, insurance companies, and other payers. For ease of description, this disclosure refers to the IDM system in terms of users. The mention of "user" is intended to cover all types of users without limitation. Additionally, in some cases, this disclosure refers to patients or diabetic patients. This is in the context of non-limiting examples and is not intended to be restrictive. Thus, the mention of patients or diabetic patients is intended to refer to all types of users without limitation. The IDM system 150 can include an interactive interface that is simple, engaging, and provides scalable means for users to seek information and support when needed, so that they feel more in control of their condition.

[0069] The IDM system 150 can also include or have access to a user database and a content database (not shown). For example, healthcare professionals or related organizations can develop recommended disease management programs and recommended lifestyle choices to optimize patient treatment outcomes and store the diabetes information content in the content database. The IDM system 150 is configured to securely (e.g., encrypt) transmit data to a remote server, such as a cloud storage server, analyze the received data (e.g., disease management data), provide feedback to the user (e.g., customized feedback based on the user's data and interface interactions with the organization's content), and send all or part of the data and / or the organization's content to another user device or a remote health management access point (e.g., cloud storage), where the information can be accessed by healthcare stakeholders such as the patient's physician or other HCPs, family members or other caregivers, pharmacists, disease management companies, medical suppliers, or payers. Conversely, alerts, reminders, and interventions can be securely provided to the user (e.g., HCP) through the user's network via the IDM system 150.

[0070] User access to the IDM system 150 is via a user device 20 having an interactive interface that can be accessed via a web browser or a software application such as an application for a smart phone or a computer application. The user device 20 can be, but is not limited to, a smart phone, a smart watch, a tablet computer, a laptop computer, a computer, a personal digital assistant (“PDA”), etc. In some cases, the user device 20 is a mobile device, such as any mobile device known in the art, including but not limited to a smart phone, a tablet computer, or any telecommunications device with computing capabilities, a mobile device connection module, and preferably an adaptive user interface, such as but not limited to a touch screen. The user typically uses such a mobile device for various functions, such as making and receiving phone calls, sending and receiving text messages, and / or browsing the Internet. The user device 20 communicates with the IDM system via a wireless network and / or a wired network.

[0071] According to one aspect of the embodiment shown in FIG. 9, the IDM system operates in conjunction with an IDM personal application 140 installed on a user device 20 operated by a patient and an IDM professional application downloaded or otherwise installed on a user device 20 operated by a professional, such as a clinician 124, a pharmacist 126, a payer 130, and a pharmaceutical company 132. The IDM application (e.g., the IDM personal application 140) can operate in a cloud-dependent configuration, whereby the mobile device having the application transfers data to and receives data from the cloud (e.g., the IDM system) during, for example, an application session or periodically or continuously in the background, or the IDM application can operate in a distributed configuration, whereby the application works in an independent mode and then selectively connects to the cloud (e.g., the IDM system) as needed.

[0072] For example, the IDM personal application 140 may be displayed as a single icon on the patient's device(s) 20. The IDM personal application 140 provides an interface to the IDM system for the patient or the patient's caregiver to enable functions and experiences such as viewing dosage data, messaging with clinicians, adding meal data to the patient's stored data, importing BG data, etc. The IDM personal application 140 may incorporate the operations of the medical event image capture application 40 to obtain and store information from the captured images, such as dosage amounts, event date / time stamps associated with the captured images, products 22 identified via the captured images, and other data. The IDM professional application provides an interface to the IDM system for other users (such as clinicians 124, pharmacists 126, payers 130, pharmaceutical companies 132, or other medical companies, etc.) to enable functions such as viewing data of a patient or a patient population, sending short messages to patients, and performing dosage titration. The IDM personal (patient, caregiver) software 140 may include, for example, one or more applications. The IDM professional (clinician, pharmacist, etc.) software may be web-based, for different user types, and provide separate experiences for providers of the patient's care team, payers, and pharmacists. For example, the IDM professional application may be programmed to import data from the patient's IDM personal application 140.

[0073] Continuing to refer to FIG. 9, the user device 20 may be connected to other devices (e.g., via Bluetooth TM), such as one or more medication delivery devices (MDDs) 22 (e.g., insulin syringe 22 and / or pump indicated at 120), and other devices (such as glucose or lifestyle monitoring devices generally indicated at 122). For example, other devices can include, but are not limited to, a carbohydrate input device or application running on the user's cellular phone that allows the patient to enter the food and beverages consumed, a device or related application having an oral medication input element that allows the user to track the oral medications ingested, a BGM and / or CGM, and one or more of a device or application for entering health data such as the user's activity level. Once downloaded, the IDM personal application 140 allows the user to selectively activate additional functions associated with the corresponding smart device (such as, (one or more) MDDs) (e.g., injection pen 22 application, pump 120 application, or other dose capture application). The MDD application can provide device connectivity and data offloading functions, as well as dose data storage and access functions, dose data to cloud transfer functions, user profile creation and authentication functions, connected third-party experiences (e.g., interaction between the user and a third party such as a BG data tracking vendor), and output and analysis of dose data and BGM data. Referring to FIG. 9, some device data can be sent to the user device 20 having the IDM personal application 140 for storage on a private cloud, while other data (e.g., non-proprietary or unregulated medical device data from the device) can be transmitted by the device or its vendor 138 to the public cloud 136 for access by the user device 20.

[0074] Similarly, the IDM professional application can be selectively configured by different stakeholders to have different functions, including, for example, a patient population management sub-application and a patient treatment outcome sub-application, and a data and communication protocol application programming interface (API) that enables data transfer between the user and the system. Some examples are a proprietary cloud or “closed API” that allows the user to create an account and directly access data and functions through the application view, a commercial cloud or “open API” where the data is passed to another entity (e.g., Glooko) to facilitate use by the end user (e.g., via the open API), or a hybrid model that provides both the above open and closed API options to leverage proprietary data generated from devices with a closed API as well as data generated from devices with an open API.

[0075] According to one aspect of the embodiment shown in FIG. 9, one or more of devices 22, 120, and 122 are connected devices that can directly transmit data (e.g., drug delivery amounts and blood glucose readings) to the IDM system. Examples of connected drug delivery devices (MDDs) are described in co-owned US20160074587, which is incorporated herein by reference. A platform for communicating with the IDM system using connected devices is described below. The IDM system and connected devices (e.g., MDDs 12, 120, and other devices 122) advantageously provide an end-to-end IDM solution for people with diabetes (PWDs) and their care network (e.g., (one or more) healthcare providers, (one or more) caregivers, pharmacists, and insurance companies) to relieve the burden on PWDs and other disease management stakeholders of managing diabetes. The IDM solution can transform data to bring about an enhanced end-user experience, thus improving treatment outcomes. Although the IDM solution can be implemented as a collection of products that broadly meet the needs associated with a particular medical condition such as diabetes, the IDM solution can be configured to manage different medical conditions. The products can be hardware and / or software that provide value to a defined group of people such as patients or caregivers or professional disease managers such as healthcare providers, pharmacists, and insurance companies. The software products described herein (e.g., phone applications or computer applications) can include one or more modules, which should be understood as a collection of functions that provide a set of experiences such as, for example, predicted events, tasks, and actions.

[0076] Those skilled in the art will understand that the present disclosure is not limited to the details of the construction and the arrangement of the components set forth in the above description or illustrated in the drawings. The embodiments herein are capable of having other embodiments and of being practiced or carried out in various ways. Also, it will be understood that the language and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of "including", "comprising", or "having" and variations thereof herein is intended to cover the items listed thereafter and their equivalents as well as additional items. Unless otherwise limited, the terms "connected", "coupled", and "mounted" and variations thereof are used broadly herein and include direct and indirect connections, couplings, and mountings. In addition, the terms "connected" and "coupled" and variations thereof are not limited to physical or mechanical connections or couplings. Further, terms such as upper, lower, bottom, and top are relative and are used to aid illustration but are not limiting.

[0077] The components of the illustrative devices, systems, and methods according to the illustrated embodiments may be implemented at least in part in digital electronic circuitry, in analog electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof. For example, these components may be implemented, for instance, as a computer program product, such as a computer program, program code, or computer instructions tangibly embodied in an information carrier or a machine-readable storage device for execution or control of operations by a data processing apparatus, such as a programmable processor, a computer, or multiple computers.

[0078] A computer program may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to execute on one computer or on multiple computers at one site or on multiple computers distributed across multiple sites and interconnected by a communication network. Moreover, the functional programs, code, and code segments for implementing the illustrative embodiments can be readily interpreted by programmers in the field of the illustrative embodiments to be within the scope illustrated by the illustrative embodiments. The method steps associated with the illustrative embodiments may be executed by one or more programmable processors that execute a computer program, code, or instructions to perform the functions (e.g., by operating on input data and / or generating output). For example, the method steps may also be implemented by dedicated logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), and the apparatus of the illustrative embodiments may be implemented as dedicated logic circuitry, such as an FPGA or an ASIC.

[0079] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or executed with a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device designed to perform the functions described herein, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0080] As an example, processors suitable for executing computer programs include any one or more processors of general and special purpose microprocessors, as well as any type of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, or operatively coupled to receive data from or transfer data to or both from the one or more mass storage devices, such as magnetic disks, magneto-optical disks or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices such as electrically programmable read-only memory or ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory devices and data storage disks (e.g., magnetic disks, internal hard disks or removable disks, magneto-optical disks and CD-ROM and DVD-ROM disks). The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0081] Those skilled in the art will appreciate that any of a variety of different techniques and tricks can be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0082] Those skilled in the art will also recognize that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope illustrated by the exemplary embodiments. Software modules may reside in random access memory (RAM), flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor. In other words, the processor and the storage medium may reside in an integrated circuit or may be implemented as discrete components.

[0083] A computer-readable non-transitory medium includes all types of computer-readable media, including magnetic storage media, optical storage media, flash memory media, and solid-state storage media. It should be understood that software can be installed in a central processing unit (CPU) device and sold with it. Alternatively, the software can be obtained and loaded into the CPU device, including obtaining the software through a physical medium or a distribution system, including, for example, obtaining the software from a server owned by the software creator or from a server not owned by but used by the software creator. For example, the software can be stored on a server for distribution, for example, via the Internet.

[0084] The descriptions and drawings given above are only intended as examples and are not intended to limit the illustrative embodiments in any way. It should be particularly noted that those skilled in the art can easily combine the various technical aspects of the various elements of the various illustrative embodiments described above in many other ways, and all such aspects are considered to be within the scope of what is claimed in this application.

Claims

1. A portable device for capturing images of medical events to reduce medical errors, comprising: an imaging device for imaging at least one medical product used during a medical event; a memory storing the images captured by the imaging device and program instructions for processing the captured images; a user interface configured to generate an output to a user; and a processor adapted to execute the program instructions to: analyze the captured images associated with the medical event to detect characteristics of the medical product, the characteristics being selected from the group consisting of markings on the medical product and designated attributes of the medical product, analyze the detected characteristics to determine when a medical error occurs, the medical error corresponding to a situation where the medical product is incompatible with the medical event, mishandled by the user, or malfunctioning; and generate an output to the user via the user interface including an alert related to the medical error; wherein at least one of the captured images in the memory corresponds to a medical event involving at least two medical products used together; and wherein the program instructions when executed by the processor cause the processor to: analyze at least one of the captured images to detect markings on each of the at least two medical products, analyze the markings on each of the at least two medical products using previously stored medical product data that is locally or remotely accessible by the processor, the previously stored medical product data including markings for corresponding products in a plurality of different medical products, and for each of the plurality of different medical products, corresponding markings of one or more other medical products indicated as being compatible with that medical product, and generate an output to the user when the processor determines based on the previously stored medical product data that the at least two medical products are incompatible.

2. A portable device for capturing images of medical events to reduce medical errors, comprising: an imaging device for imaging at least one medical product used during a medical event; a memory storing the images captured by the imaging device and program instructions for processing the captured images; a user interface configured to generate an output to a user; and a processor adapted to execute the program instructions to: analyze the captured images associated with the medical event to detect characteristics of the medical product, the characteristics being selected from the group consisting of markings on the medical product and designated attributes of the medical product, analyze the detected characteristics to determine whether a medical error occurs, the medical error corresponding to a situation where the medical product is incompatible with the medical event, mishandled by the user, or malfunctioning; and generate an output to the user via the user interface including an alert related to the medical error; wherein at least one of the medical products is a drug delivery device having markings; and wherein the processor is configured to: analyze the captured image of the drug delivery device and detect the markings, analyze the captured image of the drug in the drug delivery device or other captured images and detect an indication of the amount of drug delivered by the drug delivery device, Using previously stored medical product data that can be accessed locally or remotely by a processor, determine a specified amount of a drug corresponding to a drug delivery device associated with a marker detected from a captured image, the previously stored medical product data including a plurality of different drug delivery devices and their corresponding markers, and for each of the plurality of different drug delivery devices including a specification of the specified amount of drug that can be delivered via that drug delivery device, and When it is determined that the detected amount of drug to be delivered is different from the specified amount of drug, generate an alert via a user interface.

3. The portable device according to claim 2, wherein the detected amount of drug to be delivered corresponds to a notation associated with at least one of a dose input on an injection pen and a level indicator adjacent to the liquid level in a syringe barrel in the captured image.

4. The portable device according to claim 2, wherein the processor uses an algorithm selected from a two-dimensional image processing algorithm and a three-dimensional image processing algorithm to analyze the captured image or other captured images and detect the amount of drug to be delivered by the drug delivery device.

5. The portable device according to claim 2, wherein at least one of the medical products is a drug delivery device having a marker; and wherein the processor is configured to generate an alert via a user interface when it is determined that the amount of drug in a detected prescription is different from the specified amount of drug.

6. The portable device according to claim 5, wherein the processor is configured to: Analyze the captured image or other captured images of the drug in the drug delivery device and detect the amount of drug to be delivered by the drug delivery device, the detected amount of drug to be delivered corresponding to a notation associated with at least one of a dose input on an injection pen and a level indicator adjacent to the liquid level in a syringe barrel in the captured image, and When it is determined that the amount of drug in a detected prescription is different from the detected amount of drug to be delivered, generate an alert via a user interface.

7. The portable device according to claim 2, wherein the detected amount of drug to be delivered corresponds to a notation associated with at least one of a dose input on an injection pen and a level indicator adjacent to the liquid level in a syringe barrel in the captured image.

8. The portable device according to claim 1, wherein the program instructions include an algorithm selected from a two-dimensional image processing algorithm and a three-dimensional image processing algorithm for the processor to analyze the captured image.

9. The portable device according to claim 1, wherein the processor is configured to: Analyze at least one captured image to detect characteristics of a medical product, the characteristics including at least one specified attribute of the medical product selected from the group consisting of: a selected color of the medical product, a selected size of the medical product, a selected shape factor of the medical product, the presence of a safety mechanism on the medical product, the absence of a safety mechanism on the medical product compared to a stored image of the medical product having a safety mechanism, and Analyze the detected characteristics using previously stored medical product data that can be accessed locally or remotely by a processor to determine whether a medical error has occurred, the previously stored medical product data including specified specifications of image characteristics of a medical product corresponding to at least one specified attribute.

10. The portable device according to claim 9, wherein at least one of the medical products is a liquid drug aspirated into a syringe, and at least one specified attribute of the liquid drug is selected from the group consisting of: the opacity of the liquid drug, the presence of air bubbles in the liquid drug, the presence of particulate matter in the liquid drug; and wherein the previously stored medical product data includes specified specifications of image features for the at least one specified attribute of the liquid drug.

11. The portable device according to claim 1, wherein the portable device is a mobile phone or a computing device having a wireless communication interface, the memory is configured to store an integrated disease management IDM application, the IDM application including an IDM personal application operated by a user as a patient and / or an IDM professional application operated by a healthcare professional, and the processor is further adapted to execute instructions according to the IDM application to operate the portable device in a cloud configuration with a remote IDM system, whereby the IDM application transmits data to and receives data from the IDM system during an application session.

12. The portable device according to claim 11, wherein the portable device operates according to the IDM personal application to transmit information from a captured image to the IDM system and store the information from the captured image at the IDM system, the information being selected from the group consisting of: the dose amount determined from at least one of the captured images, the date and / or timestamp of a medical event determined from at least one of the captured images, and the medical product identified from at least one of the captured images.

13. The portable device according to claim 12, wherein the portable device operates according to the IDM professional application to determine patient information from the information stored in the IDM system, the patient information including compliance data for a prescription regimen based on information related to the dose amount and the date and / or time of a medical event, medical product prescription update data based on information related to the medical products identified from the captured images and the date and / or time of the medical events corresponding to the use of these medical products, and / or billing data corresponding to the medical products identified from the captured images and the date and / or time of the medical events corresponding to the use of these medical products.

14. The portable device according to claim 13, wherein the portable device is capable of wirelessly connecting to at least one other medical condition management device and obtaining medical event information therefrom, and the processor is further adapted to execute instructions according to the IDM application to transmit the medical event information to the IDM system.

15. The portable device according to claim 12, wherein the cloud configuration includes a private cloud and a public cloud, and the portable device operates according to an IDM application to determine whether at least one of the user-related information and other data stored in the memory is proprietary data or non-proprietary data, and selectively transmits the proprietary data via the private cloud and the non-proprietary data via the public cloud.

16. A portable device for capturing an image of a medical event, comprising: an imaging device for imaging at least one medical product used during a medical event; a memory storing the image captured by the imaging device and program instructions for processing the captured image; a user interface configured to generate an output to a user; and a processor adapted to execute the program instructions to: analyze the captured image associated with the medical event to detect a characteristic of the medical product, the characteristic being selected from the group consisting of a mark on the medical product and a designated attribute of the medical product; store data related to the detected characteristic in the memory; and generate an output to the user via the user interface using the data related to the detected characteristic; wherein the portable device is a monitor for a selected medical condition, and the detected characteristic is a monitoring parameter detected by the monitor and indicated via a user output interface associated with the monitor.

17. The portable device according to claim 16, wherein the program instructions for processing the captured image are adapted to execute the program instructions to record a date and / or time associated with the detected characteristic.

18. The portable device according to claim 16, wherein the monitor is selected from the group consisting of a pulse oximeter, a thermometer, a blood pressure monitor, and a blood glucose monitor.

Images