Method, system and computer program product for removing redundant content from pharmaceutical packaging and facilitating verification of content in said pharmaceutical packaging

An AI-assisted system removes label content from pharmaceutical packaging using convolutional neural networks to improve drug identification accuracy, addressing the obscuration issue and enhancing verification processes.

JP7595178B2Active Publication Date: 2024-12-05PARATA SYSTEMS LLC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023545301
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-01-29
Filing Date
2022-01-28
Publication Date
2024-12-05
Estimated Expiration
2042-01-28

AI Technical Summary

Technical Problem

Pharmaceutical packaging systems often have label content such as commercial marketing information, patient identification, and personal health information that obscures the verification of drug contents, making it difficult to accurately identify individual drugs using imaging techniques.

Method used

An AI-assisted system using convolutional neural networks and image processing techniques to remove label content from pharmaceutical packaging images, allowing for the detection and identification of individual drugs through bounding boxes and reference data matching.

Benefits of technology

Enhances the accuracy of drug verification processes by clearly distinguishing and identifying drugs within packaging, even in the presence of extraneous label information, facilitating efficient drug dispensing and inventory management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595178000001
    Figure 0007595178000001
  • Figure 0007595178000002
    Figure 0007595178000002
  • Figure 0007595178000003
    Figure 0007595178000003
Patent Text Reader

Abstract

The method includes receiving an image of a pharmaceutical package containing one or more pharmaceutical products, where the image includes label content displayed on a surface of the pharmaceutical package; detecting the label content on the surface of the pharmaceutical package using an artificial intelligence engine; and generating a modified image of the pharmaceutical package with the label content removed from the surface.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 143,400, filed January 29, 2021, the disclosure of which is incorporated herein by reference in its entirety. [Background technology]

[0002] The present disclosure relates generally to pharmaceutical packaging, and more particularly to methods, systems and computer program products for removing excess content from the surface of pharmaceutical packaging to facilitate verification of the contents of the pharmaceutical packaging.

[0003] Pharmaceutical packaging systems may be used in facilities, such as pharmacies, hospitals, long-term care facilities, etc., to dispense pharmaceuticals to fill prescriptions. These pharmaceutical packaging systems may include systems designed to package pharmaceuticals in a variety of container types, including, but not limited to, pouches, vials, bottles, blister cards, and strip packaging. Strip packaging is a type of packaging in which medications are packaged in individual pouches for administration on a specific date and, in some cases, at a specific time. Typically, the individual pouches are removably attached and are often provided in a roll. The pouches can be separated from the roll as needed.

[0004] Some types of pharmaceutical packages, e.g., pouches and blister cards, may have printed thereon, for example, personal health information (PHI), manufacturer information, such as logos, names, contact information, etc., and / or other details regarding the contents of the pharmaceutical package, such as the number of medications, the names of medications, the administration time, the dosing strength, bar codes, etc. Such content on the surface of the pharmaceutical package may make it more difficult to verify the contents of the pharmaceutical package through imaging of the pharmaceutical package. Summary of the Invention [Means for solving the problem]

[0005] In some embodiments of the present inventive concept, a method includes receiving an image of a pharmaceutical package containing one or more pharmaceutical products, wherein the image includes label content displayed on a surface of the pharmaceutical package; detecting the label content on the surface of the pharmaceutical package using an artificial intelligence engine; and generating a modified image of the pharmaceutical package with the label content removed from the surface.

[0006] In other embodiments, the label content includes commercial marketing information, patient identifying information, or personal healthcare information.

[0007] In yet other embodiments, the commercial marketing information includes a logo or business name; the patient identification information includes a patient name, a patient telephone number, a patient address, or a patient identification number; and the personal healthcare information includes the names of the one or more medications, the number of each of the one or more medications, a predetermined administration time for each of the one or more medications, or one or more barcodes associated with the one or more medications, a prescription order, a patient account, or an identification number.

[0008] In yet another embodiment, the method further includes, in response to receiving the image of the pharmaceutical package, performing gamma correction on the image of the pharmaceutical package to generate a gamma corrected image of the pharmaceutical package; performing Gaussian blur denoising on the gamma corrected image of the pharmaceutical package to generate a noise reduced image of the pharmaceutical package; and performing automatic image thresholding on the noise reduced image of the pharmaceutical package to generate a foreground / background separated image of the pharmaceutical package. Detecting the label content using the artificial intelligence engine includes detecting the label content on the surface of the foreground / background separated image of the pharmaceutical package using the artificial intelligence engine.

[0009] In yet another embodiment, the artificial intelligence engine is a convolutional neural network.

[0010] In yet another embodiment, the convolutional neural network comprises a plurality of convolutional layers, wherein at least some of the plurality of convolutional layers are connected to each other via skip connections.

[0011] In yet another embodiment, the artificial intelligence engine is a first artificial intelligence engine and the modified image is a first modified image, the method further including receiving order information for the one or more pharmaceuticals and an identifier for the pharmaceutical package; detecting individual pharmaceuticals of the one or more pharmaceuticals in the first modified image using a second artificial intelligence engine; and generating a second modified image of the pharmaceutical package including indicia that distinguish the individual pharmaceuticals of the one or more pharmaceuticals and associate the one or more pharmaceuticals with the order information and the identifier for the pharmaceutical package.

[0012] In yet other embodiments, the indicia distinguishing the individual pharmaceutical agents of the one or more pharmaceutical agents comprises one or more bounding boxes.

[0013] In yet another embodiment, the order information includes names for the one or more medications in the medication package. The method further includes identifying, using a third artificial intelligence engine, at least a portion of the one or more medications in the second modified image based on the names for the one or more medications. The names are associated with medication attributes in a reference database.

[0014] In yet other embodiments, the at least a portion of the one or more pharmaceutical agents comprises a fragmented version of the one or more pharmaceutical agents.

[0015] In yet other embodiments, the method further comprises identifying a portion of the one or more pharmaceutical products as debris resulting from damage to the one or more pharmaceutical products using a third artificial intelligence engine.

[0016] In yet another embodiment, the method further comprises annotating the at least a portion of the one or more pharmaceutical agents in the second modified image with the names of the one or more pharmaceutical agents.

[0017] In yet other embodiments, the method further comprises annotating an unknown label of a pharmaceutical product to any of the one or more pharmaceutical products whose names have not been annotated.

[0018] In yet another embodiment, the name is a first name and the order information includes National Drug Codes (NDCs) for the one or more pharmaceutical products in the pharmaceutical product package, and the method further includes matching NDCs not associated with at least some of the one or more pharmaceutical products annotated with the first name against pharmaceutical product reference data; and annotating any of the one or more pharmaceutical products not annotated with a name that has an associated NDC that matches the pharmaceutical product reference data with a second name based on pharmaceutical product reference data.

[0019] In yet other embodiments, the pharmaceutical reference data comprises pharmaceutical shape, pharmaceutical color, pharmaceutical etching, pharmaceutical marking, pharmaceutical weight, and / or pharmaceutical label.

[0020] In some embodiments of the inventive concept, a system includes a processor; and a memory coupled to the processor, wherein the memory includes computer-readable program code embodied therein that is executable by the processor to execute instructions, the instructions including receiving an image of a pharmaceutical package including one or more pharmaceutical products, the image including label content displayed on a surface of the pharmaceutical package; detecting the label content on the surface of the pharmaceutical package using an artificial intelligence engine; and generating a modified image of the pharmaceutical package with the label content removed from the surface.

[0021] In a further embodiment, the instructions further include, in response to receiving the image of the pharmaceutical package, performing gamma correction on the image of the pharmaceutical package to generate a gamma corrected image of the pharmaceutical package; performing Gaussian blur denoising on the gamma corrected image of the pharmaceutical package to generate a noise reduced image of the pharmaceutical package; and performing automatic image thresholding on the noise reduced image of the pharmaceutical package to generate a foreground / background separated image of the pharmaceutical package. Detecting the label content using the artificial intelligence engine includes detecting the label content on the surface of the foreground / background separated image of the pharmaceutical package using the artificial intelligence engine.

[0022] In still further embodiments, the artificial intelligence engine is a convolutional neural network.

[0023] In some embodiments of the inventive concept, a computer program product comprises a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including computer-readable program code embodied therein that is executable by the processor to execute instructions, the instructions including receiving an image of a pharmaceutical package containing one or more pharmaceutical products, the image including label content displayed on a surface of the pharmaceutical package; detecting the label content on the surface of the pharmaceutical package using an artificial intelligence engine; and generating a modified image of the pharmaceutical package with the label content removed from the surface.

[0024] In another embodiment, the instructions further include, in response to receiving the image of the pharmaceutical package, performing gamma correction on the image of the pharmaceutical package to generate a gamma corrected image of the pharmaceutical package; performing Gaussian blur denoising on the gamma corrected image of the pharmaceutical package to generate a noise reduced image of the pharmaceutical package; and performing automatic image thresholding on the noise reduced image of the pharmaceutical package to generate a foreground / background separated image of the pharmaceutical package. Detecting the label content using the artificial intelligence engine includes detecting the label content on the surface of the foreground / background separated image of the pharmaceutical package using the artificial intelligence engine.

[0025] Other methods, systems, articles of manufacture, and / or computer program products according to embodiments of the inventive concepts will be or become apparent to one with skill in the art upon examination of the accompanying drawings and the following detailed description. All such additional systems, methods, articles of manufacture, and / or computer program products are intended to be included within this specification, be within the scope of the inventive subject matter, and be protected by the accompanying claims.

[0026] Other features of the embodiments will be more readily understood from the following detailed description of specific embodiments thereof when read in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a block diagram illustrating a communication network including an artificial intelligence (AI)-assisted drug package analysis system in accordance with some embodiments of the inventive concepts. [Figure 2] FIG. 2 is a block diagram of the AI-assisted drug package analysis system of FIG. 1, in accordance with some embodiments of the inventive concepts. [Figure 3] FIG. 3 is a block diagram of a convolutional neural network for detecting label content on the surface of a pharmaceutical package, in accordance with some embodiments of the inventive concepts. [Figure 4] FIG. 4 is a block diagram of a skip connection arrangement between convolutional layers of the convolutional neural network of FIG. 3 in accordance with some embodiments of the inventive concepts. [Figure 5] FIG. 5 is a flow chart diagram illustrating operations for performing pharmaceutical package analysis in accordance with some embodiments of the inventive concepts. [Figure 6] FIG. 6 is a block diagram illustrating pre-processing of pharmaceutical package images in accordance with some embodiments of the inventive concepts. [Figure 7] FIG. 7 is a flow chart diagram illustrating further operations for performing pharmaceutical package analysis in accordance with some embodiments of the inventive concepts. [Figure 8] FIG. 8 is a flow chart diagram illustrating further operations for performing pharmaceutical package analysis in accordance with some embodiments of the inventive concepts. [Figure 9] FIG. 9 is a flow chart diagram illustrating further operations for performing pharmaceutical package analysis in accordance with some embodiments of the inventive concepts. [Figure 10]FIG. 10 is a flow chart diagram illustrating further operations for performing pharmaceutical package analysis in accordance with some embodiments of the inventive concepts. [Figure 11] FIG. 11 is a data processing system that may be used to implement one or more servers in the AI-assisted drug package analysis system of FIG. 1, according to some embodiments of the inventive concepts. [Figure 12] FIG. 12 is a block diagram illustrating a software / hardware architecture for use in the AI-assisted drug package analysis system of FIG. 1, in accordance with some embodiments of the inventive concepts. [Figure 13] FIG. 13 is a diagram illustrating indicia for distinguishing individual medications in a medication package image, in accordance with some embodiments of the inventive concepts. [Figure 14] FIG. 14 is a diagram illustrating indicia for distinguishing individual medications in a medication package image, in accordance with some embodiments of the inventive concepts. DETAILED DESCRIPTION OF THE INVENTION

[0028] In the following detailed description of the invention, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the inventive concepts. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In some instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the inventive concepts. All embodiments disclosed herein are intended to be practiced separately or combined in any manner and / or combination. Aspects described with respect to one embodiment may be incorporated in a different embodiment, even if not specifically described therein. That is, all embodiments and / or features of any embodiment may be combined in any manner and / or combination.

[0029] As used herein, the term "data processing facility" encompasses, but is not limited to, hardware elements, firmware components, and / or software components. A data processing system may be configured with one or more data processing facilities.

[0030] As used herein, the term "pharmaceutical packaging system" refers to any type of pharmaceutical formulation dispensing system, including, but not limited to, automated systems that fill pharmaceuticals into vials, bottles, containers, pouches, blister cards, etc.; semi-automated systems that fill pharmaceuticals into vials, bottles, containers, pouches, blister cards, etc.; and any combination of automated and semi-automated systems that fill pharmaceutical packages. Pharmaceutical packaging systems also include packaging systems for pharmaceutical formulation substitutes, such as dietary supplements and / or biopharmaceuticals.

[0031] As used herein, the terms "pharmaceutical" and "medication" are used interchangeably and refer to a medicament prescribed for a patient, either human or animal. A pharmaceutical or medicament may be embodied in various ways, including but not limited to, a pill, a capsule, a tablet, etc.

[0032] The term "drug product" refers to any type of pharmaceutical product that can be packaged in vials, bottles, containers, pouches, blister cards, etc. by automated and semi-automated pharmaceutical packaging systems, including, but not limited to, pills, capsules, tablets, caplets, gelcaps, lozenges, etc. Drug products also refer to pharmaceutical formulation substitutes, such as dietary supplements and / or biological products. Examples of pharmaceutical packaging systems, such as those described above, including management techniques for filling packaging orders, are described in U.S. Pat. No. 10,492,987, the disclosure of which is incorporated herein by reference.

[0033] The term "pharmaceutical package" refers to any type of object capable of holding a pharmaceutical product, such as a pharmaceutical product as described above, including vials, bottles, containers, pouches, blister cards, and the like.

[0034] Embodiments of the inventive concepts are described herein in the context of a pharmaceutical packaging analysis engine comprising one or more machine learning engines and artificial intelligence (AI) engines. It will be understood that embodiments of the inventive concepts are not limited to a particular implementation of the pharmaceutical packaging analysis engine, and that various types of AI systems may be used, including, but not limited to, multilayer neural networks, deep learning systems, natural language processing systems, and / or computer vision systems. Furthermore, it will be understood that the multilayer neural networks are multilayer artificial neural networks comprising artificial neurons or nodes, and do not encompass biological neural networks comprising actual biological neurons. Embodiments of the inventive concepts may be implemented using multiple AI systems, or by combining various functions into fewer or a single AI system.

[0035] Some embodiments of the inventive concept stem from the recognition that when verifying the contents of a pharmaceutical package, such as a pouch or blister card, image analysis of the pharmaceutical package can reveal label content on the surface of the pharmaceutical package, obscuring the pharmaceutical product contained therein. Embodiments of the inventive concept provide an AI-assisted pharmaceutical package analysis system that can detect label content on the surface of the pharmaceutical package using an AI engine and generate a modified image of the pharmaceutical package with the label content removed. The label content can include, for example, commercial marketing information, patient identification information, and personal healthcare information (PHI). With the label content removed from the surface of the pharmaceutical package, an AI system is used to detect one or more individual pharmaceutical products contained in the package, and a second modified image is generated that includes indicia, e.g., bounding boxes, that distinguish the individual pharmaceutical products contained in the pharmaceutical package. In some embodiments, pharmaceutical products that are fragmented or even broken can be distinguished by the indicia. An AI system can then be used to identify one or more of the medications contained in the pharmaceutical package by a specific pharmaceutical name. In some embodiments, whole medications, fragmented medications, and / or debris can be identified by name using the AI ​​system based on attributes of the medication, such as shape, color, one or more etchings, one or more markings, weight, and / or one or more labels, along with knowledge of the contents of the pharmaceutical package and successful identification of other medications contained in the pharmaceutical package. Unidentifiable medications can be further analyzed by matching the National Drug Code (NDC) or Drug Identification Number (DIN) of the medication contained in the pharmaceutical package with pharmaceutical reference data, which can include information on the medication's shape, color, one or more etchings, one or more markings, weight, and / or one or more labels.As used herein, NDC can be used to represent both NDC and DIN information. If a match is found, the medications can be annotated with a name based on the medication reference data and the NDC. Embodiments of the inventive concepts can also be used for filling and verifying unit-of-use packages (i.e., each package containing a single dose of a single medication). Unit-of-use packages can be patient-specific, but can also be manufactured without patient information, e.g., for one-time or emergency use, and used as floor stock in hospitals or long-term care facilities. Each of these pouches (or blisters on a card) can have a subset or slightly different label information than a typical medication package. For example, a single-use package can have the medication name, dosage, NDC, manufacturer, lot number, expiration date, and / or beyond-use-date (BUD). The package can also include pharmacy information or information about the facility where the unit-of-use package is provided.

[0036] Referring to FIG. 1, a communication network 100 including an AI-assisted drug package analysis system according to some embodiments of the inventive concepts is shown including a pharmacy management system (PMS) or host system 110, a packaging system server 120, one or more package analysis engine servers 155, and one or more drug packaging systems 130a and 130b connected via a network 140.

[0037] The PMS system 110 may be configured to manage and fill prescriptions for customers. As used herein, a PMS system may be used in a pharmacy or generally as a batch production system for other applications, such as dispensing nutritional supplements or biologics. The PMS system 110 may be associated with various types of facilities, such as pharmacies, hospitals, long-term care facilities, etc. The PMS system, or host system 110, may be any system capable of transmitting valid prescriptions to one or more product packaging systems 130a and 130b. The packaging system server 120 may include a packaging system interface module 135 and may be configured to manage the operation of the pharmaceutical packaging systems 130a and 130b. For example, the packaging system server 120 may be configured to receive packaging orders from the PMS system 110 and to identify which of the pharmaceutical packaging systems 130a and 130b should be used to package a particular individual order or batch of orders. Additionally, packaging system server 120 may be configured to manage the operation of pharmaceutical packaging systems 130a and 130b. For example, packaging system server 120 may be configured to manage an inventory of pharmaceuticals available through each of pharmaceutical packaging systems 130a and 130b, manage pharmaceutical dispensing canisters allocated or registered to one or more of pharmaceutical packaging systems 130a and 130b, generally manage the operational status of pharmaceutical packaging systems 130a and 130b, and / or manage reports regarding the status (e.g., allocated, completed, etc.) of packaging orders, pharmaceutical inventory, order requisitions, etc. Users 150, such as pharmacists or pharmacy technicians, may communicate with packaging system server 120 using any suitable computing device via wired and / or wireless connections.While FIG. 1 shows user 150 communicating with packaging system server 120 via a direct connection, it will be understood that user 150 may communicate with packaging system server 120 via one or more network connections. User 150 may interact with packaging system server 120 to approve or override various recommendations made by packaging system server 120 when operating pharmaceutical packaging systems 130a and 130b. User 150 may also initiate the running of various reports, such as those described above for pharmaceutical packaging systems 130a and 130b. While only two pharmaceutical packaging systems 130a and 130b are shown in FIG. 1, it will be understood that three or more pharmaceutical packaging systems may be managed by packaging system server 120.

[0038] The AI-assisted pharmaceutical package analysis system may include one or more package analysis engine servers 155, which may include one or more package analysis engine modules 160 for removing excess content from an image of a pharmaceutical package to facilitate verification of the contents of the pharmaceutical package. The one or more package analysis engine servers 155 and the one or more package analysis engine modules 160 may represent one or more AI systems that may be configured to generate the modified image of the pharmaceutical package with label content removed from one or more surfaces, to detect within the pharmaceutical package image one or more individual pharmaceuticals contained within the pharmaceutical package, and / or to identify those pharmaceuticals detected within the pharmaceutical package image. In accordance with various embodiments of the inventive concepts, the label content may be removed from any surface on the pharmaceutical package, including multiple surfaces of the pharmaceutical package, such as the top, bottom, and sides of a vial, the front and back surfaces of a pouch and blister pack, etc.

[0039] It will be understood that the division of functionality described herein between packaging system server 120 / packaging system interface module 135 and one or more package analysis engine servers 155 / one or more package analysis engine modules 160 is by way of example. In accordance with different embodiments of the inventive concept, various functionality and capabilities can be transferred between packaging system server 120 / packaging system interface module 135 and one or more package analysis engine servers 155 / one or more package analysis engine modules 160. Moreover, in some embodiments, packaging system server 120 / packaging system interface module 135 and one or more package analysis engine servers 155 / one or more package analysis engine modules 160 may be integrated as a single logical and / or physical entity.

[0040] Network 140 connects pharmaceutical packaging systems 130a and 130b, PMS system 110, and packaging system server 120 to one another. Network 140 may be a global network, such as the Internet or other publicly accessible network. The various elements of network 140 may be interconnected by wide area networks, local area networks, intranets, and / or other private networks, which may not be accessible by the general public. Thus, communication network 140 may represent a combination of public and private networks, or a virtual private network (VPN). Network 140 may be a wireless network, a wired network, or a combination of both wireless and wired networks. In some embodiments, one or more package analysis engine servers 155 may also be connected to network 140.

[0041] In some embodiments, the AI-assisted drug package analysis service provided through one or more package analysis engines 155 and one or more package analysis engine modules 160 may be implemented as a cloud service. In some embodiments, the AI-assisted drug package analysis service may be implemented as a Representational State Transfer Web Service (RESTful Web Service).

[0042] While FIG. 1 illustrates an exemplary communications network including an AI-assisted pharmaceutical package analysis system, it will be understood that embodiments of the present subject matter are not limited to such configurations and are intended to encompass any configuration capable of performing the operations described herein.

[0043] As described above, the one or more package analysis engine servers 155 and the one or more package analysis engine modules 160 may represent one or more AI systems that may be configured to generate a modified image of the pharmaceutical package with label content removed from the surface, to detect within the pharmaceutical package image one or more individual pharmaceuticals contained within the pharmaceutical package, and / or to identify those pharmaceuticals detected within the pharmaceutical package image. Figure 2 is a block diagram of one or more package analysis engine modules 160 for implementing an AI system, e.g., a machine learning system, that can be used to detect within the pharmaceutical package image one or more individual pharmaceuticals contained within the pharmaceutical package and / or to identify those pharmaceuticals detected within the pharmaceutical package image. The AI ​​system of Figure 2 may be implemented as a single AI system for detecting within the pharmaceutical package image one or more individual pharmaceuticals contained within the pharmaceutical package and to identify those pharmaceuticals detected within the pharmaceutical package image. In other embodiments, the architecture of the AI ​​system of Figure 2 may be replicated to form separate AI systems for detecting within the pharmaceutical package image one or more individual pharmaceuticals contained within the pharmaceutical package and for identifying each of these detected pharmaceuticals within the pharmaceutical package image. As shown in Figure 2, one or more package analysis engine modules 160 may include both a training module and modules used to process new data for detecting and / or identifying pharmaceuticals within the pharmaceutical package image. The modules used in the training portion of one or more package analysis engine modules 160 include a training data module 205, a characterization module 225, a labeling module 230, and a machine learning engine 240.

[0044] The training data 205 includes one or more images of a pharmaceutical package, each containing one or more pharmaceuticals therein. The one or more pharmaceutical packages may include label content on their surface, which may include, but is not limited to, commercial marketing information, patient identifying information, and / or personal healthcare information (PHI). The commercial marketing information may include, for example, a logo and / or a business name. The patient identifying information may include, for example, a patient name, a patient telephone number, a patient address, and / or a patient identification number. The personal healthcare information may include, for example, the names of one or more pharmaceuticals contained in the pharmaceutical package, the administration time of each of the one or more pharmaceuticals, one or more barcodes associated with the one or more pharmaceuticals, a prescription order, a patient account, an identification number, and / or other information. In some embodiments, the pharmaceutical package image may be modified to remove at least a portion of the label content contained on its surface through the use of an AI system, e.g., a neural network, as described below with reference to FIG. 3. In some embodiments, training data 205 may further include ordering information for one or more pharmaceuticals contained in the pharmaceutical package and / or an identifier for the pharmaceutical package, to detect individual pharmaceuticals of one or more pharmaceuticals contained in the pharmaceutical package in a modified pharmaceutical package image from which at least a portion of the label content on the surface of the pharmaceutical package has been removed. In further embodiments, the ordering information included in training data 205 may include the names of the one or more pharmaceuticals in the pharmaceutical package to identify the pharmaceuticals detected in the pharmaceutical package image. Characterization module 225 is configured to identify individual independent variables, which may be considered one or more dependent variables, used by one or more package analysis engine modules 160 to detect and / or identify one or more pharmaceuticals, e.g., to detect and / or identify one or more pharmaceuticals in a pharmaceutical package image from which label content has been removed.For example, the training data 205 may generally be raw or formatted and may include extra information in addition to the drug information and / or drug packaging information. For example, the training data 205 may include account codes, business address information, etc., which may be filtered by the characterization module 225. Features extracted from the training data 205 may be referred to as attributes, and the number of features may be referred to as dimensions. The labeling module 230 may be configured to assign defined labels to the training data and the detected and / or identified drugs to ensure consistent naming conventions for both the input features and the generated output. The machine learning engine 240 may process both the characterized training data 205, e.g., the training data 205 described above including the labels provided by the labeling module 230, and may be configured to test multiple functions to establish quantitative relationships between the characterized and labeled input data and the generated output. The machine learning engine 240 may use modeling techniques to evaluate the impact of various input data features on the generated output. These effects may then be used to adjust and refine the quantitative relationships between the characterized and labeled input data and the generated outputs. The adjusted and refined quantitative relationships between the characterized and labeled input data generated by machine learning engine 240 are output for use in AI engine 245. Machine learning engine 240 may be referred to as a machine learning algorithm.

[0045] The modules used to detect individual pharmaceuticals within a pharmaceutical package image of one or more pharmaceuticals contained within the pharmaceutical package and / or identify those pharmaceuticals detected within the pharmaceutical package image include a new data module 255, a characterization module 265, an AI engine module 245, and a pharmaceutical package processing and analysis module 275. The new data 255 may be the same data / information in content and format as the training data 205, except that the new data 255 is used for analysis of new pharmaceutical packages rather than for training purposes. Similarly, the characterization module 265 performs the same functions on the new data 255 as the characterization module 225 performs on the training data 205. The AI ​​engine 245 may be generated by the machine learning engine 240 substantially in the form of determined quantitative relationships between characterized and labeled input data and output pharmaceutical package content analysis. In some embodiments, the AI ​​engine 245 may be referred to as an AI model. The AI ​​engine 245 may be configured to generate a modified image of a pharmaceutical package that includes indicia distinguishing between the one or more pharmaceuticals contained therein while associating the one or more pharmaceuticals with order information and / or an identifier for the pharmaceutical package. The indicia may be embodied in a variety of ways, including but not limited to, bounding boxes, i.e., polygons; circles; enclosed shapes, e.g., enclosed shapes that include straight and curved surfaces; enclosed shapes that include only curved surfaces; and / or lines or symbols that define boundaries between one or more pharmaceuticals. In some embodiments, the indicia may approximate the shape of the pharmaceuticals. The AI ​​engine 245 may also be configured to identify one or more pharmaceuticals based on the name of the pharmaceutical.The AI ​​engine 245 may use various modeling techniques, including, but not limited to, regression techniques, neural network techniques, autoregressive integrated moving average (ARIMA) techniques, deep learning techniques, linear discriminant analysis techniques, decision tree techniques, naive Bayes techniques, K-nearest neighbor techniques, learning vector quantization techniques, support vector machine techniques, and / or bagging / random forest techniques, to detect individual medications of one or more medications contained in the medication package within the medication package image and to identify those medications detected within the medication package image according to different embodiments of the inventive concepts.

[0046] The medication package processing and analysis module 275 may be configured to output a modified medication package image having one or more medications identified by display of an indicia, e.g., a bounding box, along with the name of the one or more medications, to the medication package verification system.

[0047] As described above, the one or more package analysis engine servers 155 and the one or more package analysis engine modules 160 may represent one or more AI systems that may be configured to generate modified images of pharmaceutical packages having label content removed from their surfaces. FIG. 3 is a block diagram of the one or more package analysis engine modules 160 for implementing a neural network-based AI system that may be used to generate the modified images of pharmaceutical packages having label content removed from their surfaces. In the exemplary embodiment of FIG. 3, the neural network is a convolutional neural network. However, it will be understood that an AI system for removing label content from pharmaceutical package images may be embodied as a fully connected neural network in accordance with other embodiments of the inventive concept. However, convolutional neural networks may be useful when processing or classifying images due to the large number of pixels and, consequently, the large number of weights to manage in the neural network layers. A convolutional neural network may reduce the main image matrix through convolution to a matrix with lower dimensions in the first layer, thereby reducing the number of weights used and the impact on training time.

[0048] Referring now to FIG. 3, an image preprocessor 305 may receive one or more images of a pharmaceutical package including label content displayed thereon. As described below with reference to FIG. 6, the image preprocessor may perform various corrections to the image data, including, for example, gamma correction, noise reduction, and / or image segmentation. The preprocessed pharmaceutical package image may be an image represented by a matrix of dimensions AxBx3 (where the number 3 represents red, green, and blue colors) and may then be provided to a convolutional neural network 310. As shown in FIG. 3, the convolutional neural network 310 includes a first convolutional layer 320 and a second convolutional layer 330, along with a first pooling layer 325 and a second pooling layer 335. Each of the first convolutional layer 320 and the second convolutional layer 330 may be a matrix of smaller dimensions than an input matrix and may be configured to perform a convolution operation with a portion of the input matrix having the same dimensions. The sum of the products of corresponding elements is the output of the convolutional layer. The output of each of the convolutional layers may also be processed through a rectified linear unit operation, in which any number less than zero is converted to zero and any positive number is left unchanged. The convolutional neural network 310 further includes a first pooling layer 325 and a second pooling layer 335. The pooling layers 325 and 335 may be configured to filter the outputs of the first convolutional layer 320 and the second convolutional layer 330, respectively, by performing a downsampling operation. The size of the pooling operation or filter is smaller than the size of the input feature map, and in some embodiments, is 2x2 pixels applied with a stride of 2 pixels. This means that the pooling layer always reduces the size of each feature map by a factor of 2, e.g., reduces each dimension by half and reduces the number of pixels or values ​​in each feature map by a factor of 4. For example, a pooling layer applied to a 6x6 (36 pixel) feature map will result in a pooled output of a 3x3 (9 pixel) feature map.The final output layer is a conventional fully connected neural network layer 340, which provides output as a modified pharmaceutical package image 345 with at least a portion of the label content on its surface removed.

[0049] In some implementations of the inventive concept, the convolutional neural network 310 may be a residual neural network in which a skip connection is used between the first convolutional layer 320 and the second convolutional layer 330. An example of such a skip connection is shown in Figure 4. Specifically, in a skip connection, the convolutional neural network includes a convolutional layer that receives as input both the output of the previous convolutional layer and the input to the previous convolutional layer.

[0050] Although two convolutional layers 320 and 330 are shown in the example convolutional neural network 310 of FIG. 3 for purposes of illustration, it will be understood that convolutional neural networks according to various embodiments of the inventive concepts may comprise many convolutional layers, and in some embodiments may exceed 100 layers.

[0051] 3 and 7-10 are flow chart diagrams illustrating operations for performing pharmaceutical package analysis, including removing label content therefrom to facilitate verification of the content therein, in accordance with some embodiments of the inventive concepts. Referring now to FIG. 5, operations begin at block 500, where a convolutional neural network 310 receives an image of a pharmaceutical package having label content displayed on its surface. The label content may include, but is not limited to, commercial marketing information, patient identifying information, and / or personal healthcare information (PHI). The commercial marketing information may include, for example, a logo and / or a business name. The patient identifying information may include, for example, a patient name, a patient telephone number, a patient address, and / or a patient identification number. The personal healthcare information may include, for example, the names of one or more medications contained in the pharmaceutical package, predetermined administration times for each of the one or more medications, one or more barcodes associated with the one or more medications, a prescription order, a patient account, an identification number, and / or other information. At block 505, label content on the surface of the pharmaceutical package may be detected using the convolutional neural network 310. The convolutional neural network 310 may then generate a modified image of the pharmaceutical package with the label content removed from its surface in block 510.

[0052] As described above, the pharmaceutical package image may undergo pre-processing to perform various corrections on the image data. Referring now to Figures 6 and 7, operations begin at block 700, where the gamma correction module 605 performs gamma correction on the pharmaceutical package image to generate a gamma-corrected image. While one or more cameras may darken the image, the gamma correction may brighten the image, allowing the convolutional neural network 310 to better recognize the edges of various elements displayed in the image. The gamma correction may be implemented as a power law transform, except for low luminance, which may be linear to avoid infinite derivatives at zero luminance. This is the classic nonlinearity applied to encode SDR images. The exponent, or "gamma," may have a value of 0.45, but due to the linear portion of the curve below, the final gamma correction function may be closer to a power-low exponent of 0.5, i.e., a square root transformation; therefore, the gamma correction may conform to the DeVries-Rose law of brightness perception. In block 705, a Gaussian blur denoising module or filter 610 is used to perform Gaussian blur denoising on the gamma-corrected image to produce a noise-reduced image. The Gaussian blur denoising module or filter 610 may be a linear filter. It may be used to blur the image and / or reduce noise. Two Gaussian blur denoising filters or filters 610 may be used so that their outputs are subtracted for "unsharp masking" (edge ​​detection). The Gaussian blur denoising module or filter 610 may blur edges and reduce contrast. A median filter is a non-linear filter that can be used as a method to reduce noise in an image. In block 710, the automatic image thresholding module 615 may perform automatic image thresholding on the noise-reduced image to generate an image with separated foreground and background. Thresholding is a technique used in image segmentation applications.The thresholding involves selecting a desired gray-level threshold to separate an object of interest in an image from the background based on the gray-level distribution. Otsu's method is a type of global thresholding that relies only on the gray values ​​of the image. Otsu's method is a global threshold selection method that involves calculating a gray-level histogram. When applied in only one dimension, the image may not be adequately segmented. A two-dimensional Otsu method can be used that is based on both the gray-level threshold of each pixel and spatial correlation information with the surrounding neighborhood. As a result, when applied to noisy images, Otsu's method can provide satisfactory segmentation. The output image from the preprocessing module of FIG. 6 can be applied to a pharmaceutical package correction engine, such as the convolutional neural network 310 of FIG. 3.

[0053] Referring now to FIG. 8 , at block 800, the pharmaceutical package image may be further processed to facilitate verification of the contents therein by detecting individual pharmaceuticals of one or more pharmaceuticals in the modified image with label content removed from its surface. The detection may be performed using an AI engine, such as AI engine 245 described above with respect to FIG. 2 , based on the modified pharmaceutical package image with label content removed from its surface, along with order information for one or more pharmaceuticals contained in the pharmaceutical package and / or an identifier for the pharmaceutical package. At block 805, the AI ​​engine may generate a second modified image of the pharmaceutical package, the second modified image including indicia that distinguish individual pharmaceuticals of the pharmaceuticals and associate the pharmaceuticals with the order information and identifier for the pharmaceutical package. In some embodiments, a bounding box may be used as an indicia that distinguishes individual pharmaceuticals of the pharmaceuticals, as shown, for example, in FIG. 13 , which illustrates the use of a circle as an indicia to identify the location of a particular pharmaceutical.

[0054] Referring now to FIG. 9 , at block 900, the pharmaceutical package image may be further processed to facilitate verification of its contents by identifying at least a portion of one or more pharmaceuticals in the second modified image having the detected pharmaceuticals based on the names of the one or more pharmaceuticals. The identification may be performed using an AI engine, such as AI engine 245 described above with respect to FIG. 2 , based on the modified pharmaceutical package image including the detected pharmaceuticals and order information (including the names of the one or more pharmaceuticals in the pharmaceutical package). In accordance with some embodiments of the inventive concepts, the name may be associated with pharmaceutical attributes in a reference database. These attributes include, but are not limited to, the pharmaceutical's shape, color, one or more etchings, one or more imprints, weight, and / or one or more labels. FIG. 14 shows the pharmaceutical of FIG. 13 annotated with the name "Drug A." In some embodiments, the identified pharmaceutical may include fragmented pharmaceuticals, such as portions of pills or tablets. The identified one or more medications may further include identification of debris resulting from damage to the medication, such as powdering all or part of the medication. Identifying the one or more medications by name in the medication package image, along with identifying any portions of the medication and packaging debris, may facilitate generating a count of the medications in the medication package for use in verifying the contents of the medication package. While the medication package image may be annotated with the determined names of the one or more medications, there may be situations in which the AI ​​engine is unable to determine the names of one or more medications in the medication package image. If only one or a few medications could not be named, these medications may be annotated with a new or previously unseen temporary name or National Drug Code (NDC). In some embodiments, further operations may be performed to determine the names of these medications, such as those described below with reference to FIG. 10.

[0055] 10, operations for determining names of medications not identified by name using the AI ​​system operations of FIG. 9 begin at block 1000, where a matching operation is performed between National Drug Codes (NDCs) included in the order information but not associated with any named medications and medication reference data, e.g., the medication's shape, color, one or more etchings, and / or one or more label information. Next, at block 1005, a determination may be made whether any of the unnamed medications in the medication package image match the shape, color, one or more etchings, one or more imprints, weight, and / or one or more label information associated with an NDC included in the order. If there is a match, the unnamed medication may be assigned a name corresponding to the associated NDC.

[0056] 11, a data processing system 1100 that may be used to implement one or more pharmaceutical package image analysis engine servers 155 of FIG. 1 according to some embodiments of the inventive concepts includes one or more input devices 1102, such as a keyboard or keypad, a barcode scanner or RFID reader, a display 1104, and a memory 1106 in communication with a processor 1108. The data processing system 1100 may further include a storage system 1110, a speaker 1112, and one or more input / output (I / O) data ports 1114 that also communicate with the processor 1108. The processor 1108 may be, for example, a commercially available or custom microprocessor. The storage system 1110 may include removable and / or fixed media, such as a floppy disk, ZIP drive, or hard disk, as well as virtual storage, such as a RAMDISK. One or more I / O data ports 1114 may be used to transfer information between data processing system 1100 and other computer systems or networks (e.g., the Internet). These components may be conventional components, e.g., components used in many conventional computing devices, and their functions for conventional operation are generally known to those of ordinary skill in the art. Memory 1106 may be configured with computer-readable program code 1116 for facilitating AI-assisted removal of excess label content from the surface of a pharmaceutical package and / or for verifying the contents of the pharmaceutical package and detecting and identifying one or more pharmaceuticals contained therein, in accordance with some embodiments of the inventive concepts.

[0057] FIG. 12 illustrates memory 1205 that may be used in embodiments of a data processing system, such as one or more pharmaceutical package analysis engine servers 155 of FIG. 1 and data processing system 1100 of FIG. 11, respectively, to facilitate AI-assisted removal of excess label content from the surface of the pharmaceutical package and / or detection and identification of one or more pharmaceutical products contained therein to verify the contents of the pharmaceutical package, in accordance with some embodiments of the inventive concepts. Memory 1205 is representative of one or more memory devices containing software and data used to facilitate operation of one or more pharmaceutical package analysis engine servers 155 and one or more pharmaceutical package analysis engine modules 160, as described herein. Memory 1205 may include, but is not limited to, the following types of devices: cache, ROM, PROM, EPROM, EEPROM, flash, SRAM, and DRAM. As shown in FIG. 12, memory 1205 may include five or more categories of software and / or data: operating system 1210, one or more pharmaceutical package processing and analysis engine modules 1225, and communications module 1240. In particular, the operating system 1210 may manage software and / or hardware resources of the data processing system and coordinate the execution of programs by the processor. The one or more pharmaceutical package processing and analysis engine modules 1125 may include a machine learning engine module 1230 and an AI engine module 1235. The machine learning engine module 1230 may be configured to perform one or more of the operations described above with respect to the machine learning engine 240, the convolutional neural network 310, and the flowchart diagrams of Figures 5 and 7-10. The AI ​​engine module 1225 may be configured to perform one or more of the operations described above with respect to the AI ​​engine 245, the convolutional neural network 310, and the flowchart diagrams of Figures 5 and 7-10. The communications module 1240 may be configured to support communications between, for example, the one or more pharmaceutical package analysis engine servers 155 and, for example, a pharmaceutical package verification system.

[0058] While Figures 11-12 illustrate hardware / software architectures that may be used in data processing systems, such as one or more pharmaceutical package analysis engine servers 155 of Figure 1 and data processing system 1100 of Figure 11, respectively, in accordance with some embodiments of the inventive concepts, it will be understood that embodiments of the present invention are not limited to such configurations and are intended to encompass any configuration capable of performing the operations described herein.

[0059] Computer program code for carrying out operations of the data processing systems described above with respect to Figures 1-11, 13, and 14 may, for development convenience, be written in a high-level programming language, such as Python, Java, C, and / or C++. In addition, computer program code for carrying out operations of the present invention may also be written in other programming languages, such as, but not limited to, interpreted languages. Some modules or routines may also be written in assembly language or microcode to enhance performance and / or memory usage. It will be further understood that the functionality of any or all of the program modules may also be implemented using discrete hardware components, one or more application-specific integrated circuits (ASICs), or programmed digital signal processors or microcontrollers.

[0060] Moreover, the functionality of one or more medication package analysis engine servers 155 of Figure 1 and data processing system 1100 of Figure 11 may each be implemented as a single processor system, a multi-processor system, a multi-core processor system, or a network of standalone computer systems in accordance with various embodiments of the inventive concepts. Each of these processors / computer systems may be referred to as a "processor" or a "data processing system."

[0061] The data processing devices described herein with respect to Figures 1-13 may be used to facilitate AI-assisted removal of excess label content from the surface of pharmaceutical packages and / or to verify the contents of pharmaceutical packages to facilitate detection and identification of one or more pharmaceuticals contained therein, in accordance with some embodiments of the inventive concepts described herein. These devices are operable to receive, transmit, process, and store data using any suitable combination of software, firmware, and / or hardware, and may be standalone or embodied as one or more enterprise, application, personal, broadband, and / or embedded computer systems and / or devices that may be interconnected by public and / or private, real and / or virtual, wired and / or wireless networks, including all or part of the global communications network known as the Internet, and may include various types of tangible, non-transitory computer-readable media. In particular, memory 1205, when coupled to a processor, includes computer-readable program code that, when executed by the processor, causes the processor to perform operations, including one or more of the operations described herein with respect to Figures 1-10, 13, and 14.

[0062] As described above, embodiments of the inventive concepts may provide an AI-assisted pharmaceutical package analysis system that may use a convolutional neural network to detect label content on the surface of a pharmaceutical package and generate a modified image of the pharmaceutical package with the label content removed, and one or more machine learning engines to detect and identify the pharmaceutical product contained in the pharmaceutical package, which may improve the accuracy of the package verification process, for example, before a pharmacy or medical center releases the packaged pharmaceutical product to a customer or patient.

[0063] Further definitions and embodiments:

[0064] In the foregoing description of various embodiments of the present disclosure, aspects of the present disclosure may be illustrated and described herein in any of numerous patentable classes or contexts, encompassing any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof. Accordingly, aspects of the present disclosure may be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of software and hardware implementations, all of which may be generally referred to herein as "circuits," "modules," "components," or "systems." Moreover, aspects of the present disclosure may take the form of a computer program product including one or more computer-readable media having computer-readable program code embodied thereon.

[0065] Any combination of one or more computer-readable media may be used. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, i.e., flash memory), a suitable optical fiber with repeaters, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this specification, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0066] A computer-readable signal medium may include a propagated data signal having computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium, but may be any computer-readable medium capable of communicating, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable signal medium may be transmitted using any suitable medium, for example, wireless, wired, fiber optic cable, RF, etc., or any suitable combination of those described above.

[0067] Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc.; conventional procedural programming languages ​​such as the "C" programming language, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP; dynamic programming languages ​​such as Python, Ruby, Groovy; or other programming languages. The computer program code may execute entirely on a user's computer as a standalone software package, partially on a user's computer as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including, for example, a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider) or in a cloud computing environment, or may be provided as a service, for example, Software as a Service (SaaS).

[0068] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, and the instructions, executed by the processor of the computer or other programmable instruction execution device, can create a machine such that it generates mechanisms for implementing the functions / acts specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0069] These computer program instructions may also be stored in a computer-readable medium and, when executed, direct a computer, other programmable data processing device, or other device to function in a particular manner to create an article of manufacture comprising instructions that cause the computer to perform one or more specified functions / acts in the flowchart illustrations and / or block diagrams. The computer program instructions may also be loaded into a computer, other programmable instruction execution device, or other device to cause a series of operational steps to be executed on the computer, other programmable device, or other device to produce a computer-implemented process, thus providing a process for implementing the function(s) / act(s) specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0070] The flowchart diagrams and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products or computer programs according to various embodiments of the present invention. In this regard, each block in the flowchart diagrams or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative implementations, the functions depicted in the block diagrams may occur out of the order depicted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially simultaneously, or the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a special-purpose hardware-based system that performs the specified functions or acts, or a combination of special-purpose hardware and computer instructions.

[0071] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. As used herein, the words "comprises," "comprising," "include," "including," "includes," "have," "has," "having," or variations thereof, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the word "and / or" includes any and all combinations of one or more of the associated listed items. Like reference numbers refer to like elements throughout the description of the figures.

[0072] It will also be understood that although terms such as first, second, etc. may be used herein to describe various elements, these elements are not intended to be limited by these terms; these terms are merely used to distinguish one element from another.

[0073] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, e.g., those defined in commonly used dictionaries, should be interpreted to have a meaning consistent with their meaning in the context of this application and the related art, and should not be interpreted in an idealized or overly formal sense unless expressly defined as such herein. Well-known functions or structures may not be described in detail for the sake of brevity and / or clarity.

[0074] The description of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the disclosure in the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present disclosure. The aspects of the present disclosure herein have been chosen and described to best explain the principles and practical applications of the present disclosure and to enable others skilled in the art to understand the disclosure with various modifications suited to the particular uses contemplated.

Claims

1. receiving an image of a pharmaceutical package containing one or more pharmaceutical products, the image including label content displayed on a surface of the pharmaceutical package; detecting the label content on the surface of the pharmaceutical package using an artificial intelligence engine; and generating a modified image of the pharmaceutical package with the label content removed from the surface; A method comprising:

2. The method of claim 1 , wherein the label content comprises commercial marketing information, patient identification information, or personal health care information.

3. said commercial marketing information includes a logo or business name; The patient identification information includes a patient name, a patient telephone number, a patient address, or a patient identification number; and 3. The method of claim 2, wherein the personal healthcare information includes names of the one or more medications, numbers for each of the one or more medications, predetermined administration times for each of the one or more medications, or one or more bar codes associated with the one or more medications, prescription orders, patient account or identification numbers.

4. responsive to receiving the image of the pharmaceutical package, performing gamma correction on the image of the pharmaceutical package to generate a gamma corrected image of the pharmaceutical package; performing a Gaussian blur denoising on the gamma corrected image of the pharmaceutical package to generate a noise reduced image of the pharmaceutical package; and performing an automated image thresholding process on the noise reduced image of the pharmaceutical package to generate a foreground and background separated image of the pharmaceutical package; Further comprising: wherein detecting the label content using the artificial intelligence engine comprises: Detecting the label content on the surface of the foreground and background separated image of the pharmaceutical package using the artificial intelligence engine. Including, The method of claim 1.

5. The method of claim 4 , wherein the artificial intelligence engine is a convolutional neural network.

6. 6. The method of claim 5, wherein the convolutional neural network comprises a plurality of convolutional layers, and wherein at least some of the plurality of convolutional layers are connected to each other via skip connections.

7. the artificial intelligence engine is a first artificial intelligence engine, and the modified image is a first modified image; The method further comprising: receiving order information for the one or more pharmaceutical products and an identifier for the pharmaceutical package; detecting individual ones of the one or more pharmaceutical agents in the first modified image using a second artificial intelligence engine; and generating a second modified image of the pharmaceutical package including indicia that distinguishes the individual pharmaceuticals of the one or more pharmaceuticals and associates the one or more pharmaceuticals with the order information and the identifier for the pharmaceutical package. The method of claim 1 further comprising:

8. 8. The method of claim 7, wherein the indicia distinguishing the individual ones of the one or more pharmaceutical agents include one or more bounding boxes.

9. the order information includes a name for the one or more medications in the medication package; The method further comprising: Identifying at least a portion of the one or more pharmaceutical products in the second modified image based on the names of the one or more pharmaceutical products using a third artificial intelligence engine. Further comprising: wherein the name is associated with drug attributes in a reference database. The method of claim 7.

10. 10. The method of claim 9, wherein the at least a portion of the one or more pharmaceutical agents comprises a fragmented version of the one or more pharmaceutical agents.

11. The method further comprising: identifying, using a third artificial intelligence engine, a portion of the one or more pharmaceutical products as debris resulting from damage to the one or more pharmaceutical products. The method of claim 9 further comprising:

12. annotating the at least a portion of the one or more pharmaceutical agents in the second modified image with the names of the one or more pharmaceutical agents. The method of claim 9 further comprising:

13. annotating a provisional name to any of said one or more pharmaceutical products whose names have not been annotated; The method of claim 12 further comprising:

14. the name is a first name, and the order information includes a National Drug Code (NDC) for the one or more medications in the medication package; The method further comprising: Matching NDCs that are not associated with at least some of the one or more medicines annotated with the first name against medicine reference data; and annotating any of one or more pharmaceutical products not annotated with a name having an associated NDC that matches said pharmaceutical reference data with a second name based on said pharmaceutical reference data. The method of claim 12 further comprising:

15. The method of claim 14 , wherein the drug reference data comprises drug shape, drug color, drug etching, drug imprint, drug weight, and / or drug label.

16. 1. A system comprising: A processor; and A memory coupled to the processor wherein the memory includes computer readable program code embodied in the memory executable by the processor to execute instructions; The instruction: receiving an image of a pharmaceutical package containing one or more pharmaceutical products, the image including label content displayed on a surface of the pharmaceutical package; detecting the label content on the surface of the pharmaceutical package using an artificial intelligence engine; and generating a modified image of the pharmaceutical package with the label content removed from the surface; Including, The system.

17. The instruction: responsive to receiving the image of the pharmaceutical package, performing gamma correction on the image of the pharmaceutical package to generate a gamma corrected image of the pharmaceutical package; performing a Gaussian blur denoising on the gamma corrected image of the pharmaceutical package to generate a noise reduced image of the pharmaceutical package; and performing an automated image thresholding process on the noise reduced image of the pharmaceutical package to generate a foreground and background separated image of the pharmaceutical package; Further comprising: wherein detecting the label content using the artificial intelligence engine comprises: Detecting the label content on the surface of the foreground and background separated image of the pharmaceutical package using the artificial intelligence engine. Including, 17. The system of claim 16.

18. 20. The system of claim 17, wherein the artificial intelligence engine is a convolutional neural network.

19. 1. A computer program product, comprising: Non-transitory computer-readable storage medium wherein the non-transitory computer-readable storage medium includes computer-readable program code embodied in the non-transitory computer-readable storage medium executable by a processor to execute instructions; The instruction: receiving an image of a pharmaceutical package containing one or more pharmaceutical products, the image including label content displayed on a surface of the pharmaceutical package; detecting the label content on the surface of the pharmaceutical package using an artificial intelligence engine; and generating a modified image of the pharmaceutical package with the label content removed from the surface; Including, The computer program product.

20. The instruction: responsive to receiving the image of the pharmaceutical package, performing gamma correction on the image of the pharmaceutical package to generate a gamma corrected image of the pharmaceutical package; performing a Gaussian blur denoising on the gamma corrected image of the pharmaceutical package to generate a noise reduced image of the pharmaceutical package; and performing an automated image thresholding process on the noise reduced image of the pharmaceutical package to generate a foreground and background separated image of the pharmaceutical package; Further comprising: wherein detecting the label content using the artificial intelligence engine comprises: Detecting the label content on the surface of the foreground and background separated image of the pharmaceutical package using the artificial intelligence engine. Including, 20. A computer program product as claimed in claim 19.

Citation Information

Patent Citations

  • Method for inspecting ic appearance

    JP2003097926A