Methods, systems, and computer program products for removing extraneous content from a package of pharmaceutical products to facilitate verification of the contents therein

Abstract

A method includes receiving an image of a drug package containing one or more drug products therein, the image including label content displayed on a surface of the drug package; detecting, using an artificial intelligence engine, the label content on the surface of the drug package; and generating a modified image of the drug package from which the label content is removed from the surface thereof.

Description

[0001] Related applications

[0002] This application claims priority and benefit to U.S. Provisional Application No. 63 / 143,400, filed January 29, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates generally to pharmaceutical packaging, and more specifically to methods, systems, and computer program products for removing irrelevant content from the surface of pharmaceutical packaging in order to verify the contents of the pharmaceutical packaging. Background Technology

[0004] In facilities such as pharmacies, hospitals, and long-term care facilities, pharmaceutical packaging systems can be used to dispense medications for prescription dispensing. These systems can include those designed to package medications in various container types, including but not limited to sachets, vials, bottles, blister packs, and strip packs. Strip packs are a type of packaging where medications are packaged in individual sachets for administration on a specific date and, in some cases, at a specific time. Typically, the individual sachets are detachably linked together and are often supplied in rolls. The sachets can be separated from the roll when needed.

[0005] Certain types of pharmaceutical packaging (e.g., sachets and blister packs) may contain printed information such as Personal Health Information (PHI), manufacturer information (e.g., logo, name, contact information, etc.), and / or other detailed information about the contents of the pharmaceutical packaging (e.g., quantity, name, administration time, dosage, barcode, etc.). Such content on the surface of pharmaceutical packaging can make it more difficult to verify the contents of the pharmaceutical packaging through imaging. Summary of the Invention

[0006] In some embodiments of the present invention, a method includes: receiving an image of a pharmaceutical package containing one or more pharmaceutical products, the image including label content displayed on the surface of the pharmaceutical package; detecting the label content on the surface of the pharmaceutical package using an artificial intelligence engine; and generating a modified image of the pharmaceutical package with the label content removed from its surface.

[0007] In other embodiments, the label content includes commercial marketing information, patient identification information, or personal healthcare information.

[0008] In other embodiments, commercial marketing information includes a logo or company name; patient identification information includes a patient name, patient telephone number, patient address, or patient identification number; and personal healthcare information includes the name of one or more medicines, the quantity of each of the one or more medicines, the prescribed dosing time for each of the one or more medicines, or one or more barcodes, prescriptions, patient accounts, identification numbers, or other information associated with one or more medicines.

[0009] In other embodiments, the method further includes: in response to receiving an image of the drug packaging, performing gamma correction on the image of the drug packaging to generate a gamma-corrected image of the drug packaging; performing Gaussian blur denoising on the gamma-corrected image of the drug packaging to generate a denoised image of the drug packaging; and performing automatic image thresholding processing on the denoised image of the drug packaging to generate a foreground-background separated image of the drug packaging. Detecting label content using an artificial intelligence engine includes: using an artificial intelligence engine to detect label content on the surface of the foreground-background separated image of the drug packaging.

[0010] In other embodiments, the artificial intelligence engine is a convolutional neural network.

[0011] In other embodiments, the convolutional neural network includes multiple convolutional layers, at least some of which are interconnected by skip connections.

[0012] In other embodiments, the artificial intelligence engine is a first artificial intelligence engine, and the modified image is a first modified image. The method further includes: receiving order information for one or more pharmaceutical products and an identifier for pharmaceutical packaging; using a second artificial intelligence engine to detect individual drugs among the one or more pharmaceutical products in the first modified image; and generating a second modified image of the pharmaceutical packaging, the second modified image including markers that distinguish individual drugs among the one or more pharmaceutical products and associate the one or more drugs with the order information and the identifier for pharmaceutical packaging.

[0013] In other embodiments, the markers that distinguish individual drugs among one or more drugs include one or more bounding boxes.

[0014] In other embodiments, the order information includes the names of one or more drugs in the drug packaging. The method further includes using a third artificial intelligence engine to identify at least some of the drugs in a second modified image based on their names. The names are associated with drug attributes in a reference database.

[0015] In other embodiments, at least some of the drugs in one or more medicines include the broken drug in one or more medicines.

[0016] In other embodiments, the method further includes using a third artificial intelligence engine to identify a portion of one or more pharmaceutical products as fragments resulting from damage to one or more pharmaceutical products.

[0017] In other embodiments, the method further includes annotating at least some of the one or more drugs in a second modified image with the names of one or more drugs.

[0018] In other embodiments, the method further includes: labeling any drug among one or more drugs that is not labeled with a name using an unknown drug label.

[0019] In other embodiments, the name is a first name, and the order information includes the National Drug Code (NDC) of one or more drugs in the drug packaging. The method further includes: matching NDCs not associated with at least some of the drugs already annotated with the first name with drug reference data; and, based on the drug reference data, annotating any unnamed drug among the one or more drugs that has an associated NDC matching the drug reference data with a second name.

[0020] In other embodiments, the drug reference data includes drug shape, drug color, drug etching, drug markings, drug weight, and / or drug label.

[0021] In some embodiments of the present invention, a system includes: a processor; and a memory coupled to the processor and including computer-readable program code embodied in the memory, the computer-readable program code being executable by the processor to perform operations including: receiving an image of a pharmaceutical package containing one or more pharmaceutical products, the image including label content displayed on the surface of the pharmaceutical package; detecting the label content on the surface of the pharmaceutical package using an artificial intelligence engine; and generating a modified image of the pharmaceutical package with the label content removed from its surface.

[0022] In a further embodiment, the operation further includes: in response to receiving an image of the drug packaging, performing gamma correction on the image of the drug packaging to generate a gamma-corrected image of the drug packaging; performing Gaussian blur denoising on the gamma-corrected image of the drug packaging to generate a denoised image of the drug packaging; and performing automatic image thresholding processing on the denoised image of the drug packaging to generate a foreground-background separated image of the drug packaging. Detecting label content using an artificial intelligence engine includes: using an artificial intelligence engine to detect label content on the surface of the foreground-background separated image of the drug packaging.

[0023] In a further embodiment, the artificial intelligence engine is a convolutional neural network.

[0024] In some embodiments of the present invention, a computer program product includes a non-transitory computer-readable storage medium including computer-readable program code embodied therein, the computer-readable program code being executable by a processor to perform operations including: receiving an image of a pharmaceutical package containing one or more pharmaceutical products, the image including label content displayed on the surface of the pharmaceutical package; detecting the label content on the surface of the pharmaceutical package using an artificial intelligence engine; and generating a modified image of the pharmaceutical package with the label content removed from its surface.

[0025] In other embodiments, the operation further includes: in response to receiving an image of the drug packaging, performing gamma correction on the image of the drug packaging to generate a gamma-corrected image of the drug packaging; performing Gaussian blur denoising on the gamma-corrected image of the drug packaging to generate a denoised image of the drug packaging; and performing automatic image thresholding processing on the denoised image of the drug packaging to generate a foreground-background separated image of the drug packaging. Detecting label content using an artificial intelligence engine includes: using an artificial intelligence engine to detect label content on the surface of the foreground-background separated image of the drug packaging.

[0026] Other methods, systems, articles of manufacture, and / or computer program products based on embodiments of the present invention will become apparent to those skilled in the art upon review of the following drawings and detailed description. All such additional systems, methods, articles of manufacture, and / or computer program products are intended to be included within this specification, within the scope of the subject matter of the invention, and protected by the appended claims. Attached Figure Description

[0027] Other features of the embodiments will be more readily understood when read in conjunction with the accompanying drawings, based on the following detailed description of specific embodiments, in which:

[0028] Figure 1 This is a block diagram illustrating a communication network for an artificial intelligence (AI) assisted pharmaceutical packaging analysis system, according to some embodiments of the present invention.

[0029] Figure 2 These are some embodiments of the concept of the present invention. Figure 1 A block diagram of an AI-assisted drug packaging analysis system;

[0030] Figure 3 This is a block diagram of a convolutional neural network for detecting label content on the surface of pharmaceutical packaging, according to some embodiments of the present invention.

[0031] Figure 4 These are some embodiments of the concept of the present invention. Figure 3 A block diagram showing the arrangement of skip connections between convolutional layers in a convolutional neural network;

[0032] Figure 5 This is a flowchart illustrating some embodiments of the present invention for performing pharmaceutical packaging analysis;

[0033] Figure 6 This is a block diagram illustrating some embodiments of pharmaceutical packaging image preprocessing according to the present invention;

[0034] Figures 7 to 10 This is a flowchart illustrating further operations for performing pharmaceutical packaging analysis according to some embodiments of the present invention;

[0035] Figure 11 Some embodiments of the present invention can be used in Figure 1 The AI-assisted drug packaging analysis system implements a data processing system on one or more servers;

[0036] Figure 12 This illustrates some embodiments of the concept according to the present invention for use in Figure 1 A block diagram of the software / hardware architecture of the AI-assisted drug packaging analysis system; and

[0037] Figure 13 and Figure 14 This is a diagram illustrating some embodiments of the concept according to the present invention, showing markings for distinguishing individual medicines in a pharmaceutical packaging image. Detailed Implementation

[0038] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the inventive concept. However, those skilled in the art will understand that the invention can be practiced without these specific details. In some cases, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the inventive concept. It is intended that all embodiments disclosed herein can be implemented individually or in any manner and / or in combination. Aspects described with respect to one embodiment may be incorporated into different embodiments, although no specific description is given with respect to these embodiments. That is, features of all embodiments and / or any embodiments may be combined in any manner and / or in any combination.

[0039] As used herein, the term "data processing apparatus" includes, but is not limited to, hardware components, firmware components, and / or software components. A data processing system may be configured with one or more data processing apparatuses.

[0040] As used herein, the term "pharmaceutical packaging system" refers to any type of drug dispensing system, including but not limited to: automated systems for filling vials, bottles, containers, pouches, blister packs, etc., with pharmaceuticals; semi-automated systems for filling vials, bottles, containers, pouches, blister packs, etc., with pharmaceuticals; and any combination of automated and semi-automated systems for filling pharmaceutical packaging with pharmaceuticals. Pharmaceutical packaging systems also include packaging systems for drug alternatives, such as nutritional supplements and / or biopharmaceuticals.

[0041] As used herein, the terms “drug” and “medicine” are used interchangeably and refer to a drug prescribed to a patient (human or animal). Drugs or medicines can be embodied in various forms, including but not limited to pills, capsules, tablets, etc.

[0042] The term "pharmaceutical" refers to any type of medicine that can be packaged in vials, bottles, containers, pouches, blister packs, etc., using automated and semi-automated packaging systems, including but not limited to pills, capsules, tablets, sachets, gels, lozenges, etc. Pharmaceuticals also refer to drug alternatives, such as nutritional supplements and / or biopharmaceuticals. An example pharmaceutical packaging system including management techniques for fulfilling packaging orders is described in U.S. Patent No. 10,492,987, the disclosure of which is incorporated herein by reference.

[0043] The term "pharmaceutical packaging" refers to any type of object that can contain pharmaceuticals, including but not limited to vials, bottles, containers, pouches, blister packs, etc.

[0044] Embodiments of the present invention are described herein in the context of a pharmaceutical packaging analysis engine that includes one or more machine learning engines and artificial intelligence (AI) engines. It will be understood that embodiments of the present invention are not limited to a specific implementation of a pharmaceutical analysis engine, and various types of AI systems can be used, including but not limited to multilayer neural networks, deep learning systems, natural language processing systems, and / or computer vision systems. Furthermore, it will be understood that multilayer neural networks are multilayer artificial neural networks that include artificial neurons or nodes, and do not include biological neural networks that include real biological neurons. Embodiments of the present invention can be implemented using multiple AI systems, or by combining various functions into fewer functions or a single AI system.

[0045] Some embodiments of this invention stem from the understanding that when verifying the contents of pharmaceutical packaging (e.g., sachets or blister packs), label content on the surface of the packaging may obscure the contained pharmaceuticals during image analysis. Embodiments of this invention provide an AI-assisted pharmaceutical packaging analysis system that can use an AI engine to detect label content on the surface of pharmaceutical packaging and generate a modified image of the packaging with the label content removed. Label content may include, for example, commercial marketing information, patient identification information, Personal Healthcare Information (PHI), etc. With the label content removed from the pharmaceutical packaging surface, the AI ​​system can detect one or more individual pharmaceuticals contained in the packaging and generate a second modified image including markers, such as bounding boxes, to distinguish the individual pharmaceuticals contained in the packaging. In some embodiments, broken or even damaged pharmaceuticals reduced to fragments can be distinguished by these markers. The AI ​​system can then be used to identify one or more pharmaceuticals contained in the packaging by the name of the specific pharmaceutical product. In some embodiments, an AI system can be used to identify whole medicines, broken medicines, and / or fragments by name, based on the properties of the medicine (e.g., shape, color, etching, marking, weight, and / or labeling) and knowledge of the contents of the medicine packaging, as well as the successful identification of other medicines contained in the medicine package. Unidentified medicines can be further analyzed by matching the National Drug Code (NDC) or Drug Identification Number (DIN) of the medicine contained in the medicine package with medicine reference data, which may include medicine shape, color, etching, marking, weight, and / or labeling information. As used herein, NDC can be used to represent both NDC and DIN information. Upon obtaining a match, these medicines can be annotated by name based on the medicine reference data and NDC. Embodiments of the inventive concept can also be used to fill and validate unit of use packages (i.e., each package containing a single dose of a single medicine). Unit of use packages can be patient-specific, but can also be generated without patient information, allowing them to be used as a basic stock in hospitals or long-term care facilities, for example, for single-use or emergency purposes. Each of these small packets (or blister packs on cards) may have label information that may be a subset of typical prescription drug packaging or slightly different from typical prescription drug packaging. For example, a unit pack may have the drug name, dosage, NDC, manufacturer, batch number, expiry date, and / or used-up date (BUD). The packaging may also include pharmacy information or information related to the facility that provides the unit pack.

[0046] refer to Figure 1According to some embodiments of the present invention, a communication network 100a including an AI-assisted pharmaceutical packaging analysis system includes a pharmacy management system (PMS) or host system 110, a packaging system server 120, a packaging analysis engine server 155, and one or more pharmaceutical packaging systems 130a and 130b coupled via a network 140, as shown in the figure.

[0047] PMS system 110 can be configured to manage prescriptions and dispense medications according to prescriptions for customers. As used herein, the PMS system can be used in pharmacies or, generally, as a batch production system for other applications, such as distributing nutritional supplements or biopharmaceuticals. PMS system 110 can be associated with various types of facilities, such as pharmacies, hospitals, long-term care facilities, etc. PMS system or host system 110 can be any system capable of sending valid prescriptions to one or more pharmaceutical packaging systems 130a and 130b. Packaging system server 120 may include packaging system interface module 135 and can be configured to manage the operation of pharmaceutical packaging systems 130a and 130b. For example, packaging system server 120 can be configured to receive packaging orders from PMS system 110 and identify which of pharmaceutical packaging systems 130a and 130b should be used to package a specific individual order or batch. Furthermore, packaging system server 120 can be configured to manage the operation of pharmaceutical packaging systems 130a and 130b. For example, packaging system server 120 can be configured to manage the inventory of each available medicine through pharmaceutical packaging systems 130a and 130b, manage medicine dispensing containers allocated to or registered to one or more pharmaceutical packaging systems 130a and 130b, manage the general operational status of pharmaceutical packaging systems 130a and 130b, and / or manage reporting on the status (e.g., allocation, completion, etc.) of packaging orders, medicine inventory, order invoices, etc. User 150 (e.g., pharmacist or pharmacy technician) can communicate with packaging system server 120 via wired and / or wireless connections using any suitable computing device. Although in Figure 1 The diagram illustrates user 150 communicating with packaging system server 120 via a direct connection, but it will be understood that user 150 may communicate with packaging system server 120 via one or more network connections. User 150 can interact with packaging system server 120 to approve or reject various suggestions made by packaging system server 120 in the operation of pharmaceutical packaging systems 130a and 130b. User 150 can also initiate the execution of various reports for pharmaceutical packaging systems 130a and 130b as described above. Although Figure 1 Only two pharmaceutical packaging systems, 130a and 130b, are shown in the diagram, but it will be understood that the packaging system server 120 can manage more than two pharmaceutical packaging systems.

[0048] An AI-assisted pharmaceutical packaging analysis system may include a packaging analysis engine server 155, which includes a packaging analysis engine module 160 to verify the contents of the pharmaceutical packaging by removing irrelevant content from an image of the packaging. The packaging analysis engine server 155 and the packaging analysis engine module 160 may represent one or more AI systems configured to generate modified images of the pharmaceutical packaging from one or more surfaces, detect individual drugs among one or more drugs contained in the packaging, and / or identify drugs already detected in the packaging image. According to various embodiments of the invention, label content can be removed from any surface of the pharmaceutical packaging, including multiple surfaces such as the top, bottom, and sides of a vial, the front and back of a pouch or blister pack, etc.

[0049] It will be understood that the functional division between the packaging system server 120 / packaging system interface module 135 and the packaging analysis engine server 155 / packaging analysis engine module 160 described herein is illustrative. In various embodiments of the invention, various functions and capabilities can be moved between the packaging system server 120 / packaging system interface module 135 and the packaging analysis engine server 155 / packaging analysis engine module 160. Furthermore, in some embodiments, the packaging system server 120 / packaging system interface module 135 and the packaging analysis engine server 155 / packaging analysis engine module 160 may be merged into a single logical and / or physical entity.

[0050] Network 140 couples together pharmaceutical packaging systems 130a and 130b, PMS system 110, and packaging system server 120. Network 140 may be a global network (e.g., the Internet) or other publicly accessible network. Various components of network 140 may be interconnected via wide area networks, local area networks, intranets, and / or other private networks that may not be accessible to the general public. Therefore, communication network 140 may represent a combination of public and private networks or virtual private networks (VPNs). Network 140 may be a wireless network, a wired network, or a combination of both. In some embodiments, packaging analysis engine server 155 may also be coupled to network 140.

[0051] In some embodiments, the AI-assisted drug packaging analysis service provided by the packaging analysis engine 155 and the packaging analysis engine module 160 can be implemented as a cloud service. In some embodiments, the AI-assisted drug packaging analysis service can be implemented as a Representational State Transfer Web Service (RESTful Web service).

[0052] although Figure 1 An example communication network including an AI-assisted pharmaceutical packaging analysis system is shown, but it will be understood that embodiments of the subject matter of this invention are not limited to such a configuration, but are intended to cover any configuration capable of performing the operations described herein.

[0053] As described above, the packaging analysis engine server 155 and the packaging analysis engine module 160 may represent one or more AI systems that can be configured to generate modified images of pharmaceutical packaging with label content removed from its surface, detect individual drugs among one or more drugs contained in the pharmaceutical packaging in the pharmaceutical packaging image, and / or identify these drugs that have already been detected in the pharmaceutical packaging image. Figure 2 This is a block diagram of a packaging analysis engine module 160 for implementing an AI system (e.g., a machine learning system) that can be used to detect individual drugs in one or more drugs contained in a drug packaging image, and / or identify such drugs that have already been detected in the drug packaging image. Figure 2 The AI ​​system can be implemented as a single AI system to detect individual drugs among one or more drugs contained in a drug packaging image, and to identify these drugs that have already been detected in the drug packaging image. In other embodiments, Figure 2 The architecture of the AI ​​system can be replicated to form separate AI systems for detecting individual drugs within one or more drugs contained in a drug packaging image, as well as identifying these drugs already detected in the drug packaging image. For example... Figure 2 As shown, the packaging analysis engine module 160 may include a training module and a module for processing new data to detect and / or identify drugs in drug packaging images. The modules used in the training portion of the packaging analysis engine module 160 include a training data module 205, a featureization module 225, a labeling module 230, and a machine learning engine 240.

[0054] Training data 205 may include one or more images of pharmaceutical packaging, each image containing one or more pharmaceutical products. The pharmaceutical packaging may include label content on its surface, which may include, but is not limited to, commercial marketing information, patient identification information, and / or Personal Healthcare Information (PHI). Commercial marketing information may include, for example, a logo and / or company name. Patient identification information may include, for example, a patient's name, patient's phone number, patient's address, and / or patient identification number. Personal Healthcare Information may include, for example, the name of one or more pharmaceutical products contained in the pharmaceutical packaging, the time of administration for each of the one or more pharmaceutical products, one or more barcodes associated with the one or more pharmaceutical products, prescriptions, patient accounts, identification numbers, and / or other information. In some embodiments, the pharmaceutical packaging images may be modified to allow for the use of AI systems (e.g., neural networks, referred to below). Figure 3(Description) At least a portion of the label content contained on its surface is removed. In some embodiments, in order to detect individual drugs among one or more drugs contained in a modified drug packaging image where at least a portion of the label content on its surface has been removed, training data 205 may also include order information for one or more drugs contained in the drug packaging and / or identifiers for the drug packaging. In a further embodiment, in order to identify these drugs already detected in the drug packaging image, the order information included in training data 205 may include the names of one or more drugs in the drug packaging. Characterization module 225 is configured to identify individual independent variables used by packaging analysis engine module 160 to detect and / or identify one or more drugs in, for example, a drug packaging image where label content has been removed, and these independent variables may be considered as dependent variables. For example, training data 205 may typically be unprocessed or formatted and includes additional information beyond drug and / or drug packaging information. For example, training data 205 may include account codes, business address information, etc., which may be filtered out by characterization module 225. Features extracted from training data 205 may be called attributes, and the number of features may be called dimensions. The labeling module 230 can be configured to assign defined labels to training data and detected and / or identified drugs to ensure consistent naming conventions between input features and generated outputs. The machine learning engine 240 can process the characteristicized training data 205 (including labels provided by the labeling module 230) and can be configured to test various functions to establish quantitative relationships between the characteristicized and labeled input data and the generated outputs. The machine learning engine 240 can use modeling techniques to evaluate the impact of various input data features on the generated outputs. These impacts can then be used to adjust and improve the quantitative relationships between the characteristicized and labeled input data and the generated outputs. The adjusted and improved quantitative relationships between the characteristicized and labeled input data generated by the machine learning engine 240 are output for use in the AI ​​engine 245. The machine learning engine 240 can be referred to as a machine learning algorithm.

[0055] The module for detecting individual drugs among one or more drugs contained in a drug packaging image and / or identifying such drugs already detected in the drug packaging image includes a new data module 255, a featureization module 265, an AI engine module 245, and a drug packaging processing and analysis module 275. New data 255 may be data / information identical in content and form to training data 205, except that new data 255 will be used to analyze new drug packaging, not for training purposes. Similarly, the function performed by featureization module 265 on new data 255 is the same as the function performed by featureization module 225 on training data 205. In practice, AI engine 245 may be generated by machine learning engine 240 in the form of a quantitative relationship determined between the featureized and labeled input data and the output drug packaging content analysis. In some embodiments, AI engine 245 may be referred to as an AI model. AI engine 245 may be configured to generate a modified image of the drug packaging that includes tags that distinguish individual drugs among one or more drugs contained therein and simultaneously associate one or more drugs with order information and / or identifiers specific to the drug packaging. The markers can be represented in various ways, including but not limited to bounding boxes or polygons, circles, closed shapes including straight and curved surfaces, closed shapes including only curved surfaces, and / or lines or symbols delineating boundaries between one or more pharmaceutical products. In some embodiments, the markers may take the shape approximating the shape of the pharmaceutical product. The AI ​​engine 245 can also be configured to identify one or more pharmaceutical products based on the names of one or more pharmaceutical products. In various embodiments of the present invention, the AI ​​engine 245 can use various modeling techniques to detect individual pharmaceutical products contained in one or more pharmaceutical products in a pharmaceutical packaging image, and to identify such pharmaceutical products already detected in the pharmaceutical packaging image, including but not limited to: regression techniques, neural network techniques, autoregressive integral moving average (ARIMA) techniques, deep learning techniques, linear discriminant analysis techniques, decision tree techniques, Naive Bayes techniques, K-nearest neighbor techniques, learned vector quantization techniques, support vector machine techniques, and / or bagging / random forest techniques.

[0056] The pharmaceutical packaging processing and analysis module 275 can be configured to output a modified pharmaceutical packaging image to a pharmaceutical packaging verification system, the modified pharmaceutical packaging image having one or more drugs identified by markers (e.g., bounding boxes) and the names of one or more drugs.

[0057] As described above, the packaging analysis engine server 155 and the packaging analysis engine module 160 may represent one or more AI systems that can be configured to generate modified images of pharmaceutical packaging with label content removed from its surface. Figure 3This is a block diagram of a packaging analysis engine module 160 used to implement an AI system via neural networks. This AI system can be used to generate modified images of pharmaceutical packaging with label content removed from its surface. Figure 3 In the example embodiment, the neural network is a convolutional neural network. However, it will be understood that, according to other embodiments of the invention, an AI system for removing label content from pharmaceutical packaging images can also be embodied as a fully connected neural network. However, convolutional neural networks can be useful when processing or classifying images because a large number of pixels and a large number of weights need to be managed across the neural network layers. A convolutional neural network can simplify the main image matrix into a lower-dimensional matrix in the first layer through convolution, which reduces the number of weights used and reduces the impact on training time.

[0058] For reference Figure 3 The image preprocessor 305 can receive one or more images of a pharmaceutical package, including label content displayed on its surface. (See below for reference.) Figure 6 As described, the image preprocessor can perform various corrections on the image data, including, for example, gamma correction, noise reduction, and / or image segmentation. The preprocessed pharmaceutical packaging image (which can be represented by an AxBx3 dimension matrix, where the number 3 represents the colors red, green, and blue) can then be fed into a convolutional neural network 310. Figure 3 As shown, the convolutional neural network 310 includes a first convolutional layer 320 and a second convolutional layer 330, as well as a first pooling layer 325 and a second pooling layer 335. Each of the convolutional layers 320 and 330 is a matrix with a dimension smaller than the input matrix and can be configured to perform a convolution operation on a portion of the input matrix having the same dimension. The sum of the products of corresponding elements is the output of the convolutional layer. The output of each convolutional layer can also be processed by a modified linear unit operation, where any number below 0 is converted to 0, and any positive numbers remain unchanged. The convolutional neural network 310 also includes a first pooling layer 325 and a second pooling layer 335. Pooling layers 325 and 335 can each be configured to filter the outputs of convolutional layers 320 and 330 respectively by performing a downsampling operation. The size of the pooling operation or filter is smaller than the size of the input feature map; in some embodiments, it is a 2×2 pixel array applied with a stride of 2 pixels. This means that the pooling layer will always reduce the size of each feature map by half, for example, by halving each dimension, thereby reducing the number of pixels or values ​​in each feature map to one-quarter of its size. For example, a pooling layer applied to a 6×6 (36 pixels) feature map will result in a 3×3 (9 pixels) pooled feature map as the output. The final output layer is a normal fully connected neural network layer 340, which outputs a modified pharmaceutical packaging image 345 with at least a portion of the labeled content on its surface removed.

[0059] In some embodiments of the present invention, the convolutional neural network 310 may be a residual neural network that uses skip connections between convolutional layers 320 and 330. Examples of skip connections are as follows: Figure 4 As shown. Specifically, in skip connections, a convolutional neural network involves a convolutional layer that receives both the output of the previous convolutional layer and the input to that previous convolutional layer as input.

[0060] It will be understood that, although for the purpose of explanation... Figure 3 The example convolutional neural network 310 shows two convolutional layers 320 and 330, but convolutional neural networks in various embodiments of the present invention may contain many convolutional layers, and in some embodiments may have more than 100 layers.

[0061] Figure 3 and Figures 7 to 10 This is a flowchart illustrating some embodiments of the invention, for performing pharmaceutical packaging analysis, including removing label content to facilitate verification of the contents. Reference is now made to... Figure 5 The operation begins at box 500, where convolutional neural network 310 receives an image of a pharmaceutical package displaying label content on its surface. Label content may include, but is not limited to, commercial marketing information, patient identification information, and / or personal healthcare information (PHI). Commercial marketing information may include, for example, a logo and / or company name. Patient identification information may include, for example, a patient's name, patient's phone number, patient's address, and / or patient identification number. Personal healthcare information may include, for example, the name of one or more medications contained in the pharmaceutical package, the prescribed dosing time for each of the one or more medications, one or more barcodes associated with the one or more medications, prescriptions, patient accounts, identification numbers, and / or other information. At box 505, convolutional neural network 310 can be used to detect the label content on the surface of the pharmaceutical package. Then, at box 510, convolutional neural network 310 can generate a modified image of the pharmaceutical package with the label content removed from its surface.

[0062] As mentioned above, pharmaceutical packaging images can be preprocessed to perform various corrections on the image data. (See also...) Figure 6 and Figure 7The operation begins at box 700, where gamma correction module 605 performs gamma correction on the pharmaceutical packaging image to generate a gamma-corrected image. One or more cameras may darken the image; gamma correction can brighten the image, allowing convolutional neural network 310 to better identify the edges of various elements displayed in the image. Gamma correction can be embodied as a power-law transform, and except at low luminance, gamma correction can be linear to avoid infinite derivatives at zero luminance. This is a conventional non-linearity used for encoding SDR images. The value of the exponent, or "gamma," can be 0.45, but the linear portion of the curve can make the final gamma correction function closer to a low power exponent of 0.5, i.e., a square root transform; therefore, gamma correction can conform to the luminance-perceived DeVries-Rose law. At box 705, Gaussian blur denoising module 610 performs Gaussian blur denoising on the gamma-corrected image to generate a denoised image. Gaussian blur denoising module or filter 610 can be a linear filter. It can be used to blur the image and / or reduce noise. Two Gaussian blur denoising filters 610 can be used, with the output subtracted for use in an "anti-sharpening mask" (edge ​​detection). The Gaussian blur denoising module or filter 610 can blur edges and reduce contrast. A median filter is a non-linear filter that can be used as a way to reduce noise in an image. At box 710, an automatic image thresholding module 615 can perform automatic image thresholding on the denoised image to generate a foreground-background separated image. Thresholding is a technique used in image segmentation applications. Thresholding involves selecting a desired grayscale threshold to separate the object of interest from the background based on the grayscale distribution of the object of interest in the image. Otsu's method is a global thresholding method that depends only on the image's grayscale values. Otsu's method is a global thresholding selection method that involves calculating a grayscale histogram. When applied in only one dimension, the image may not be sufficiently segmented. A two-dimensional Otsu's method can be used, which is based on the grayscale threshold of each pixel and its spatially relevant information in the neighborhood around the pixel. Therefore, Otsu's method can achieve satisfactory segmentation when applied to noisy images. Figure 6 The output image from the preprocessing module can be applied to a pharmaceutical packaging modification engine, for example. Figure 3 The convolutional neural network 310.

[0063] For reference Figure 8 The contents of a drug packaging image can be further processed by detecting individual drugs within one or more drugs in a modified image where label content has been removed from its surface, at box 800, to facilitate verification of the contents. This detection can be based on the modified drug packaging image with label content removed from its surface, along with order information for one or more drugs contained within the drug packaging and / or identifiers for the drug packaging, using an AI engine (e.g., as mentioned above regarding...). Figure 2 The AI ​​engine 245 described above is used for this purpose. At box 805, the AI ​​engine can generate a second modified image of the pharmaceutical packaging, which includes markers that distinguish individual drugs within the pharmaceutical package and associate the drugs with order information and identifiers specific to the pharmaceutical packaging. In some embodiments, bounding boxes can be used as markers to distinguish individual drugs within the pharmaceutical package, for example, as... Figure 13 As shown, Figure 13 The diagram illustrates how circles are used as markers to identify the location of specific medications.

[0064] For reference Figure 9 The drug packaging image can be further processed by identifying at least some of the drugs in a second modified image containing individual drugs detected in the drug packaging, based on the names of one or more drugs, at box 900, to facilitate verification of its contents. This identification can be performed using an AI engine (e.g., as mentioned above) based on the modified drug packaging image containing individual drugs detected in the drug packaging and order information (which includes the names of one or more drugs in the drug packaging). Figure 2 The AI ​​engine 245 described herein may be used. According to some embodiments of the invention, the name may be associated with pharmaceutical attributes in a reference database. These attributes may include, but are not limited to, pharmaceutical shape, color, etching, markings, weight, and / or labeling. Figure 14 It shows Figure 13 The identified drugs are those already labeled with the name "Drug A". In some embodiments, the identified one or more drugs may include fragments of the drug, such as a portion of a pill or tablet. The identified one or more drugs may also include identifying debris resulting from damage to the drug, such as damage that turns all or part of the drug into powder. Identifying one or more drugs in a drug packaging image by name, as well as identifying portions of the drug and packaging debris, can help generate a count of the drugs in the drug packaging for use in verifying the contents of the drug packaging. Drug packaging images can be labeled with names determined for one or more drugs, but there may be situations where the AI ​​engine cannot determine the names of one or more drugs in a drug packaging image. If only one or more drugs cannot be identified by name, these drugs can be labeled with a new or previously unseen temporary name or National Drug Code (NDC). In some embodiments, further steps can be performed to determine the names of these drugs, as referenced below. Figure 10 The described operation.

[0065] For reference Figure 10 , for use Figure 9The operation of the AI ​​system to determine the name of a drug not identified by name begins at box 1000, where a matching operation is performed between the National Drug Code (NDC) included in the order information but not associated with any named drug and drug reference data (e.g., drug shape, color, etching, and / or label information). Then, at box 1005, it can be determined whether any unnamed drug in the drug packaging image matches the shape, color, etching, marking, weight, and / or label information associated with the NDC included in the order. If a match exists, a name corresponding to the associated NDC can be assigned to the unnamed drug.

[0066] For reference Figure 11 According to some embodiments of the present invention, it can be used to implement Figure 1 The data processing system 1100 of the pharmaceutical packaging image analysis engine server 155 includes an input device 1102 (e.g., a keyboard or keypad, barcode scanner, or RFID reader), a display 1104, and a memory 1106 communicating with a processor 1108. The data processing system 1100 may also include a storage system 1110, a speaker 1112, and an input / output (I / O) data port 1114, also communicating with the processor 1108. The processor 1108 may be, for example, a commercially available or custom-made microprocessor. The storage system 1110 may include removable and / or fixed media (e.g., floppy disks, ZIP drives, hard disks, etc.) and virtual storage (e.g., RAMdisk). The I / O data port 1114 may be used to transfer information between the data processing system 1100 and another computer system or network (e.g., the Internet). These components may be conventional components, such as those used in many conventional computing devices, and their functionality is generally well known to those skilled in the art relative to routine operation. The memory 1106 may be configured with computer-readable program code 1116 to enable AI-assisted removal and / or detection and identification of one or more drugs contained therein from the surface of the pharmaceutical packaging to verify the contents of the pharmaceutical packaging, according to some embodiments of the present invention.

[0067] Figure 12 A memory 1205 is shown, which can be stored in a data processing system (e.g., Figure 1 Drug packaging analysis engine server 155 and Figure 11The data processing system 1100 is used in embodiments to facilitate the AI-assisted removal and / or detection and identification of one or more drugs contained therein from the surface of pharmaceutical packaging to verify the contents of the pharmaceutical packaging, according to some embodiments of the present invention. Memory 1205 represents one or more memory devices containing software and data that contribute to the operation of the pharmaceutical packaging analysis engine server 155 and the pharmaceutical packaging analysis engine module 160 described herein. Memory 1205 may include, but is not limited to, devices of the following types: cache, ROM, PROM, EPROM, EEPROM, flash memory, SRAM, and DRAM. Figure 12 As shown, memory 1205 may contain five or more classes of software and / or data: operating system 1210, drug packaging processing and analysis engine module 1225, and communication module 1240. Specifically, operating system 1210 can manage the software and / or hardware resources of the data processing system and can coordinate program execution performed by the processor. Drug packaging processing and analysis engine module 1212 may include machine learning engine module 1230 and AI engine module 1235. Machine learning engine module 1230 can be configured to execute the above-mentioned machine learning engine 240, convolutional neural network 310, and... Figure 5 and Figures 7 to 10 The flowchart describes one or more operations. AI engine module 1225 can be configured to perform the operations described above regarding AI engine 245, convolutional neural network 310, and... Figure 5 and Figures 7 to 10 The flowchart describes one or more operations. The communication module 1240 can be configured to support communication, for example, between a pharmaceutical packaging analysis engine server 155 and a pharmaceutical packaging validation system.

[0068] Although some embodiments of the present invention are based on the concept of the invention Figures 11 to 12 These are respectively shown in data processing systems (e.g., Figure 1 Drug packaging analysis engine server 155 and Figure 11 The hardware / software architecture used in the data processing system 1100) is described herein, but it will be understood that embodiments of the invention are not limited to such a configuration, but are intended to cover any configuration capable of performing the operations described herein.

[0069] For ease of development, it is used to execute the above-mentioned... Figures 1 to 11 , Figure 13 and Figure 14The computer program code for the operation of the data processing system discussed can be written in high-level programming languages ​​such as Python, Java, C, and / or C++. Furthermore, the computer program code for performing the operations of this invention can also be written in other programming languages, such as, but not limited to, interpreted languages. Some modules or routines can be written in assembly language or even microcode to improve performance and / or memory usage. It will be further appreciated that the functionality of any or all program modules can also be implemented using discrete hardware components, one or more application-specific integrated circuits (ASICs), or programmable digital signal processors or microcontrollers.

[0070] Furthermore, according to various embodiments conceived in this invention... Figure 1 Drug packaging analysis engine server 155 and Figure 11 The data processing system 1100 can be implemented as a single-processor system, a multi-processor system, a multi-core processor system, or even a network of independent computer systems. Each of these processor / computer systems can be referred to as a "processor" or a "data processing system".

[0071] Based on some embodiments of the inventive concept described herein, this document relates to... Figures 1 to 13 The described data processing apparatus can be used to facilitate AI-assisted removal and / or detection and identification of one or more pharmaceutical products contained within pharmaceutical packaging to verify the contents of the packaging. These apparatuses can be embodied as one or more enterprise, application, personal, general and / or embedded computer systems and / or devices (operable to receive, transmit, process, and store data using any suitable combination of software, firmware, and / or hardware, and can be standalone or interconnected via any public and / or private, real and / or virtual, wired and / or wireless networks (including all or part of a global communications network known as the Internet)) and can include various types of tangible, non-transitory computer-readable media. Specifically, memory 1205, when coupled to a processor, includes computer-readable program code that, when executed by the processor, causes the processor to perform actions including those described herein. Figures 1 to 10 , Figure 13 and Figure 14 Operations including one or more operations described.

[0072] As described above, embodiments of the present invention can provide an AI-assisted pharmaceutical packaging analysis system. This system can use AI technology (e.g., convolutional neural networks) to detect label content on the surface of pharmaceutical packaging to generate a modified image of the pharmaceutical packaging with the label content removed, and uses one or more machine learning engines to detect and identify the pharmaceutical product contained in the packaging. This can improve the accuracy of the packaging verification process before, for example, pharmacies or medical centers dispense packaged pharmaceutical products to customers or patients.

[0073] Other definitions and examples

[0074] In the above description of various embodiments of this disclosure, aspects of this disclosure may be described and illustrated herein in any of many patentable classes or contexts, including any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof. Therefore, aspects of this disclosure may be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of software and hardware, and may generally be referred to herein as “circuit,” “module,” “component,” or “system.” Furthermore, aspects of this disclosure may take the form of a computer program product comprising one or more computer-readable media having computer-readable program code embodied thereon.

[0075] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be (e.g., but not limited to) an electronic, magnetic, optical, electromagnetic, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (not an exhaustive list) of computer-readable storage media will include: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), suitable optical fiber with a repeater, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program used by or associated with an instruction execution system, apparatus, or device.

[0076] Computer-readable signal media may include propagated data signals embodying computer-readable program code therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and can transmit, propagate, or transfer a program used by or associated with an instruction execution system, apparatus, or device. The program code embodied on a computer-readable signal medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0077] Computer program code for performing operations relating to various aspects of this disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc.), conventional procedural programming languages ​​(e.g., the "C" programming language, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP), dynamic programming languages ​​(e.g., Python, Ruby, and Groovy), or other programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., via the Internet through an Internet service provider) or in a cloud computing environment, or provided as a service such as Software as a Service (SaaS).

[0078] Aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executable via a processor of a computer or other programmable instruction execution apparatus, create mechanisms for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0079] These computer program instructions may also be stored in a computer-readable medium that, when executed, can direct a computer, other programmable data processing apparatus, or other device to operate in a particular manner, such that the instructions, when stored in the computer-readable medium, produce an article of manufacture including the instructions, which, when executed, cause the computer to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram. The computer program instructions may also be loaded onto a computer, other programmable instruction execution apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide a process for implementing the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of this disclosure. In this regard, each box in a flowchart or block diagram may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the boxes may occur in a different order than indicated in the drawings. For example, two boxes shown consecutively may actually be executed substantially simultaneously, or these boxes may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each box in the block diagrams and / or flowcharts, and combinations of boxes in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware that performs the specified function or action, or a combination of dedicated hardware and computer instructions.

[0081] The terminology used herein is for descriptive purposes only and is not intended to limit the disclosure. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “include,” “including,” “have,” “has,” “having,” or variations thereof, when used in this specification, indicate the presence of a described feature, integer, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. The same reference numerals denote the same elements throughout the description of the accompanying drawings.

[0082] It will also be understood that although terms such as "first," "second," etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0083] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It will be further understood that terms (such as those defined in common dictionaries) should be interpreted as having the same meaning as they have in the context of the specification and related art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein. For the sake of brevity and / or clarity, well-known functions or constructions may not be described in detail.

[0084] The description in this disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the form of the disclosure. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the disclosure. Various aspects of the disclosure herein have been chosen and described to best explain the principles and practical applications of the disclosure and to enable others skilled in the art to understand the disclosure with various modifications suitable for the particular intended use.

Claims

1. A method for detecting a drug in a drug package, comprising: Receive an image of a pharmaceutical package containing one or more pharmaceutical products, the image including label content displayed on the surface of the pharmaceutical package; The label content on the surface of the drug packaging is detected using an artificial intelligence engine; as well as Generate a modified image of the drug packaging with the label content removed from its surface. The artificial intelligence engine is a first artificial intelligence engine, and the modified image is a first modified image. The method further includes: Receive order information for one or more of the medicines and an identifier for the medicine packaging; A second artificial intelligence engine is used to detect individual drugs among the one or more drugs in the first modified image.

2. The method according to claim 1, wherein the label content includes commercial marketing information, patient identification information, or personal healthcare information.

3. The method according to claim 2, wherein the commercial marketing information includes a logo or company name; The patient identification information includes the patient's name, patient's phone number, patient's address, or patient identification number; and The personal healthcare information mentioned therein includes the name of the one or more medicines, the quantity of each of the one or more medicines, the prescribed dosing time for each of the one or more medicines, one or more barcodes associated with the one or more medicines, prescriptions, patient accounts, or identification numbers.

4. The method according to claim 1, further comprising: In response to receiving the image of the drug packaging, gamma correction is performed on the image of the drug packaging to generate a gamma-corrected image of the drug packaging; Gaussian blur denoising is performed on the gamma-corrected image of the drug packaging to generate a denoised image of the drug packaging; as well as Automatic image thresholding is performed on the denoised image of the drug packaging to generate a foreground-background separated image of the drug packaging; The use of the artificial intelligence engine to detect the tag content includes: The artificial intelligence engine is used to detect the label content on the surface of the foreground-background separated image of the drug packaging.

5. The method of claim 4, wherein the artificial intelligence engine is a convolutional neural network.

6. The method of claim 5, wherein the convolutional neural network comprises a plurality of convolutional layers, at least some of the plurality of convolutional layers being interconnected by skip connections.

7. The method according to claim 1, further comprising: A second modified image of the drug packaging is generated, the second modified image including markings that distinguish the individual drugs among the one or more drugs and associate the one or more drugs with the order information and the identifier for the drug packaging.

8. The method of claim 7, wherein the marker distinguishing the individual drug among the one or more drugs comprises one or more bounding boxes.

9. The method of claim 7, wherein the order information includes the name of one or more drugs in the drug packaging, the method further comprising: Using a third artificial intelligence engine, at least some of the drugs are identified in the second modified image based on the names of the one or more drugs; The name is associated with drug attributes in a reference database.

10. The method of claim 9, wherein the at least some of the drugs in the one or more drugs include the broken drugs in the one or more drugs.

11. The method of claim 9, wherein the method further comprises: The third artificial intelligence engine is used to identify a portion of the one or more pharmaceutical products as fragments resulting from damage to the one or more pharmaceutical products.

12. The method according to claim 9, further comprising: In the second modified image, annotate at least some of the one or more drugs with the names of the drugs.

13. The method of claim 12, further comprising: Use a temporary name to annotate any of the one or more medicines that are not annotated with the name.

14. The method of claim 12, wherein the name is a first name, and the order information includes the National Drug Code (NDC) of the one or more drugs in the drug packaging, the method further comprising: Match the NDCs that are not associated with at least some of the drugs that have been annotated with the first name among the one or more drugs with the drug reference data; as well as Based on the drug reference data, any drug among the one or more drugs that is not annotated with the name but has an associated NDC that matches the drug reference data is annotated with a second name.

15. The method of claim 14, wherein the drug reference data includes drug shape, drug color, drug etching, drug marking, drug weight and / or drug label.

16. A system comprising: processor; as well as A memory coupled to the processor and including computer-readable program code embodied in the memory, the computer-readable program code being executable by the processor to perform operations including: Receive an image of a pharmaceutical package containing one or more pharmaceutical products, the image including label content displayed on the surface of the pharmaceutical package; The label content on the surface of the drug packaging is detected using an artificial intelligence engine; and Generate a modified image of the drug packaging with the label content removed from its surface. The artificial intelligence engine is a first artificial intelligence engine, and the modified image is a first modified image. The operation further includes: Receive order information for one or more of the aforementioned pharmaceutical products and an identifier for the pharmaceutical product packaging; and A second artificial intelligence engine is used to detect individual drugs among the one or more drugs in the first modified image.

17. The system of claim 16, wherein the operation further comprises: In response to receiving the image of the drug packaging, gamma correction is performed on the image of the drug packaging to generate a gamma-corrected image of the drug packaging; Gaussian blur denoising is performed on the gamma-corrected image of the drug packaging to generate a denoised image of the drug packaging; as well as Automatic image thresholding is performed on the denoised image of the drug packaging to generate a foreground-background separated image of the drug packaging; The use of the artificial intelligence engine to detect the tag content includes: The artificial intelligence engine is used to detect the label content on the surface of the foreground-background separated image of the drug packaging.

18. The system of claim 17, wherein the artificial intelligence engine is a convolutional neural network.

19. A computer program product comprising: A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including computer-readable program code embodied in the medium, the computer-readable program code being executable by a processor to perform operations, the operations including: Receive an image of a pharmaceutical package containing one or more pharmaceutical products, the image including label content displayed on the surface of the pharmaceutical package; The label content on the surface of the drug packaging is detected using an artificial intelligence engine; and Generate a modified image of the drug packaging with the label content removed from its surface. The artificial intelligence engine is a first artificial intelligence engine, and the modified image is a first modified image. The operation further includes: Receive order information for one or more of the aforementioned pharmaceutical products and an identifier for the pharmaceutical product packaging; and A second artificial intelligence engine is used to detect individual drugs among the one or more drugs in the first modified image.

20. The computer program product of claim 19, wherein the operation further comprises: In response to receiving the image of the drug packaging, gamma correction is performed on the image of the drug packaging to generate a gamma-corrected image of the drug packaging; Gaussian blur denoising is performed on the gamma-corrected image of the drug packaging to generate a denoised image of the drug packaging; as well as Automatic image thresholding is performed on the denoised image of the drug packaging to generate a foreground-background separated image of the drug packaging; The use of the artificial intelligence engine to detect the tag content includes: The artificial intelligence engine is used to detect the label content on the surface of the foreground-background separated image of the drug packaging.

Images